423 Commits

Author SHA1 Message Date
marpio
9dd0eaaa3a Filter out pseudo versions from storage on list endpoint (#1114)
* list should not return pseudo versions

* rm empty line

* handle case when repo does not exist anymore
2019-03-11 15:58:10 -04:00
Marwan Sulaiman
03f06db0f4 storage/gcp: Optimize Exists check from Getter (#1118)
* storage/gcp: remove Exists check from Getter

* pr fixes
2019-03-11 12:18:27 -04:00
Marwan Sulaiman
fb9437ddff pkg/stash: implement a Redis lock for stashing (#1116)
* pkg/stash: implement a Redis lock for stashing

* fix tests

* fix op
2019-03-11 02:48:04 -04:00
Thanatat Tamtan
c3d8a055d7 ZipHandler set content type (#1117) 2019-03-10 16:15:25 -04:00
Marwan Sulaiman
3b0a9b5a68 ci/cd: add .drone.yml file (#1091)
* ci/cd: add .drone.yml file

* add docker step

* put tag in right place

* separate docker builds for push vs tag

* move e2e to .drone.yml

* comment

* try gcp backend

* temp remove clean

* temp remove clean

* buckets confirmed

* fix docker latest

* sign yaml file + remove rc
2019-03-10 14:14:14 -04:00
Marwan Sulaiman
36dae02c03 storage/minio: chunk zip file (#1107)
* storage/minio: chunk zip file

* multi err on clean up

* fix doc
2019-03-06 16:13:24 -05:00
Marwan Sulaiman
26d565d7bc catalog: fix vet errors (#1090)
* catalog: fix vet errors

* fix typo
2019-02-26 10:57:18 -08:00
Marwan Sulaiman
dbe44e627a storage/gcp: use base64 for json key (#1089) 2019-02-25 15:16:13 -08:00
Marwan Sulaiman
a27ef13b71 storage/gcp: allow json key to be used for GCS access (#1088) 2019-02-25 13:59:00 -08:00
Marwan Sulaiman
12ce2a6499 pkg/stash: add Etcd as a SingleFlight backend (#1070)
* pkg/stash: add Etcd as a SingleFlight backend

* fix config tests

* fmt

* pr fixes

* fix conflicts
2019-02-22 11:35:45 -08:00
fsouza
3525cdaa4a pkg/config: make sure envconfig doesn't override config values for port (#1074)
Fixes #1037.

envconfig can't actually support that use case, and because athens was
using the `default:` tag only for the port field, handling this manually
is the easiest and functional solution.
2019-02-19 12:11:33 -05:00
marpio
831b5616b5 Check code defaults and config.toml defaults are the same (#1073)
* rm TraceExporterURL and StatsExporter default vals

* add test

* fix test

* revert TraceExporterURL and StatsExporter changes

* improve test error msg

* ignore GoEnv since it's set in CI and dev differently
2019-02-19 07:46:29 -08:00
Marwan Sulaiman
94360f3930 pkg/config: make StatsExporter consistent with config.toml (#1072) 2019-02-18 22:19:44 +01:00
Marwan Sulaiman
dc358ba096 Keep code defaults and config.toml defaults the same (#1062) 2019-02-15 20:48:21 -08:00
Boris Popovschi
4dfa99320f Add catalog to minio (#1040)
* Implement cataloger interface for Minio

* Catalog fix

* Implemented Catalog method in the Minio storage package

* code fmt fix

* fmt fix

* remove unused channel
2019-02-15 20:02:59 -08:00
marpio
c3d1d14d23 Use default conf values if file not found (#1022)
* default conf if no file provided

* move to the config pkg

* rm default config path

* rm log
2019-02-14 11:32:53 -08:00
Chris Mills
2cf10c3ead Allow mgo.ParseURL to set [/database] (#997)
* Allow mgo.ParseURL to set [/database]

As per Michael's solution we need to be able to set the database from
either the parsed connection string, or fallback to athens if not set.

Signed-off-by: Chris M <me@christophermills.co.uk>

* Remove TimeoutDuration which was removed #928

Signed-off-by: Chris M <me@christophermills.co.uk>

* Correct conf_test to accommodate for changes

I'd missed out the addition of DefaultDBName from the test, this has now
been corrected

Signed-off-by: Chris M <me@christophermills.co.uk>

* Make sure gofmt runs automatically on this machine

Y'know simple stuff.

Signed-off-by: Chris M <me@christophermills.co.uk>
2019-02-14 10:21:34 -08:00
Marwan Sulaiman
aa3fb3389c pkg/config: Update GoGetWorkers default/docs (#1052)
* pkg/config: Update GoGetWorkers default/docs

* fix tests
2019-02-11 11:51:38 -05:00
Chad Kunde
7811524c22 Version filtering (#1050)
* extend filtering logic and configuration to include version lists

Module filtering is very useful, but many deployments will need to
satisfy even more granular constrainsts.  Enterprises may need
approved lists specific down to the minor (or patch) version element.

Version filtering logic is similar to the module filtering, in that
it's a prefix match of the version requested against each entry in the
version filter list.

Closes #1045

* include version filtering in documentation with example

* allow filtering when version is missing

Endpoints that do not specify a version, such as "@v/list", only need
to be filtered by module rules.
2019-02-05 09:04:09 +01:00
marpio
128a7d6b3c gcp compliance tests (#1030)
* gcp compliance tests

* fix error wrapping, rm redundant funcs

* add mod, ver info to the error

* adjust bucket name

* adjust the catalog test
2019-01-16 16:21:34 +01:00
Joshua Rubin
84eff04ed2 Use ambient aws credential provider for s3 (#1041)
* use ambiant aws credentials

Signed-off-by: Joshua Rubin <joshua@rubixconsulting.com>

* better naming, docs and examples for default aws config

Signed-off-by: Joshua Rubin <joshua@rubixconsulting.com>
2019-01-16 08:47:21 +01:00
Pascal Dierich
fd10bed609 pkg/download/addon: fix test error msg in TestPoolLogic. (#1034) 2019-01-07 22:04:09 -05:00
Marwan Sulaiman
f0e931311b pkg/stash: allow nonsemver requests to be resolved properly (#1015) 2019-01-07 20:51:22 -05:00
Marwan Sulaiman
0d94af2503 catalog: clean up unused interfaces (#1027)
* catalog: clean up unused interfaces

* fix op name
2019-01-05 15:04:46 -05:00
Pascal Dierich
9964e84902 fixes typos (#1031)
* fix typo in DEVELOPMENT.md

* fix doc typo in upstream_lister.go

* Remove Olympus references

* fix typo in DEVELOPMENT.md

* Update DEVELOPMENT.md
2019-01-05 13:21:24 -05:00
Marwan Sulaiman
2bd4c9d12f mongo: use explicit URL for testing (#1025) 2019-01-04 20:23:49 +01:00
Manu Gupta
23b285d49d Catalog endpoint support for mongo (#985)
* Added new cataloger interface

* Implementing catalog protocol

* Propagated to protocol and over

* First round of fixes

* S3 almost ready, need to be tested

* Going on with testing s3

* Better testing with s3

* Simplified catalog tests

* Preparing gcp tests to access a gcp instance

* Fixing initialization errors

* Removed some prints

* Gcp ready, to be tested

* Gcp working

* Aligned bucket mock to catalog method

* Switched res payload to json

* Added catalog method to all storage instances

* Added catalog method to unsupported storages

* Fixed with pool test

* Restored tests

* Fixed gcp constructor

* Implemented catalog for fs

* Removed trace

* E2e tests, fixed fs

* Fixed module name return value

* Added cataloger method to azure storage

* Added docs

* Changed pagesize parameter name

* Fixed gofmt error

* Added json tags to result. Fixed lint warning

* Removed extra line

* Changed not implemented error to http.KindNotImplemented

* Checking for inequality on results

* Lower-cased json keys

* Added cleaning of path separator

* Fixed review comments

* Add catalog endpoint for mongo

* Add omitempty to ID

* Fix catalog tests

* update for next token

* fix e2e

* Make query readable

* Fix language and e2e script

* remove new line
2019-01-02 22:13:56 -08:00
Manu Gupta
3b621caa85 Move catalog handler out of download protocol (#1020)
* Move catalog handler out of download protocol

* Bring back logger

* update comment
2019-01-02 21:55:07 -08:00
M4RK
a7aaf3f1e4 Modified S3 storage to check for AWS default credentials. (#1009)
* Modified S3 storage to check for AWS default credentials.

* Updated S3 storage documentation.

* Fixed typo.
2018-12-29 22:11:50 -08:00
Marwan Sulaiman
5870aeee8d Remove Buffalo (#1010)
* Remove Buffalo

* gofmt

* pr fixes

* fix subrouter

* bring back secure middleware + pr fixes

* better place for subrouter

* vendor
2018-12-22 20:24:25 -05:00
Marwan Sulaiman
36aba5915c Revert TraceExporter Default (#1012)
* Revert TraceExporter Default

* fix tests
2018-12-22 11:52:02 -05:00
Marwan Sulaiman
2d2bd8eff9 only check config perms in production (#1013) 2018-12-22 11:33:57 -05:00
Nafis Faysal
dd801d86b1 Refactoring code to improve HTTP status code (#1005)
* refactoring code to improve HTTP status code

* refactoring code to improve HTTP status code

* refactoring code to improve HTTP status code

* refactoring code to improve HTTP status code
2018-12-20 22:15:03 +01:00
Tomasz Tomalak
fb696b2d84 Add prometheus metrics collectors for http handlers (#958)
* Add prometheus metrics exporter for http handlers

* Use prometheus from opencensus

* Add comment to unused func() param

* Add package names to operation description in observ package

* Sort exported/unexported functions
2018-12-20 10:19:22 +01:00
Manu Gupta
5eba6f2e47 File permissions on config files allow more restrictive setting (#966)
* File permissions on config files allow more restrictive setting

* Make the tests a bit more readable

* Bring back the tests dude

* Update error message

* remove else

* Add more test cases

* Change Lstat to Stat

* Add note for umask

* Make sure the permissions are 0600 or lower

* Update config file

* Do not check for windows

* Fix CI errors

* Fix CI test
2018-12-18 16:26:04 +01:00
Manu Gupta
ef3d0130f6 Pass ctx down the vcs lister stack and add tracing to it (#992) 2018-12-13 23:27:02 -05:00
Federico Paolinelli
0258e17d89 Catalogendpoint (#955)
* Added new cataloger interface

* Implementing catalog protocol

* Propagated to protocol and over

* First round of fixes

* S3 almost ready, need to be tested

* Going on with testing s3

* Better testing with s3

* Simplified catalog tests

* Preparing gcp tests to access a gcp instance

* Fixing initialization errors

* Removed some prints

* Gcp ready, to be tested

* Gcp working

* Aligned bucket mock to catalog method

* Switched res payload to json

* Added catalog method to all storage instances

* Added catalog method to unsupported storages

* Fixed with pool test

* Restored tests

* Fixed gcp constructor

* Implemented catalog for fs

* Removed trace

* E2e tests, fixed fs

* Fixed module name return value

* Added cataloger method to azure storage

* Added docs

* Changed pagesize parameter name

* Fixed gofmt error

* Added json tags to result. Fixed lint warning

* Removed extra line

* Changed not implemented error to http.KindNotImplemented

* Checking for inequality on results

* Lower-cased json keys

* Added cleaning of path separator

* Fixed review comments

* Fixed docs
2018-12-12 21:17:26 +01:00
Manu Gupta
0b2aece18d 1st attempt at speeding up list endpoint (#991) 2018-12-11 21:10:24 +01:00
Manu Gupta
1d7155eeb7 Only fetch versions while list is called (#986) 2018-12-09 19:13:01 +01:00
Pontus Leitzler
fe032bc5ca Add support for TLS (#979)
* Add support for TLS

* Fix review comments
2018-12-09 07:27:49 -05:00
Pontus Leitzler
5ab820de96 Propagate HOME env to go command execution (#960) 2018-12-06 11:12:24 -08:00
Manu Gupta
c097fc8d79 Add test when go mod download fails (#962)
* Add test when go mod fails

* Do not export the mocked fetcher
2018-12-05 13:21:47 -08:00
Tomasz Tomalak
edc2f7842c Add check if GCP bucket exists and remove GCP Bucket create (#952)
* Add check if GCP bucket exists, Remove bucket creation

* Update GCP section in Storage Configuration docs

* Add more explicit info if bucket not exists
2018-12-01 17:53:15 +01:00
Aravind
48f7ca7780 Implement storage.Backend for azureblob (#917) 2018-11-29 17:06:19 +01:00
Manu Gupta
66cb01e668 Add tracing defaults for jaeger (#934)
* Add tracing defaults for jaeger

* Fix tests

* Add comment in config file as well
2018-11-29 10:44:34 -05:00
Yorman
b92a642f1a Add error checks in a few test files and fix misspelled words (#919)
* Remove inefficient assignment of variable cxt in pkg/storage/fs

* Remove inefficient assignment of variable cxt in pkg/storage/minio

* Remove inefficient assignment of variable cxt in pkg/storage/mongo

* Fix commonly misspelled English words successfully and accessible

* Remove inefficient assignment of variable cxt in pkg/download

* Remove inefficient assignment of variable cxt in pkg/module

* Add missing error checks in a few unit-test files

* Revert removal of inefficient assignment of variable cxt

The variable, although is not being used, it maintains the readability
in the functions where it’s being assigned. The removal of `cxt` in
these files may introduce errors when a new function taking these
values is called. Two other contributors agree that it’s easy to miss
this detail when passing the variable to other functions and the traces
are not being recorded.

* Modify unit-test error reporting for the config package

* Modify config test to exit when ioutil.TempFile or os.Chmod fail
2018-11-21 20:16:05 +01:00
Manu Gupta
ff99d30bc2 skip test for chmod on windows (#938)
* add logs

* apples and oranges

* apples and oranges

* Fix bug

* stringify mode

* fix stuff

* figure out why chmod does not work

* see what happens outside the temp folder

* Does it have a long path

* check if proper runtimes are called

* Skip tests for windows

* clean up
2018-11-21 06:23:23 -05:00
Manu Gupta
c11212601b Checks for windows permissions (#902)
* Checks for windows permissions

* Fix bug

* fix test

* Update perms

* Fix perms
2018-11-20 20:56:35 +01:00
azure-pipelines[bot]
1984dbdaaf Set up CI with Azure Pipelines (#862)
* Set up CI with Azure Pipelines

* trial & error

* use robocopy

* fix slash

* handle robocopy exit code

* rm goroot

* build proxy

* build & test proxy

* Skip Test_checkFilePerms on Win until issue resolved

* start minio and mongo

* do not create a new network on win 2016

* use win docker img for mongo

* download and start minio since there is no win docker img

* fix minio download & exec

* add minio access key and secret

* use new mongodb img

* fix docker-compose mongo

* trying to fix mongo tests

* ipconfig

* lets try that

* use docker inspect to get the container ip

* echo mongo container ip

* debuging docker inspect ipaddress

* set mongo env var to internal addr

* debug

* quotation is difficult

* without docker

* pass db dir

* split steps

* add some echos to know what takes so long

* cleanup

* add downloadURL var for minio and mongo

* rm windows docker compose file
2018-11-18 09:04:42 -05:00
marpio
329c86b843 use minio docker img on travis (#930) 2018-11-18 09:19:59 +01:00