164 Commits

Author SHA1 Message Date
Marwan Sulaiman
90af973c3d pkg/storage/external: remove debug lines (#1606) 2020-04-20 09:38:47 -04:00
Marwan Sulaiman
3c4db4ce86 pkg/storage: add External implementation (#1587)
* pkg/storage: add External implementation

* fix conflicts

* use newly instantiated client
2020-03-27 13:35:52 -04:00
Marwan Sulaiman
0bb95c7351 pkg/storage: make Checker optional in storage.Backend (#1580)
* pkg/storage: make Checker optional in storage.Backend

* pass storage
2020-03-18 19:07:00 -04:00
vapod
4530a58359 Add forcepathstyle config for s3 (#1533)
* Add forcepathstyle config for s3

* Bump chart version

* Fix typos in s3 config struct
2020-02-10 16:03:25 -08:00
Aleksandr Razumov
5b194688a0 storage/s3: add custom endpoint url support (#1467) 2019-12-05 13:04:03 -08:00
Aaron Schlesinger
707b7b5413 Fixing the build failures from a missing GCS Key (#1480)
* Fixing the build failures from a missing GCS Key

Accidentally introduced in https://github.com/gomods/athens/pull/1428

* Adding a TODO to the drone build

* adding a step to test the Athens server startup

* Adding the JSONKey back to the default config

* use temporary image

* Switching back to the old env var for the GCS JSON Key

cc/ @marwan-at-work

* removing the ServiceAccount. we want to stick with JSONKey

* reverting to original key fetching code

* fixing build err

* bumping chart ver
2019-12-05 10:49:56 -08:00
Aditya Sharma
e0ec46b221 Add GCS to helm chart (#1428)
* Add GCS to helm chart

* base64 encode serviceAccount in the chart

* Standardize GCP Service Account references

* pipe storage.gcp.serviceAccount to quote
2019-12-03 18:28:07 -08:00
Jens Hausherr
f578472490 Unescape key before attempting to split module and version (#1457)
* Unescape key before attempting to split module and version

* PathUnescape key in minio/lister

* use unescaped path for module

* Compare againse escaped key

* Exctract common key parsing from cataloger and lister
2019-11-25 16:27:25 -08:00
Chris Mills
82ecff968f Correct issue with continuation token failures on deploy (#1415)
* Add working image for minio

Between RELEASE.2019-09-26T19-42-35Z and RELEASE.2019-10-02T21-19-38Z
there's an issue with continuation tokens that's caused #1414 and #1413
to fail.

Signed-off-by: Chris M <me@christophermills.co.uk>

* Undo docker changes, update minio to v6

* Narrow issue down to catalog compliance test

Signed-off-by: Chris M <me@christophermills.co.uk>
2019-10-08 16:54:20 -04:00
Kenshi Kamata
800024fc6f Fix ignoring error (#1356)
* fix ignoring error

* fix operator

* fix minio error handling

* fix get url to return error if failed parsing

* add error handling

* position of statement to get redirect url was wrong.
2019-09-16 16:15:51 -05:00
Kenshi Kamata
667b9fdf65 Fix typo (#1358)
* fix duplicated 'the'

* fix typo found by misspell
2019-08-27 10:55:09 -07:00
Manu Gupta
16739a1213 Add relative URI for AWS fargate. (#1337)
* f

* r

* f

* gofmt

* ep

* Container url

* Asd

* nit

* Update docs

* keep old stuff

* old is gold

* indent

* asd
2019-08-21 14:21:01 +01:00
Bob Maertz
049d58caa8 Write tests for the new Mongo query function (#1314)
* WIP: Added tests to query implemetation.

* Fixed tests for mongo getter query function.
2019-08-15 14:04:19 -07:00
Bob Maertz
994810324e Fix Mongo initialization to use environment variables for db/coll names (#1331)
* Updated mongo init to use db name env variable.

* Updated mongo init to use coll name env var.

* Fixed merge conflict.

* Updated formatting in mongo_test.
2019-08-14 21:41:21 -07:00
Bob Maertz
3a3d474e1b Add check for mongo configuration errors. (#1329) 2019-08-12 23:54:23 -07:00
Soumya GB
603e2f3dc7 Adds cataloger for Azure blob storage (#1313)
* Adds cataloger for Azure Blob storage

* Revert "Adds cataloger for Azure Blob storage"

This reverts commit 9476c9e256.

* Implements Cataloger interface for Azure blob storage. Fixes #995.

* Upgraded azblob version to v0.7.0

* Upgrades azblob version to v0.7.0

* Fix formatting

* Fix formatting issues

* Use keyed field for comp literal

* reorder import pkgs
2019-08-12 23:49:29 -07:00
Manu Gupta
be908fe713 Fixes endpoint creds (#1325)
* f

* r
2019-08-06 10:45:14 -04:00
Yi Tang
b6bcf0e191 storage/fs,minio: fix list module with same prefix (#1312) 2019-08-01 13:16:52 -04:00
Manu Gupta
5bff5fe74d endpoint creds (#1291)
* endpoint creds

* gofmt

* Add credential chain

* error handling

* Fix things

* add docs

* f

* asd

* fix
2019-07-22 22:47:36 +02:00
James Naftel
33d01b1a7a Fixed Minio NewStorage to return the MakeBucket error instead of the (#1302)
Bucket exists error.  See #1295
2019-07-16 15:23:04 -07:00
Bob Maertz
8e8e886f84 Updates to mongo getter to reduce duplicate code (#1297)
* Updates to mongo getter.

- Consolidated duplicate code in getter.Info and getter.GoMod functions.
- Due to consolidation, removed ptr to 'result' in getter.Info.

* Updates to getter.go.

Added tracing back into Info and GoMod functions.
Made query a private function.
2019-07-08 14:54:53 -07:00
Manu Gupta
e5173c57cd [s3] base uri is not used anywhere remove it (#1293)
* base uri is not used anywhere remove it

* remove from struct
2019-07-02 15:07:12 -07:00
marpio
5cec5f6366 minio : Use no. of objects for Exists check (#1276)
* minio checker: check len instead of mod only

* fix syntax err

* fix err

* handle list errs
2019-06-07 15:54:26 -07:00
marpio
3bda516c19 Azureblob tests + checker fix (#1251)
* add azureblob tests

* get account name from env

* benchmarks

* skip test if account missing
2019-06-07 15:21:09 -07:00
marpio
91d7c47ba5 gcs: check if all files exist (#1274) 2019-06-06 15:47:50 -04:00
Marwan Sulaiman
3779208d49 storage/s3: fix checker prefix (#1239) 2019-05-24 09:47:24 -04:00
marpio
a75776b4ee Simplify gcs catalog (#1224)
* simplify gcs catalog impl

* invert condition
2019-05-14 13:39:09 -04:00
marpio
b2bcdce18c fix List call - token should be on 3. position (#1212) 2019-05-08 17:56:29 -07:00
Arpit Gogia
974077e73b Mongo go driver beta integration (#1042)
* Changed mongo.go to use new driver

* Modified mongo cataloger

* More new driver related changes

* Change lister.go

* Change saver.go

* Change imports

* Remove unnecessary Count query

* Use IndexView for indexing

* Rename ModuleStore fields

* Use map of key:sorting-order for creating the index

* Minor changes

* Use client options to configure mongo client

* Use method chaining

* gofmt changes

* Change imports

* Fix some build errors

* Use new GridFS API

* Fix more build errors

* Add Go Mongo driver to dependency modules

* Use multierror

* Leave download stream open

* Remove mgo error handling

* Copy zip instead of loading all in memory

* Use context.WithTimeout() wherever possible

* Raise KindNotFound when mod@ver isn't found

* NopCloser not needed

* Fix IndexView error

* Fix build errors

* Remove another mgo error usage

* Fix build error

* Changes according to review

* Formatting changes as per gofmt

* Modify gofmt argument to show the expected formatting (diff)

* Handle ErrNoDocument error and error arising from query execution

* Fix kind of returned error

* Minor changes

* Bug fixes

* gofmt related changes

* Minor change

* Use Insecure from MongoConfig, remove Insecure from global Config

* Remove stray print statement
2019-04-17 19:59:01 +02:00
Marwan Sulaiman
4a4641b66b storage/gcp: create a bucket on every test run (#1176)
* storage/gcp: create a bucket on every test run

* ensure removal

* set the strg

* remove pkg level vars

* clean up func call
2019-04-04 11:48:41 -07:00
Marwan Sulaiman
8d46c8d026 storage/gcp: fix lister prefix (#1172)
* storage/gcp: fix lister prefix

* add slash to s3

* but really this time

* account for windows
2019-04-01 10:14:55 -04:00
Aaron Schlesinger
edd0216e09 Fixing typo in compliance test harness (#1170) 2019-03-30 14:37:12 +01:00
marpio
20aeee6c76 fs: Use no. of objects for Exists check (#1143)
* fs: check for len == 3 in Exists

* fix tests

* check for PathError

* check for isnotexists err
2019-03-28 11:09:08 -04:00
marpio
33adb2bb01 check if ver exists by counting the parts (#1145) 2019-03-28 01:47:52 -04:00
Kirill Motkov
3ff753d3c5 Improvement of len() comparisons style (#1153)
Builtin len() function can't return value smaller than zero.
2019-03-26 21:12:02 +01:00
marpio
58747ed290 aws s3: use no of objects for Exists check (#1142) 2019-03-19 20:56:12 +01:00
Marwan Sulaiman
081ec9126e pkg/stash: Add GCS implementation (#1124)
* pkg/stash: Add GCS implementation

* fix docs
2019-03-13 11:07:07 -07:00
Marwan Sulaiman
03f06db0f4 storage/gcp: Optimize Exists check from Getter (#1118)
* storage/gcp: remove Exists check from Getter

* pr fixes
2019-03-11 12:18:27 -04:00
Marwan Sulaiman
3b0a9b5a68 ci/cd: add .drone.yml file (#1091)
* ci/cd: add .drone.yml file

* add docker step

* put tag in right place

* separate docker builds for push vs tag

* move e2e to .drone.yml

* comment

* try gcp backend

* temp remove clean

* temp remove clean

* buckets confirmed

* fix docker latest

* sign yaml file + remove rc
2019-03-10 14:14:14 -04:00
Marwan Sulaiman
36dae02c03 storage/minio: chunk zip file (#1107)
* storage/minio: chunk zip file

* multi err on clean up

* fix doc
2019-03-06 16:13:24 -05:00
Marwan Sulaiman
26d565d7bc catalog: fix vet errors (#1090)
* catalog: fix vet errors

* fix typo
2019-02-26 10:57:18 -08:00
Marwan Sulaiman
dbe44e627a storage/gcp: use base64 for json key (#1089) 2019-02-25 15:16:13 -08:00
Marwan Sulaiman
a27ef13b71 storage/gcp: allow json key to be used for GCS access (#1088) 2019-02-25 13:59:00 -08:00
Boris Popovschi
4dfa99320f Add catalog to minio (#1040)
* Implement cataloger interface for Minio

* Catalog fix

* Implemented Catalog method in the Minio storage package

* code fmt fix

* fmt fix

* remove unused channel
2019-02-15 20:02:59 -08:00
Chris Mills
2cf10c3ead Allow mgo.ParseURL to set [/database] (#997)
* Allow mgo.ParseURL to set [/database]

As per Michael's solution we need to be able to set the database from
either the parsed connection string, or fallback to athens if not set.

Signed-off-by: Chris M <me@christophermills.co.uk>

* Remove TimeoutDuration which was removed #928

Signed-off-by: Chris M <me@christophermills.co.uk>

* Correct conf_test to accommodate for changes

I'd missed out the addition of DefaultDBName from the test, this has now
been corrected

Signed-off-by: Chris M <me@christophermills.co.uk>

* Make sure gofmt runs automatically on this machine

Y'know simple stuff.

Signed-off-by: Chris M <me@christophermills.co.uk>
2019-02-14 10:21:34 -08:00
marpio
128a7d6b3c gcp compliance tests (#1030)
* gcp compliance tests

* fix error wrapping, rm redundant funcs

* add mod, ver info to the error

* adjust bucket name

* adjust the catalog test
2019-01-16 16:21:34 +01:00
Joshua Rubin
84eff04ed2 Use ambient aws credential provider for s3 (#1041)
* use ambiant aws credentials

Signed-off-by: Joshua Rubin <joshua@rubixconsulting.com>

* better naming, docs and examples for default aws config

Signed-off-by: Joshua Rubin <joshua@rubixconsulting.com>
2019-01-16 08:47:21 +01:00
Marwan Sulaiman
f0e931311b pkg/stash: allow nonsemver requests to be resolved properly (#1015) 2019-01-07 20:51:22 -05:00
Marwan Sulaiman
0d94af2503 catalog: clean up unused interfaces (#1027)
* catalog: clean up unused interfaces

* fix op name
2019-01-05 15:04:46 -05:00
Marwan Sulaiman
2bd4c9d12f mongo: use explicit URL for testing (#1025) 2019-01-04 20:23:49 +01:00