Commit Graph

26 Commits

Author SHA1 Message Date
south-mer
11d674c8fb feat: GCP checksum (#2052) 2025-08-06 08:24:54 +02:00
Vanes Angelo
900cb4f7c1 Upgrade to AWS SDK v2 (#1938) (#1950)
Upgrades the AWS SDK to v2. AWS S3 bucket urls will now error if they are not prefixed with a schema (example: https://).
2024-04-28 21:38:42 -07:00
yatesliang
aca3cc0090 test: unit test for testing KindUnexpected case in Mongo query function (#1883) 2023-09-25 00:24:02 -07:00
Nicholas Wiersma
d932d50232 chore: lint code with golangci-lint (#1828)
* feat: add golangci-lint linting

* chore: fix linter issues

* feat: add linting into the workflow

* docs: update lint docs

* fix: cr suggestions

* fix: remove old formatting and vetting scripts

* fix: add docker make target

* fix: action go caching

* fix: depreciated actions checkout version

* fix: cr suggestion

* fix: cr suggestions

---------

Co-authored-by: Manu Gupta <manugupt1@gmail.com>
2023-02-24 20:39:17 -08:00
Nicholas Wiersma
ab64133949 feat: replace ioutil with io and os (#1816) 2023-01-27 20:42:47 +05:30
Bob Maertz
049d58caa8 Write tests for the new Mongo query function (#1314)
* WIP: Added tests to query implemetation.

* Fixed tests for mongo getter query function.
2019-08-15 14:04:19 -07:00
Bob Maertz
994810324e Fix Mongo initialization to use environment variables for db/coll names (#1331)
* Updated mongo init to use db name env variable.

* Updated mongo init to use coll name env var.

* Fixed merge conflict.

* Updated formatting in mongo_test.
2019-08-14 21:41:21 -07:00
Bob Maertz
3a3d474e1b Add check for mongo configuration errors. (#1329) 2019-08-12 23:54:23 -07:00
Arpit Gogia
974077e73b Mongo go driver beta integration (#1042)
* Changed mongo.go to use new driver

* Modified mongo cataloger

* More new driver related changes

* Change lister.go

* Change saver.go

* Change imports

* Remove unnecessary Count query

* Use IndexView for indexing

* Rename ModuleStore fields

* Use map of key:sorting-order for creating the index

* Minor changes

* Use client options to configure mongo client

* Use method chaining

* gofmt changes

* Change imports

* Fix some build errors

* Use new GridFS API

* Fix more build errors

* Add Go Mongo driver to dependency modules

* Use multierror

* Leave download stream open

* Remove mgo error handling

* Copy zip instead of loading all in memory

* Use context.WithTimeout() wherever possible

* Raise KindNotFound when mod@ver isn't found

* NopCloser not needed

* Fix IndexView error

* Fix build errors

* Remove another mgo error usage

* Fix build error

* Changes according to review

* Formatting changes as per gofmt

* Modify gofmt argument to show the expected formatting (diff)

* Handle ErrNoDocument error and error arising from query execution

* Fix kind of returned error

* Minor changes

* Bug fixes

* gofmt related changes

* Minor change

* Use Insecure from MongoConfig, remove Insecure from global Config

* Remove stray print statement
2019-04-17 19:59:01 +02:00
Marwan Sulaiman
2bd4c9d12f mongo: use explicit URL for testing (#1025) 2019-01-04 20:23:49 +01:00
Manu Gupta
23b285d49d Catalog endpoint support for mongo (#985)
* Added new cataloger interface

* Implementing catalog protocol

* Propagated to protocol and over

* First round of fixes

* S3 almost ready, need to be tested

* Going on with testing s3

* Better testing with s3

* Simplified catalog tests

* Preparing gcp tests to access a gcp instance

* Fixing initialization errors

* Removed some prints

* Gcp ready, to be tested

* Gcp working

* Aligned bucket mock to catalog method

* Switched res payload to json

* Added catalog method to all storage instances

* Added catalog method to unsupported storages

* Fixed with pool test

* Restored tests

* Fixed gcp constructor

* Implemented catalog for fs

* Removed trace

* E2e tests, fixed fs

* Fixed module name return value

* Added cataloger method to azure storage

* Added docs

* Changed pagesize parameter name

* Fixed gofmt error

* Added json tags to result. Fixed lint warning

* Removed extra line

* Changed not implemented error to http.KindNotImplemented

* Checking for inequality on results

* Lower-cased json keys

* Added cleaning of path separator

* Fixed review comments

* Add catalog endpoint for mongo

* Add omitempty to ID

* Fix catalog tests

* update for next token

* fix e2e

* Make query readable

* Fix language and e2e script

* remove new line
2019-01-02 22:13:56 -08:00
marpio
4785e9c2b3 Remove storage timeouts (#928)
* remove storage timeouts

* fix tests
2018-11-16 16:00:08 -05:00
Tomasz Tomalak
f665b0c2eb Fix skipped mongo and minio tests (#791) 2018-10-18 14:36:28 -07:00
Marwan Sulaiman
97d8013876 Refactor storage tests (#727)
* Refactor storage tests

* minio: check object err

* remove inaccurate test

* storage/fs: use memory
2018-10-04 16:16:24 -07:00
marpio
29b6b6ffab Minio conf + tests refactoring (#680)
* cleanup tests and change minio port

* fix cleanup

* cleanup

* fix config test

* add comment to travis

* revert to generic minio addr

* fix test

* switch to test config

* adapt timeouts

* use example config

* fix test... again

* add new lines
2018-09-20 21:31:21 +02:00
Rohan Chakravarthy
0e470d0294 Plumb config - Latest (#627)
* switch proxy to config file

pull in single flight changes

* changes for single-flight

* intermediate stage. All tests passing. pkg still has env refs

* remove all env references

* delete config/env entirely

* fix failing tests

* create the config.toml file as part of dev setup

* create config file only if it doesn't exist

* update Dockerfiles to use config file

* move composing elements to the top

* verbose parameter naming

* newline

* add flag for config file path

* update docs with config file flag

* remove unnecessary nil check

* use filepath.join

* rename redis port to address

* fix path.join

* fix issues after merge

* add vendor dir
2018-09-11 15:04:20 -07:00
Arthur Mello
e5ce3f7c6e Set a default value for ATHENS_MONGO_CONNECTION_STRING env var (#580)
* Set a default value for ATHENS_MONGO_CONNECTION_STRING env var

- ATHENS_MONGO_CONNECTION_STRING will default to mongodb://127.0.0.1:27017 if not set

* Fix Get function call

* Update user of env.MongoConnectionString
2018-08-28 11:56:15 +02:00
Michal Pristas
042cbcaddf SSL Support for Mongo (#527)
* tested mongo ssl

* Added a todo
2018-08-21 18:09:49 +02:00
kteb
cdab5c515c Remove storage connector (#457)
* get rid of fake storage.Connector

* get rid of the storage.Connector for mango & rdbms

* Remove all the connector interfaces

* Updating all the test to get rid of the connect

* Updating connection to storage for proxy

* Updating connection to storage for olympus

* get rid of fake storage.Connector

* get rid of the storage.Connector for mango & rdbms

* Remove all the connector interfaces

* Updating all the test to get rid of the connect

* Updating connection to storage for proxy

* Updating connection to storage for olympus

* Fix reviews

* Fix reviews

* fixing op Name

* fix replace the check of an error insted of the object exists

* Fix error message in the test
2018-08-15 09:51:10 -07:00
Michal Pristas
767be5ba47 Common testing ground for storage (#215)
* added test for notfound

* 🚨 🔨

* hahaha

* test

* Olympus: default to PORT env var (#226)

* Olympus: default to PORT env var

* Athens: generalize env.Port function call

* Olympus: add PORT to .env

* test

* getset
'

* Docs

* separate package
2018-07-11 11:45:41 -07:00
marpio
58d42a1b57 Stream zips (#200)
* storage.saver interface - use io.reader instead of []byte

* stream zip - backendconnector

* stream zip - azurecdn

* stream zip - fs

* storage - switch to io.Reader

* update callers of Save

* rm empty line

* makezip stream

* refactor mergedb

* add multierror dep

* mergedb multierror

* scope err

* update storages

* return pipereader

* fix test

* use multierror for all strorage drivers

* review feedback

* azure cdn storage - use UploadStreamToBlockBlob

* use numUpload

* use timeout ctx
2018-07-07 08:46:43 -07:00
Martin Strobel
d0a18e7dd4 Add Context to Saver Interface (#179)
* Adding context to storage.Saver parameters

* +`buffalo.Context` as first param in storage.Saver

Fixes #177

* Parallelizing Upload to Azure Blob Storage

Fixes #175

* Fixing import clausing

* go fmt ./...

* Populating default context

* CI: Echo files modified by gofmt check to stderr

Doing so allows users to see which files still need to be formatted.

* Upgrading Go, then go fmt ./...

* Addressing Feedback

* more responding to feedback

* Adding worker save Default Context

* Removing tracing, buffalo.Context usage in Storage interface

github.com/bketelsen/buffet relies on using a `buffalo.Context`
however, often a `storage.Saver` is invoked from a buffalo worker,
which has no buffalo.Context associated with it. Therefore, we had
two options: create a buffalo context for each worker's Job, or
remove the buffalo.Context dependency in the Saver interface.

* Fixing missed changes

* Review feedback
2018-06-21 10:47:31 -07:00
Rob j Loranger
9df207561d Add .info to upload payload (#141)
* Add .info to upload payload
everywhere else as well

+ Upload payload includes Info
+ CLI uploads `version.info` as blob to storage
+ Getter and Saver methods updated for all storages implemented
+ Migration updated to include Info column
+ Tests no longer check RevInfo, now checking for Info same as Zip
+ Updated docs

* newline in .fizz

* change RevInfo.Info to []byte

+ change to []byte instead of io.Reader
+ adjust tests to check for info byte contents directly
+ add error check for minio client GetObject `pkg/storage/minio/getter.go`
2018-05-29 08:03:01 +02:00
marpio
0a3eb4c4ea fix golint issues (#110)
* fix stutters

* add comments

* add -set_exit_status flag to golint

* missing renames

* renames again
2018-03-26 11:18:18 -07:00
Aaron Schlesinger
a6721f7505 Remove the base URL (#60)
* Removing concept of a baseURL

Just using a single ‘module’ and ‘version’ concept is enough

* go build => buffalo build
2018-03-21 16:03:44 -07:00
Michal Pristas
0a6e4407a1 Implemented backing mongo storage for modules (#64)
* mongo storage initial impl

* added tests

* mongo tests passing
2018-03-20 11:28:42 -07:00