Commit Graph

16 Commits

Author SHA1 Message Date
south-mer
11d674c8fb feat: GCP checksum (#2052) 2025-08-06 08:24:54 +02:00
Nicholas Wiersma
d932d50232 chore: lint code with golangci-lint (#1828)
* feat: add golangci-lint linting

* chore: fix linter issues

* feat: add linting into the workflow

* docs: update lint docs

* fix: cr suggestions

* fix: remove old formatting and vetting scripts

* fix: add docker make target

* fix: action go caching

* fix: depreciated actions checkout version

* fix: cr suggestion

* fix: cr suggestions

---------

Co-authored-by: Manu Gupta <manugupt1@gmail.com>
2023-02-24 20:39:17 -08:00
Arpit Gogia
974077e73b Mongo go driver beta integration (#1042)
* Changed mongo.go to use new driver

* Modified mongo cataloger

* More new driver related changes

* Change lister.go

* Change saver.go

* Change imports

* Remove unnecessary Count query

* Use IndexView for indexing

* Rename ModuleStore fields

* Use map of key:sorting-order for creating the index

* Minor changes

* Use client options to configure mongo client

* Use method chaining

* gofmt changes

* Change imports

* Fix some build errors

* Use new GridFS API

* Fix more build errors

* Add Go Mongo driver to dependency modules

* Use multierror

* Leave download stream open

* Remove mgo error handling

* Copy zip instead of loading all in memory

* Use context.WithTimeout() wherever possible

* Raise KindNotFound when mod@ver isn't found

* NopCloser not needed

* Fix IndexView error

* Fix build errors

* Remove another mgo error usage

* Fix build error

* Changes according to review

* Formatting changes as per gofmt

* Modify gofmt argument to show the expected formatting (diff)

* Handle ErrNoDocument error and error arising from query execution

* Fix kind of returned error

* Minor changes

* Bug fixes

* gofmt related changes

* Minor change

* Use Insecure from MongoConfig, remove Insecure from global Config

* Remove stray print statement
2019-04-17 19:59:01 +02:00
Brent Pendergraft
3b6aeb986e mongo: remove drop dups option and return error if module exists (#913) 2018-11-16 15:35:46 -05:00
Aaron Schlesinger
c6644f82f9 Checking the number of bytes written to Mongo GridFS (#889)
* Checking the number of bytes written to Mongo GridFS

* adding missing import

* formatting imports
2018-11-12 14:09:16 -05:00
Manu Gupta
57b11e8489 Shorten op names and remove spans from handlers (#647) 2018-09-11 23:20:32 -04:00
Manu Gupta
bb250437da Instrumenting observability with opencensus (#604)
* Initial chanages for observability

* Fix some panics to start testing

* Export tracing properly

* First example of child spans using opencensus

* Add spans to download protocol

* Add url to traces

* Remove opentracing

* Remove gopkg.* files

* Start deprecating opentracing

* Resolve stupid build errors

* Use observability package

* Fix test errors

* Convert buffalo spans to observercontext

* change package name

* defer flush to the end of the app execution

* Change op names to the correct package

* Rename pkg/observability to pkg/observ

* Show traces for the package

* Keep tracing in the earlier way

* Add info from request headers

* Remove whitespace

* Move exporter url to env var

* Add to env file for documentation

* Remove opentracing stuff

* Use stdlib

* Shorten service name

* Add a service name to olympus as well

* Add test to test if there is a recursion or not

* Add Ops Suite

* Move around code

* Make sure the service is not instantiated if the exporter is not found
2018-09-11 10:31:20 -07:00
Michal Pristas
539700e598 KindNotFound for all storages (#560)
* all exc mongo

* module, version associated with each storage error

* remove confusion

* removed formatted error in favor of raw flavor

* once more
2018-08-24 20:00:00 +02:00
Marwan Sulaiman
6548509896 backend/mongo: use GridFS to stream large zips (#365)
* backend/mongo: use GridFS to stream large zips

* fix tests
2018-07-31 12:27:14 -04:00
Manu Gupta
85c3df5a54 Add tracing functions to s3, Azure, minio, rdbms, mongo (#332)
* Add opentracing to context for azurecdn

* Add opentracing context for s3

* Add opentracing for minio

* Add opentracing functions for mongodb

* Add tracing functions for rdbms

* Fix span name in aws.s3.upload

* add opentracing calls for storage.olympus

* Add opentracing functions to storage.gcp
2018-07-26 17:15:23 -07:00
marpio
58d42a1b57 Stream zips (#200)
* storage.saver interface - use io.reader instead of []byte

* stream zip - backendconnector

* stream zip - azurecdn

* stream zip - fs

* storage - switch to io.Reader

* update callers of Save

* rm empty line

* makezip stream

* refactor mergedb

* add multierror dep

* mergedb multierror

* scope err

* update storages

* return pipereader

* fix test

* use multierror for all strorage drivers

* review feedback

* azure cdn storage - use UploadStreamToBlockBlob

* use numUpload

* use timeout ctx
2018-07-07 08:46:43 -07:00
Martin Strobel
d0a18e7dd4 Add Context to Saver Interface (#179)
* Adding context to storage.Saver parameters

* +`buffalo.Context` as first param in storage.Saver

Fixes #177

* Parallelizing Upload to Azure Blob Storage

Fixes #175

* Fixing import clausing

* go fmt ./...

* Populating default context

* CI: Echo files modified by gofmt check to stderr

Doing so allows users to see which files still need to be formatted.

* Upgrading Go, then go fmt ./...

* Addressing Feedback

* more responding to feedback

* Adding worker save Default Context

* Removing tracing, buffalo.Context usage in Storage interface

github.com/bketelsen/buffet relies on using a `buffalo.Context`
however, often a `storage.Saver` is invoked from a buffalo worker,
which has no buffalo.Context associated with it. Therefore, we had
two options: create a buffalo context for each worker's Job, or
remove the buffalo.Context dependency in the Saver interface.

* Fixing missed changes

* Review feedback
2018-06-21 10:47:31 -07:00
Rob j Loranger
9df207561d Add .info to upload payload (#141)
* Add .info to upload payload
everywhere else as well

+ Upload payload includes Info
+ CLI uploads `version.info` as blob to storage
+ Getter and Saver methods updated for all storages implemented
+ Migration updated to include Info column
+ Tests no longer check RevInfo, now checking for Info same as Zip
+ Updated docs

* newline in .fizz

* change RevInfo.Info to []byte

+ change to []byte instead of io.Reader
+ adjust tests to check for info byte contents directly
+ add error check for minio client GetObject `pkg/storage/minio/getter.go`
2018-05-29 08:03:01 +02:00
marpio
0a3eb4c4ea fix golint issues (#110)
* fix stutters

* add comments

* add -set_exit_status flag to golint

* missing renames

* renames again
2018-03-26 11:18:18 -07:00
Aaron Schlesinger
a6721f7505 Remove the base URL (#60)
* Removing concept of a baseURL

Just using a single ‘module’ and ‘version’ concept is enough

* go build => buffalo build
2018-03-21 16:03:44 -07:00
Michal Pristas
0a6e4407a1 Implemented backing mongo storage for modules (#64)
* mongo storage initial impl

* added tests

* mongo tests passing
2018-03-20 11:28:42 -07:00