Commit Graph

137 Commits

Author SHA1 Message Date
Marwan Sulaiman
889de448dc Fetcher: switch to mod download (#448) 2018-08-15 12:33:00 -04:00
Michal Pristas
d088208941 Updated to go1.11rc1 (#482)
* updated to rc1

* latest test
2018-08-15 12:26:43 -04:00
Marwan Sulaiman
211eb9310c pkg/log: log msg if no data given (#484) 2018-08-15 09:10:54 -04:00
Marwan Sulaiman
78687a7948 pkg/module: fix comment (#485) 2018-08-14 17:03:16 -07:00
Michal Pristas
327fecaedd rename Private to Direct (#481) 2018-08-14 11:13:36 +02:00
Michal Pristas
72a4af135a Removed ShouldProcess (#480)
* removed shouldProcess
* avoid Default as a return value for Rule
2018-08-14 10:28:39 +02:00
Federico Paolinelli
e6d4996bad Added support for private repos (#456)
* Added filter for private repos. Non private repos get redirected to
olympus

* Changed filter to allow private modules

* Changed config file name function

* Added filter tests

* The middleware returns 200 for privates, redirects public, 404 for
disabled repos

* Removed printf

* Removed forgotten printf

* Removed logger from the middleware func, not used anymore

* Removed unused (uncommented) function

* Replaced 404 with 403 in case of banned modules

* Fixed test path / cleaned olympus endpoint suffix

* Added error operation to GetVersion

* Moved filter test in custom file, reverted injection of the filter into
the app

* Removed white line

* Clarified comment for ignoring getversion error

* Added todo comment to fill the cache and serve the request with the
cache

* Added error handling to filter

* Removed commented early version of the test
2018-08-13 16:35:17 -07:00
Marwan Sulaiman
5f02ada292 pkg/module: add errors op (#464) 2018-08-13 11:54:48 -04:00
Marwan Sulaiman
1ea2969432 pkg/storage: add err ops (#465)
* pkg/storage: add err ops

* fix tests

* fix tests
2018-08-13 11:24:58 -04:00
Marwan Sulaiman
62be80408e CDN: include error operations (#463)
* CDN: include error operations

* better op name for cdn/fs
2018-08-13 09:47:02 -04:00
Marwan Sulaiman
ca7fdeae6b pkg/module: remove unused module.Fetch (#459) 2018-08-13 08:43:15 -04:00
Dinesh Kumar
4cc1b71007 Benchmark storages (#426)
* Adding benchmark for mongo and fs

* Adding delete, exists, save with other storages

* Fixing typos, adding nonexistent module exits benchmark

* Fixing delete operation in delete benchmark

* Separating non testing existing modules benchmark, shortening the benchmark name

* running benchmark in ci

* Revert "running benchmark in ci" - mongo index make duplicate records to
fail
idempotent, will fix in a separate PR.

This reverts commit f7d78262d2.

* Adding reset timer, fixing duplicate record issue

* Revert "Revert "running benchmark in ci" - mongo index make duplicate records to"

This reverts commit 40a7b33199.

* removing benchmark from ci

* Adding operation for returning errors
2018-08-10 01:34:42 -04:00
Marwan Sulaiman
95cfa173e2 Storage: split storage.Getter interface (#406)
* Storage: split storage.Getter interface

* no need to clean up whats not there

* unimplement rdmb getter

* unimplement rdmb getter

* make storage tests happy

* comment out rdmbs from tests

* fix tests

* fix stuff we're removing anyway

* PR fixes

* oops

* fix gcp exists
2018-08-08 12:04:02 -04:00
kdvy
7dcbc3d969 In memory workers config (#400)
* Correct env variable referenced in comment for
OlympusRedisQueuePortWithDefault

* Add redis in-mem mock config for proxy

* Add redis in-mem mock config for Registry

* Use in-memory mock for Redis queue if in-mem environment variable set

* Use in-memory mock for redis queue in Olympus if environment variable
set

* Revert "Use in-memory mock for Redis queue if in-mem environment variable set"

This reverts commit 45b52e48f9.

* Revert "Add redis in-mem mock config for proxy"

This reverts commit 3373947fb5.

* Use OLYMPUS_BACKGROUND_WORKER_TYPE to control worker type used in
Olympus

* Use memory background worker type if an unexpected value is provided in
env var.
2018-08-06 17:19:25 -07:00
Marwan Sulaiman
44625459fc Download: move cache location (#404)
* Download: move cache location

* beta3

* trying differnet repo

* avoid go breaking stuff
2018-08-04 11:29:30 -07:00
Marwan Sulaiman
6b766c3f72 Download: introduce sane Entry defaults (#398) 2018-08-04 12:27:24 -04:00
Marwan Sulaiman
ef93e6759d Download: test against controlled repositories (#399) 2018-08-04 01:57:45 -04:00
Marwan Sulaiman
c41e18001b pkg/log: we do have our Error struct now (#394) 2018-08-03 15:16:35 -04:00
Marwan Sulaiman
097fde3957 Download: separate stdout from stdin when running cmd/go (#381) 2018-08-02 12:41:27 -04:00
Marwan Sulaiman
15326af2bb GCP: get credentials implicitly (#380)
* GCP: get credentials implicitly

* update name
2018-08-02 12:37:19 -04:00
Dinesh Kumar
85213f20b3 Goget Fetcher should error out if gobinpath is not valid (#377)
* Goget fetcher should error out if gobinpath is not valid

* Propogating the error from goget initialisation to main, to stop application

* wrapping errors with op, using exec.command(gobin).Run to verify gobin, clean up

* Inlining afero fs to goget call, and new go getfetcher

* Revert "Inlining afero fs to goget call, and new go getfetcher"

This reverts commit ae31fe6a2b.

* Fixing example test
2018-08-02 10:00:37 -04:00
Marwan Sulaiman
90281ccb90 Athens: move storage.NotFound to errors.NotFound (#373)
* Athens: move storage.NotFound to errors.NotFound

* see where things went wrong

* empty means unexpected

* GCP: replace AlreadyExists err

* resolve conflicts

* Errors: change not found name

* fix build
2018-08-01 03:27:16 -04:00
Marwan Sulaiman
441e21d281 Proxy: fill cache on Protocol.Version (#374) 2018-07-31 23:11:49 -07:00
Federico Paolinelli
0cdedadf35 GCP storage tests with mock (#343)
* Replace gpc endpoint with mock backed one for tests

* Removed unnecessary log

* Removed copied file

* Removed todo comment

* Replaced bucketMock initialization with the constructor

* Added url to be checked in tests

* Added a counter to check if close was called the same time of Open /
Write

* Re-initializing the map entry for every write in order to overwrite the
content

* Moving the lock before hitting the map

* Moved the counter decrement inside the critical section

* Removed unnecessary format string
2018-07-31 16:13:49 -07:00
Rohan Chakravarthy
bbe1ce1873 zip.Close() also clears the GOPATH of the underlying diskRef (#367)
* zip.Close() also clears underlying diskRef

* remove unnecessary constructor

* use GoBinPath() in example
2018-07-31 12:38:45 -04:00
Marwan Sulaiman
6548509896 backend/mongo: use GridFS to stream large zips (#365)
* backend/mongo: use GridFS to stream large zips

* fix tests
2018-07-31 12:27:14 -04:00
Komu Wairagu
af2ecfddae rm env.GoPath (#370)
1. Gopath func doesnt seem to be used much; it is used in init func of proxy/actions/app.go which doesnt do much
2. I think relying on $GOPATH post go1.11 will lead to flaky situations
2018-07-31 10:14:20 -04:00
Marwan Sulaiman
43cb601718 athens: encode/decode modules per cmd/go rules (#366) 2018-07-31 03:29:23 -04:00
Marwan Sulaiman
cd377f40c5 download: fix handler errors (#364) 2018-07-31 02:30:31 -04:00
Rohan Chakravarthy
c7d0c0a8e6 Ensure diskRef cleans up everything in it's own GOPATH (#354)
* Ensure diskRef cleans up everything in it's own GOPATH

* cleanup diskref opens. Fix err return in go_get_fetcher

* Remove Noop ref from go_get_fetcher.Fetch()

* handle temp dir cleanup and return nil diskref
2018-07-30 19:14:06 -04:00
Aaron Schlesinger
59d5d5ffac Cleaning up unused code (#352)
* removing pkg/repo

Fixes https://github.com/gomods/athens/issues/335

* removing the gomod parser
2018-07-30 14:06:17 -07:00
Manu Gupta
afa831472d Implement context threads to be passed to storage.*.getter (#346)
* Implement context threads to be passed to storage.*.getter

* Create a background context thread on athens.proxy
2018-07-30 20:11:52 +02:00
Aaron Schlesinger
f1dfba9d40 Finish spans in the download handlers (#338)
* finishing spans inside the download handlers

* finishing another span
2018-07-28 16:06:49 -07:00
Carolyn Van Slyck
fea1e94552 Include errors in returned error (#344) 2018-07-27 15:24:30 -07:00
Marwan Sulaiman
add15e68cd download: clean up Kahona (#345) 2018-07-27 15:18:14 -07:00
Carolyn Van Slyck
fd7c1326d3 Backfill the cache on a miss (#342)
* add middlewares to fill the cache and to populate contexts with module and versions

* Carolyn tries to fix Aaron's code

* #DOINSTUFF

* updated env var

* fix all the things

* magic patch for olympus

* Add latest handler

* Remove deprecated return param

* go fmt all the things
2018-07-27 14:37:23 -07:00
Marwan Sulaiman
7b590b1885 download: add list from go cli (#336)
* download: add list from go cli

* download: include goget tests + hacky hack

* download: move dummyMod to pkg/module

* Olympus: pass dp and lggr to /list

* download: add Version to interface

* download: document Protocol
2018-07-27 11:56:32 -07:00
Manu Gupta
85c3df5a54 Add tracing functions to s3, Azure, minio, rdbms, mongo (#332)
* Add opentracing to context for azurecdn

* Add opentracing context for s3

* Add opentracing for minio

* Add opentracing functions for mongodb

* Add tracing functions for rdbms

* Fix span name in aws.s3.upload

* add opentracing calls for storage.olympus

* Add opentracing functions to storage.gcp
2018-07-26 17:15:23 -07:00
Aaron Schlesinger
ab9a4c0f41 Moving the fetch logic into 'pkg/module' (#291)
* add module fetch and fetchers

* Breaking module ref out of the fetcher

* disk ref in progress

* very broken go get fetcher

* refactoring ref and fixing up the disk ref

* adding a test for the disk ref

* doing test setup right

* passing version

* fixing syntax err

* removing raw format string

in favor of a function that does the same thing

* removing obsolete godoc

* removing the switch in favor of nested if statements

hopefully this is easier to read

* passing data into the format strings

* refactoring the closer

* creating and looking for files in the root

* genericFetcher => goGetFetcher

* remove the clear method on go get fetcher

the ref returned from fetch does it

* remove unnecessary call to isVgoInstalled

calls to fetch will fail if vgo is not installed

* remove unneeded dirName

* adding preliminary test for the go get fetcher

* adding a test for fetch

* executing fetch test

* adding docs and simplifying the fetcher

* declaring the storage version up front

* fixing the source path

* creating test files in a separate function

* closing files immediately

instead of aggregating them for later. whaaat was I thinking???

* passing context into storage driver

* not closing the source file

* not putting the module name in quotes anymore

* adding a warning about deleting the package

* adding go binary name

* embedding github API limit error

* check files in the success path

* passing errors up

not just messages

* doh

* using vgo for the go get fetcher tests

* not using multierror because it's unneeded

* less code 😄

* custom error

* removing unused import

* return noop refs

* adding link to issue for the TODO

* simplifying

* simplifying moar!

* using config.FmtModVer

* simplifying

* defer closing the zip

* closing the zip

* adding comment about closing the ver.Zip

* GO_BINARY_NAME => GO_BINARY_PATH

* using errors pkg

* fixing compile err

* removing redundant args

* removing requirement that caller calls Clear on fetch error

* fixing build

* fix

* adding masterminds semver

* returning better error

and checking version format

* removing unused error

* using proper semver function

* fixin compile err

* rm that shizz
2018-07-26 13:08:07 -07:00
Manu Gupta
dc64b0d4e2 Add opentracing for storage.fs. exists, delete, list, save, get (#326)
* Add opentracing for storage.fs. exists, delete, list, save

* Add opentracing for fs.Get

* Rename trace operation names to include pkg name

* Add tracing with a TODO context for fs.Get
2018-07-25 15:26:54 -07:00
Marwan Sulaiman
0853f073f2 Athens: introduce pkg/errors (#320)
* Athens: introduce pkg/errors

* errors: fix Ops loop

* pkg/errors: introduce M and V types to get rid of unstable parsing

* fix build

* errors: fix text + add tests

* pr updates
2018-07-25 13:18:58 -05:00
Tyler Bui-Palsulich
27bbc3f42f storage: add context to storage.Exists and storage.Delete (#322)
* storage: add context to storage.Exists and storage.Delete

* eventlog: use context.TODO()

* storage/gcp: reuse ctx
2018-07-24 17:12:31 -05:00
Tyler Bui-Palsulich
484b020556 storage: add context to storage.List (#321) 2018-07-23 20:25:38 -07:00
marpio
0b9afcd220 CDN parallel uploads (#264)
* common code for the cdn uploads

* fix and use in aws s3

* cleanup and review feedback

* rename const so it makes sense for other uses

* add contenttype to s3

* parallel deletes

* common code for the cdn uploads

* fix and use in aws s3

* cleanup and review feedback

* rename const so it makes sense for other uses

* add contenttype to s3

* parallel deletes

* convert gcp

* use ctx.Err()

* fix upload and del

* tests

* use common code in azure

* refactor gcp bucket

* use delete in gcp

* fix comments

* review feedback

* change pkg alias
2018-07-20 11:15:43 -07:00
Komu Wairagu
d74e0640fe remove dead code (#267)
* remove spf13/cobra and inconshreveable/mousetrap

* remove user package

* remove user package

* remove unused types, vars etc
found by using github.com/dominikh/go-tools/tree/master/cmd/unused.running  unused $(go list github.com/gomods/athens/... | grep -v /vendor/)

* rm uneeded file
2018-07-19 08:34:16 -07:00
Tyler Bui-Palsulich
3c121582e0 storage: move Deleter tests to test suite (#305) 2018-07-19 07:59:34 -07:00
marpio
9913736bc7 Download module version from CDN (#283)
* add download function which downloads a ver from cdn

* use cdn download func

* save after rename

* review feedback

* just to trigger a build

* pass timeout and move to pkg/module

* use multierror

* rm empty line
2018-07-18 15:31:30 -07:00
marpio
4a64bf10a7 extract client for azure strg impl (#293)
* extract client for azure strg impl

* fix func signature
2018-07-18 13:52:43 -07:00
Tyler Bui-Palsulich
cd75b11801 storage: return ErrVersionNotFound from rdbms Deleter (#298)
This also fixes a bug caused by calling conn.Destroy
on a non-pointer value.
2018-07-18 13:47:40 -07:00
Tyler Bui-Palsulich
016b99923c storage: return ErrVersionNotFound from mongo Deleter (#297) 2018-07-18 10:23:58 -07:00