Commit Graph

101 Commits

Author SHA1 Message Date
Rohan Chakravarthy
697a7a5073 remove olympus storage implementation and proxy worker (#513)
* remove olympus storage implementation and proxy worker

* cleanup after merge conflicts

* remove newline
2018-08-30 03:47:04 -04:00
Arthur Mello
e5ce3f7c6e Set a default value for ATHENS_MONGO_CONNECTION_STRING env var (#580)
* Set a default value for ATHENS_MONGO_CONNECTION_STRING env var

- ATHENS_MONGO_CONNECTION_STRING will default to mongodb://127.0.0.1:27017 if not set

* Fix Get function call

* Update user of env.MongoConnectionString
2018-08-28 11:56:15 +02:00
Komu Wairagu
69a5340c91 update to go1.11 (#563)
* update travis to Go 1.11

* update check_deps ci script
ci for checking dependencies should now check Go module files and use go mod verify

* remove uneeded comment

* set GO111MODULE

* debug GO111MODULE propagation

* rm dep installation

* debug GO111MODULE propagation

* rm debug statements

* use Go point releases in ci

* remove references to Go1.11 RCs

* run ci with GO111MODULE auto and also on

* use matrix
2018-08-28 08:46:12 +02:00
Marwan Sulaiman
9860bd51e7 Proxy: return 200 on / regardless of prefix (#572) 2018-08-26 22:11:04 -04:00
Andrey Dyachkov
062f24d32f Issue 427: error check for storage exists method (#458)
* The commit brings error check for storage exists

The commit opened against ticket #427, and adds error response to
checker exists method and all corresponding places. The major changes related to renaming and processing of additional errors, because the signature of Checker.Exists was changed to return error in response.

* improved error processing

* go fmt'ed files

* added check to minio storage for no key error

* moved our constant + changed check for err

* go fmt'ed

* deleted rdbms packege, which brought merge with upsteam

* process error from minio checker
2018-08-22 15:49:36 -04:00
Manu Gupta
3a8c6429e9 Removal of cdn rdbms driver and pop support from olympus (#551)
* Inital removal of cdn rdbms driver and pop support from olympus

* Update travis and other unit test scripts

* Remove postgres from Makefile

* Update comments and remove function that is not required

* Update .env file

* clean commented code

* Update docs to remove references to rdbms

* Remove pop and run dep ensure to make sure that the tests pass without pop

* Keeping GO_ENV=test for parametter logging
2018-08-22 20:10:19 +02:00
Manu Gupta
c0635b748c Remove pop from Proxy & RDBMS storage getter (#511)
* Remove pop from models for now

* Remove rdbms storage from pkg/storage

* Completely remove pop

* Some improvements to tests

- Not sourcing from cmd/proxy/.env, because that doesn't export any variables
- Removing mysql support (I feel like 1 database is enough)
- Pruning networks on teardown

I'm happy to split these changes up into separate PRs - it's the end of the day for me so I wanted to get everything in 😄

* update to point to test instance

* Update databasy.yml for travis??

* Remove database.yml

* Do not use db migrations and creation

* Remove pop dependency from cdn

* add schema for olympus assuming cdn driver is part of olympus

* Move CI/CD and test scripts to point pop to olympus

* Update database.yml and keep it only for olympus

* Add migrations to travis

* Add debug logs to see what travis is doing

* Update travis to use olympus database.yml

* Moarrrr logs to see where travis is pointint the db towards

* Change env to test

* Use test as the default environment if not specified otherwise

* Check if test new rdbms storage succeeds

* Try to see which connection string rdbms test suite is using on travis

* Update db tags for cdn driver

* create db by using travis go_env

* Remove sql from database config as it is removed from docker-compose

* Remove extra logs that I missed

* Add newline

* remove models.go and test file from proxy

* removing extraneous network prune from Makefile
2018-08-21 19:21:40 -04:00
Federico Paolinelli
c8b64fd258 Proxy 'admission controller' (#492)
* The proxy middleware calls an hook if set to check if the module needs
to be filtered

* Fixed error handling

* Added a middleware func that passes the log entry down to other
middlewares

* Split the filter middleware in two

* Moved validation middleware in its own file

* Renamed filter to middleware, all the middleware stuff in a single file

* Moved default hook to empty string

* Split tests of filter middleware from validation middleware

* Installed the validation middleware only if the hook is set

* Moved filter test to middleware test

* Fixed gofmt issue caused by manual merge

* Fixed wrong logrus import

* Removed logger from filter middleware

* Moved middleware functions to own package

* Removed unused argument, fixed errors

* Fixed gofmt errors

* Made middleware func private

* Changed debuglevel from string to enum

* Added comments to now public functions

* Changed module alias to a shorter one

* Removed white space

* Split middleware in separate files, injected the hook as param

* Removed commented code
2018-08-21 22:51:26 +02:00
Michal Pristas
042cbcaddf SSL Support for Mongo (#527)
* tested mongo ssl

* Added a todo
2018-08-21 18:09:49 +02:00
Marwan Sulaiman
8554f91286 Proxy: MVP Dockerfile (#536)
* Proxy: MVP Dockerfile

* remove comments

* add new line
2018-08-21 10:37:17 -04:00
Marwan Sulaiman
87c92dfe28 Proxy: add NETRC file mount (#528) 2018-08-20 13:58:28 -04:00
Marwan Sulaiman
622aba0fe1 Athens: disable buffalo logger in production (#516)
* Athens: disable buffalo logger in production

* fix gofmt

* pkg/log: pass down the level instead of environment

* pkg/log: remove nop writer

* pkg/log: use indepnedent buffalo log level
2018-08-20 13:23:27 -04:00
Marwan Sulaiman
d749465e35 pkg/download: use worker pool to limit concurrent gogets (#524)
* pkg/download: use worker pool to limit concurrent gogets

* fix test closure

* use workers env
2018-08-20 11:30:12 -04:00
Marwan Sulaiman
db5eb0ef69 proxy: add path prefix (#520) 2018-08-20 09:24:51 +02:00
Marwan Sulaiman
493945c13a Athens: add health check endpoint (#514) 2018-08-19 16:34:46 -04:00
Marwan Sulaiman
4c55c813d0 Proxy: make Force SSL configurable (#515)
* Proxy: make Force SSL configurable

* fix godoc
2018-08-19 15:57:55 -04:00
Marwan Sulaiman
1ac2cf1317 Proxy: remove unused arguments (#512)
* Proxy: remove unused arguments

* remove debugging info
2018-08-19 15:17:08 -04:00
Marwan Sulaiman
70167dc486 Proxy: use a global switch for Filter Middleware (#501)
* Proxy: do not redirect on filter.Include

* Filter: add Off opt

* Filter: set Off to true when there's no config

* Proxy: use a global switch for filter
2018-08-17 18:06:37 -04:00
Marwan Sulaiman
231cd85728 Proxy: add basic authentication middleware (#500)
* Proxy: add basic authentication middleware

* Proxy: fix ok
2018-08-17 09:01:35 -04:00
kteb
cdab5c515c Remove storage connector (#457)
* get rid of fake storage.Connector

* get rid of the storage.Connector for mango & rdbms

* Remove all the connector interfaces

* Updating all the test to get rid of the connect

* Updating connection to storage for proxy

* Updating connection to storage for olympus

* get rid of fake storage.Connector

* get rid of the storage.Connector for mango & rdbms

* Remove all the connector interfaces

* Updating all the test to get rid of the connect

* Updating connection to storage for proxy

* Updating connection to storage for olympus

* Fix reviews

* Fix reviews

* fixing op Name

* fix replace the check of an error insted of the object exists

* Fix error message in the test
2018-08-15 09:51:10 -07:00
Marwan Sulaiman
feb17a7a61 Athens: turn off worker to avoid nil logs (#483) 2018-08-14 21:43:39 -04:00
Carolyn Van Slyck
afd01813e9 Default to in-memory storage for the proxy (#475)
* Default to in-memory storage for the proxy

This makes it easier for someone to try out or just run the proxy by
executing the binary without further environment setup.

* Remove obsolete comment
2018-08-14 17:54:26 -04:00
Michal Pristas
327fecaedd rename Private to Direct (#481) 2018-08-14 11:13:36 +02:00
Federico Paolinelli
e6d4996bad Added support for private repos (#456)
* Added filter for private repos. Non private repos get redirected to
olympus

* Changed filter to allow private modules

* Changed config file name function

* Added filter tests

* The middleware returns 200 for privates, redirects public, 404 for
disabled repos

* Removed printf

* Removed forgotten printf

* Removed logger from the middleware func, not used anymore

* Removed unused (uncommented) function

* Replaced 404 with 403 in case of banned modules

* Fixed test path / cleaned olympus endpoint suffix

* Added error operation to GetVersion

* Moved filter test in custom file, reverted injection of the filter into
the app

* Removed white line

* Clarified comment for ignoring getversion error

* Added todo comment to fill the cache and serve the request with the
cache

* Added error handling to filter

* Removed commented early version of the test
2018-08-13 16:35:17 -07:00
Martin Strobel
106d8c3631 Cleanup Dockerfiles (#437)
* Cleanup Dockerfiles

* Responding to review feedback

* Preventing host copy of local  and  directories from being copied into builder containers.
2018-08-13 15:59:41 -07:00
Michal Pristas
0ded8098ab removed unused cache miss reporter (#461) 2018-08-13 15:01:24 +02:00
Andrey Dyachkov
051f3884e1 Bring more clarity on how to run unit tests (#449)
* The commit brings more clarity on how to run unut tests.

The commit chages behaivour of `make test-unit` by adding sourcing of
env vars. Documentation is updated to provide more clarity on how to run
unit tests.

* changed list of commands to run before unit tests

* moved env vars to .env

* removed dduplicated env var

* returned conventional file name in script

* link to DEVELOPMENT.md

* fixed typo

* moved env var export to script

* removed export of tmp folder
2018-08-11 12:12:24 -07:00
Marwan Sulaiman
95cfa173e2 Storage: split storage.Getter interface (#406)
* Storage: split storage.Getter interface

* no need to clean up whats not there

* unimplement rdmb getter

* unimplement rdmb getter

* make storage tests happy

* comment out rdmbs from tests

* fix tests

* fix stuff we're removing anyway

* PR fixes

* oops

* fix gcp exists
2018-08-08 12:04:02 -04:00
Manu Gupta
23639c74d5 Add logo to proxy (#422)
* Add logo to proxy

* Add newline to css
2018-08-08 08:46:20 -07:00
kdvy
7dcbc3d969 In memory workers config (#400)
* Correct env variable referenced in comment for
OlympusRedisQueuePortWithDefault

* Add redis in-mem mock config for proxy

* Add redis in-mem mock config for Registry

* Use in-memory mock for Redis queue if in-mem environment variable set

* Use in-memory mock for redis queue in Olympus if environment variable
set

* Revert "Use in-memory mock for Redis queue if in-mem environment variable set"

This reverts commit 45b52e48f9.

* Revert "Add redis in-mem mock config for proxy"

This reverts commit 3373947fb5.

* Use OLYMPUS_BACKGROUND_WORKER_TYPE to control worker type used in
Olympus

* Use memory background worker type if an unexpected value is provided in
env var.
2018-08-06 17:19:25 -07:00
Marwan Sulaiman
6b766c3f72 Download: introduce sane Entry defaults (#398) 2018-08-04 12:27:24 -04:00
Manu Gupta
7aafc2692e Change service names to specify olympus / proxy (#384) 2018-08-03 07:54:45 -07:00
Marwan Sulaiman
15326af2bb GCP: get credentials implicitly (#380)
* GCP: get credentials implicitly

* update name
2018-08-02 12:37:19 -04:00
Dinesh Kumar
85213f20b3 Goget Fetcher should error out if gobinpath is not valid (#377)
* Goget fetcher should error out if gobinpath is not valid

* Propogating the error from goget initialisation to main, to stop application

* wrapping errors with op, using exec.command(gobin).Run to verify gobin, clean up

* Inlining afero fs to goget call, and new go getfetcher

* Revert "Inlining afero fs to goget call, and new go getfetcher"

This reverts commit ae31fe6a2b.

* Fixing example test
2018-08-02 10:00:37 -04:00
Komu Wairagu
af2ecfddae rm env.GoPath (#370)
1. Gopath func doesnt seem to be used much; it is used in init func of proxy/actions/app.go which doesnt do much
2. I think relying on $GOPATH post go1.11 will lead to flaky situations
2018-07-31 10:14:20 -04:00
Aaron Schlesinger
59d5d5ffac Cleaning up unused code (#352)
* removing pkg/repo

Fixes https://github.com/gomods/athens/issues/335

* removing the gomod parser
2018-07-30 14:06:17 -07:00
Manu Gupta
afa831472d Implement context threads to be passed to storage.*.getter (#346)
* Implement context threads to be passed to storage.*.getter

* Create a background context thread on athens.proxy
2018-07-30 20:11:52 +02:00
Chad Kunde
53df0f7c6a clean up app initialization if blocks (#347)
Remove global singleton.  Create and initialize a new buffalo app on
each invocation of the constructor.
2018-07-29 22:01:30 -07:00
Marwan Sulaiman
add15e68cd download: clean up Kahona (#345) 2018-07-27 15:18:14 -07:00
Carolyn Van Slyck
fd7c1326d3 Backfill the cache on a miss (#342)
* add middlewares to fill the cache and to populate contexts with module and versions

* Carolyn tries to fix Aaron's code

* #DOINSTUFF

* updated env var

* fix all the things

* magic patch for olympus

* Add latest handler

* Remove deprecated return param

* go fmt all the things
2018-07-27 14:37:23 -07:00
Marwan Sulaiman
7b590b1885 download: add list from go cli (#336)
* download: add list from go cli

* download: include goget tests + hacky hack

* download: move dummyMod to pkg/module

* Olympus: pass dp and lggr to /list

* download: add Version to interface

* download: document Protocol
2018-07-27 11:56:32 -07:00
Marwan Sulaiman
0853f073f2 Athens: introduce pkg/errors (#320)
* Athens: introduce pkg/errors

* errors: fix Ops loop

* pkg/errors: introduce M and V types to get rid of unstable parsing

* fix build

* errors: fix text + add tests

* pr updates
2018-07-25 13:18:58 -05:00
Tyler Bui-Palsulich
27bbc3f42f storage: add context to storage.Exists and storage.Delete (#322)
* storage: add context to storage.Exists and storage.Delete

* eventlog: use context.TODO()

* storage/gcp: reuse ctx
2018-07-24 17:12:31 -05:00
Aaron Schlesinger
4c54f02a0d removing the home handler on olympus (#302)
* removing the home handler on olympus

* Removing the index.html
2018-07-20 12:28:48 -07:00
Komu Wairagu
d74e0640fe remove dead code (#267)
* remove spf13/cobra and inconshreveable/mousetrap

* remove user package

* remove user package

* remove unused types, vars etc
found by using github.com/dominikh/go-tools/tree/master/cmd/unused.running  unused $(go list github.com/gomods/athens/... | grep -v /vendor/)

* rm uneeded file
2018-07-19 08:34:16 -07:00
marpio
9913736bc7 Download module version from CDN (#283)
* add download function which downloads a ver from cdn

* use cdn download func

* save after rename

* review feedback

* just to trigger a build

* pass timeout and move to pkg/module

* use multierror

* rm empty line
2018-07-18 15:31:30 -07:00
Marwan Sulaiman
eb0768f3fe Proxy: fix init nil dereference (#282) 2018-07-16 15:21:30 -04:00
Marwan Sulaiman
084e906804 Athens: remove c.Error (#278)
* Athens: remove c.Error

* Download: return err string on VersionModuleHandler
2018-07-15 19:40:45 -07:00
Komu Wairagu
94bbcd1996 update redigo (#266)
* dep ensure -v -update github.com/gocraft/work

* rm -rf vendor/github.com/garyburd

* sed /garyburd/redigo / gomodule/redigo

* dont constrain redigo version

* update redigo ver

* rm deps

* update deps

* add vendor
2018-07-15 10:37:29 -07:00
Rob j Loranger
1aaacc7b59 Zeus: remove duplicate '/' handler assignment (#272)
* remove duplicate '/' handler declaration

* Remove homeHandler() and slow()

homeHandler is unused now and slow was a helper to add tracing with a
millisecond delay to the handler
2018-07-15 08:20:32 -07:00