Commit Graph

161 Commits

Author SHA1 Message Date
Rohan Chakravarthy
7c745fb3d9 add a configuration file (#453)
* complete updated config package

* use envconfig+toml instead of viper. Add descriptions in example config file

* add unit tests

* debug gofmt on build server

* force dummy commit

* skip gofmt to validate other tests are passing

* unset env vars for example file parsing test

* cleanup tests

* test improvements

* re-enable gofmt

* naming

* PR comments

* fix failing test after olympus default endpoint change

* remove rdbms config

* set defaults in code

* add support for proxyfilteroff

* add basic auth params

* update gopkg.lock

* undo gopkg.lock changes made during merge

* remove defaults

* explicitly specify all env variables

* remove rdbms from example

* remove user and pass to disable basic auth by default

* switch to memory by default for the proxy

* fix tests after config file change
2018-08-31 14:23:41 -04:00
Rohan Chakravarthy
697a7a5073 remove olympus storage implementation and proxy worker (#513)
* remove olympus storage implementation and proxy worker

* cleanup after merge conflicts

* remove newline
2018-08-30 03:47:04 -04:00
Arthur Mello
e5ce3f7c6e Set a default value for ATHENS_MONGO_CONNECTION_STRING env var (#580)
* Set a default value for ATHENS_MONGO_CONNECTION_STRING env var

- ATHENS_MONGO_CONNECTION_STRING will default to mongodb://127.0.0.1:27017 if not set

* Fix Get function call

* Update user of env.MongoConnectionString
2018-08-28 11:56:15 +02:00
Michal Pristas
539700e598 KindNotFound for all storages (#560)
* all exc mongo

* module, version associated with each storage error

* remove confusion

* removed formatted error in favor of raw flavor

* once more
2018-08-24 20:00:00 +02:00
Marwan Sulaiman
4a618990a2 pkg/download/goget: ensure each List func gets its own GOPATH (#559)
* pkg/download/goget: ensure each List func gets its own GOPATH

* Add list tests

* fix diskref tests
2018-08-24 10:19:17 -04:00
Marwan Sulaiman
fc581c3784 pkg/storage: ensure mongo returns KindNotFound (#558) 2018-08-24 02:13:09 -04:00
marpio
76121fb5cb Fix running the athens proxy on windows (#554)
* use parent env

* rename

* use only required vars

* review feedback

* add windows env vars

* ups

* fix

* rm op
2018-08-24 01:55:55 -04:00
mydiemho
e4659ecea5 create in memory driver for cdn metadata (#498)
* create in memory driver for cdn metadata

fixes #334

* save memStorage so we're not recreating it everytime

* run gofmt
2018-08-22 23:06:09 +02:00
Andrey Dyachkov
062f24d32f Issue 427: error check for storage exists method (#458)
* The commit brings error check for storage exists

The commit opened against ticket #427, and adds error response to
checker exists method and all corresponding places. The major changes related to renaming and processing of additional errors, because the signature of Checker.Exists was changed to return error in response.

* improved error processing

* go fmt'ed files

* added check to minio storage for no key error

* moved our constant + changed check for err

* go fmt'ed

* deleted rdbms packege, which brought merge with upsteam

* process error from minio checker
2018-08-22 15:49:36 -04:00
Manu Gupta
3a8c6429e9 Removal of cdn rdbms driver and pop support from olympus (#551)
* Inital removal of cdn rdbms driver and pop support from olympus

* Update travis and other unit test scripts

* Remove postgres from Makefile

* Update comments and remove function that is not required

* Update .env file

* clean commented code

* Update docs to remove references to rdbms

* Remove pop and run dep ensure to make sure that the tests pass without pop

* Keeping GO_ENV=test for parametter logging
2018-08-22 20:10:19 +02:00
Marwan Sulaiman
a1b5060ead Proxy: add cache-control middleware (#553)
* Proxy: add cache-control middleware

* fix improt cycle + use MDN no-cache example

* remove import lines

* actually use MDN example

* remove .idea from gitignore in unrelated pr
2018-08-22 09:52:16 -04:00
Manu Gupta
c0635b748c Remove pop from Proxy & RDBMS storage getter (#511)
* Remove pop from models for now

* Remove rdbms storage from pkg/storage

* Completely remove pop

* Some improvements to tests

- Not sourcing from cmd/proxy/.env, because that doesn't export any variables
- Removing mysql support (I feel like 1 database is enough)
- Pruning networks on teardown

I'm happy to split these changes up into separate PRs - it's the end of the day for me so I wanted to get everything in 😄

* update to point to test instance

* Update databasy.yml for travis??

* Remove database.yml

* Do not use db migrations and creation

* Remove pop dependency from cdn

* add schema for olympus assuming cdn driver is part of olympus

* Move CI/CD and test scripts to point pop to olympus

* Update database.yml and keep it only for olympus

* Add migrations to travis

* Add debug logs to see what travis is doing

* Update travis to use olympus database.yml

* Moarrrr logs to see where travis is pointint the db towards

* Change env to test

* Use test as the default environment if not specified otherwise

* Check if test new rdbms storage succeeds

* Try to see which connection string rdbms test suite is using on travis

* Update db tags for cdn driver

* create db by using travis go_env

* Remove sql from database config as it is removed from docker-compose

* Remove extra logs that I missed

* Add newline

* remove models.go and test file from proxy

* removing extraneous network prune from Makefile
2018-08-21 19:21:40 -04:00
Federico Paolinelli
c8b64fd258 Proxy 'admission controller' (#492)
* The proxy middleware calls an hook if set to check if the module needs
to be filtered

* Fixed error handling

* Added a middleware func that passes the log entry down to other
middlewares

* Split the filter middleware in two

* Moved validation middleware in its own file

* Renamed filter to middleware, all the middleware stuff in a single file

* Moved default hook to empty string

* Split tests of filter middleware from validation middleware

* Installed the validation middleware only if the hook is set

* Moved filter test to middleware test

* Fixed gofmt issue caused by manual merge

* Fixed wrong logrus import

* Removed logger from filter middleware

* Moved middleware functions to own package

* Removed unused argument, fixed errors

* Fixed gofmt errors

* Made middleware func private

* Changed debuglevel from string to enum

* Added comments to now public functions

* Changed module alias to a shorter one

* Removed white space

* Split middleware in separate files, injected the hook as param

* Removed commented code
2018-08-21 22:51:26 +02:00
Michal Pristas
042cbcaddf SSL Support for Mongo (#527)
* tested mongo ssl

* Added a todo
2018-08-21 18:09:49 +02:00
Marwan Sulaiman
87c92dfe28 Proxy: add NETRC file mount (#528) 2018-08-20 13:58:28 -04:00
Marwan Sulaiman
622aba0fe1 Athens: disable buffalo logger in production (#516)
* Athens: disable buffalo logger in production

* fix gofmt

* pkg/log: pass down the level instead of environment

* pkg/log: remove nop writer

* pkg/log: use indepnedent buffalo log level
2018-08-20 13:23:27 -04:00
Marwan Sulaiman
d749465e35 pkg/download: use worker pool to limit concurrent gogets (#524)
* pkg/download: use worker pool to limit concurrent gogets

* fix test closure

* use workers env
2018-08-20 11:30:12 -04:00
marpio
a10d98d6e7 set pkg level memStorage var so we get the same instance every time (#507) 2018-08-20 09:32:38 +02:00
Marwan Sulaiman
db5eb0ef69 proxy: add path prefix (#520) 2018-08-20 09:24:51 +02:00
mydiemho
c4bf6a13eb refactor SetPointer return base on golint suggestion (#522)
fixes #521
2018-08-20 09:04:28 +02:00
Marwan Sulaiman
4c55c813d0 Proxy: make Force SSL configurable (#515)
* Proxy: make Force SSL configurable

* fix godoc
2018-08-19 15:57:55 -04:00
Marwan Sulaiman
70167dc486 Proxy: use a global switch for Filter Middleware (#501)
* Proxy: do not redirect on filter.Include

* Filter: add Off opt

* Filter: set Off to true when there's no config

* Proxy: use a global switch for filter
2018-08-17 18:06:37 -04:00
Marwan Sulaiman
231cd85728 Proxy: add basic authentication middleware (#500)
* Proxy: add basic authentication middleware

* Proxy: fix ok
2018-08-17 09:01:35 -04:00
kteb
cdab5c515c Remove storage connector (#457)
* get rid of fake storage.Connector

* get rid of the storage.Connector for mango & rdbms

* Remove all the connector interfaces

* Updating all the test to get rid of the connect

* Updating connection to storage for proxy

* Updating connection to storage for olympus

* get rid of fake storage.Connector

* get rid of the storage.Connector for mango & rdbms

* Remove all the connector interfaces

* Updating all the test to get rid of the connect

* Updating connection to storage for proxy

* Updating connection to storage for olympus

* Fix reviews

* Fix reviews

* fixing op Name

* fix replace the check of an error insted of the object exists

* Fix error message in the test
2018-08-15 09:51:10 -07:00
Marwan Sulaiman
889de448dc Fetcher: switch to mod download (#448) 2018-08-15 12:33:00 -04:00
Michal Pristas
d088208941 Updated to go1.11rc1 (#482)
* updated to rc1

* latest test
2018-08-15 12:26:43 -04:00
Marwan Sulaiman
211eb9310c pkg/log: log msg if no data given (#484) 2018-08-15 09:10:54 -04:00
Marwan Sulaiman
78687a7948 pkg/module: fix comment (#485) 2018-08-14 17:03:16 -07:00
Michal Pristas
327fecaedd rename Private to Direct (#481) 2018-08-14 11:13:36 +02:00
Michal Pristas
72a4af135a Removed ShouldProcess (#480)
* removed shouldProcess
* avoid Default as a return value for Rule
2018-08-14 10:28:39 +02:00
Federico Paolinelli
e6d4996bad Added support for private repos (#456)
* Added filter for private repos. Non private repos get redirected to
olympus

* Changed filter to allow private modules

* Changed config file name function

* Added filter tests

* The middleware returns 200 for privates, redirects public, 404 for
disabled repos

* Removed printf

* Removed forgotten printf

* Removed logger from the middleware func, not used anymore

* Removed unused (uncommented) function

* Replaced 404 with 403 in case of banned modules

* Fixed test path / cleaned olympus endpoint suffix

* Added error operation to GetVersion

* Moved filter test in custom file, reverted injection of the filter into
the app

* Removed white line

* Clarified comment for ignoring getversion error

* Added todo comment to fill the cache and serve the request with the
cache

* Added error handling to filter

* Removed commented early version of the test
2018-08-13 16:35:17 -07:00
Marwan Sulaiman
5f02ada292 pkg/module: add errors op (#464) 2018-08-13 11:54:48 -04:00
Marwan Sulaiman
1ea2969432 pkg/storage: add err ops (#465)
* pkg/storage: add err ops

* fix tests

* fix tests
2018-08-13 11:24:58 -04:00
Marwan Sulaiman
62be80408e CDN: include error operations (#463)
* CDN: include error operations

* better op name for cdn/fs
2018-08-13 09:47:02 -04:00
Marwan Sulaiman
ca7fdeae6b pkg/module: remove unused module.Fetch (#459) 2018-08-13 08:43:15 -04:00
Dinesh Kumar
4cc1b71007 Benchmark storages (#426)
* Adding benchmark for mongo and fs

* Adding delete, exists, save with other storages

* Fixing typos, adding nonexistent module exits benchmark

* Fixing delete operation in delete benchmark

* Separating non testing existing modules benchmark, shortening the benchmark name

* running benchmark in ci

* Revert "running benchmark in ci" - mongo index make duplicate records to
fail
idempotent, will fix in a separate PR.

This reverts commit f7d78262d2.

* Adding reset timer, fixing duplicate record issue

* Revert "Revert "running benchmark in ci" - mongo index make duplicate records to"

This reverts commit 40a7b33199.

* removing benchmark from ci

* Adding operation for returning errors
2018-08-10 01:34:42 -04:00
Marwan Sulaiman
95cfa173e2 Storage: split storage.Getter interface (#406)
* Storage: split storage.Getter interface

* no need to clean up whats not there

* unimplement rdmb getter

* unimplement rdmb getter

* make storage tests happy

* comment out rdmbs from tests

* fix tests

* fix stuff we're removing anyway

* PR fixes

* oops

* fix gcp exists
2018-08-08 12:04:02 -04:00
kdvy
7dcbc3d969 In memory workers config (#400)
* Correct env variable referenced in comment for
OlympusRedisQueuePortWithDefault

* Add redis in-mem mock config for proxy

* Add redis in-mem mock config for Registry

* Use in-memory mock for Redis queue if in-mem environment variable set

* Use in-memory mock for redis queue in Olympus if environment variable
set

* Revert "Use in-memory mock for Redis queue if in-mem environment variable set"

This reverts commit 45b52e48f9.

* Revert "Add redis in-mem mock config for proxy"

This reverts commit 3373947fb5.

* Use OLYMPUS_BACKGROUND_WORKER_TYPE to control worker type used in
Olympus

* Use memory background worker type if an unexpected value is provided in
env var.
2018-08-06 17:19:25 -07:00
Marwan Sulaiman
44625459fc Download: move cache location (#404)
* Download: move cache location

* beta3

* trying differnet repo

* avoid go breaking stuff
2018-08-04 11:29:30 -07:00
Marwan Sulaiman
6b766c3f72 Download: introduce sane Entry defaults (#398) 2018-08-04 12:27:24 -04:00
Marwan Sulaiman
ef93e6759d Download: test against controlled repositories (#399) 2018-08-04 01:57:45 -04:00
Marwan Sulaiman
c41e18001b pkg/log: we do have our Error struct now (#394) 2018-08-03 15:16:35 -04:00
Marwan Sulaiman
097fde3957 Download: separate stdout from stdin when running cmd/go (#381) 2018-08-02 12:41:27 -04:00
Marwan Sulaiman
15326af2bb GCP: get credentials implicitly (#380)
* GCP: get credentials implicitly

* update name
2018-08-02 12:37:19 -04:00
Dinesh Kumar
85213f20b3 Goget Fetcher should error out if gobinpath is not valid (#377)
* Goget fetcher should error out if gobinpath is not valid

* Propogating the error from goget initialisation to main, to stop application

* wrapping errors with op, using exec.command(gobin).Run to verify gobin, clean up

* Inlining afero fs to goget call, and new go getfetcher

* Revert "Inlining afero fs to goget call, and new go getfetcher"

This reverts commit ae31fe6a2b.

* Fixing example test
2018-08-02 10:00:37 -04:00
Marwan Sulaiman
90281ccb90 Athens: move storage.NotFound to errors.NotFound (#373)
* Athens: move storage.NotFound to errors.NotFound

* see where things went wrong

* empty means unexpected

* GCP: replace AlreadyExists err

* resolve conflicts

* Errors: change not found name

* fix build
2018-08-01 03:27:16 -04:00
Marwan Sulaiman
441e21d281 Proxy: fill cache on Protocol.Version (#374) 2018-07-31 23:11:49 -07:00
Federico Paolinelli
0cdedadf35 GCP storage tests with mock (#343)
* Replace gpc endpoint with mock backed one for tests

* Removed unnecessary log

* Removed copied file

* Removed todo comment

* Replaced bucketMock initialization with the constructor

* Added url to be checked in tests

* Added a counter to check if close was called the same time of Open /
Write

* Re-initializing the map entry for every write in order to overwrite the
content

* Moving the lock before hitting the map

* Moved the counter decrement inside the critical section

* Removed unnecessary format string
2018-07-31 16:13:49 -07:00
Rohan Chakravarthy
bbe1ce1873 zip.Close() also clears the GOPATH of the underlying diskRef (#367)
* zip.Close() also clears underlying diskRef

* remove unnecessary constructor

* use GoBinPath() in example
2018-07-31 12:38:45 -04:00
Marwan Sulaiman
6548509896 backend/mongo: use GridFS to stream large zips (#365)
* backend/mongo: use GridFS to stream large zips

* fix tests
2018-07-31 12:27:14 -04:00