Commit Graph

188 Commits

Author SHA1 Message Date
marpio
31b66be9a2 close zip reader (#724) 2018-10-03 18:21:50 +02:00
Don McNamara
39d10b3a1b Clean up store before test. (#719) 2018-10-03 18:16:04 +02:00
marpio
d370b5a240 ignore line endings in gomod tests (#718) 2018-10-02 13:38:59 +02:00
marpio
bc4e533d49 Use go mod download -json (#710)
* use go mod download -json

* annotate json, fix build

* add comment

* address review feedback
2018-10-01 20:53:17 +02:00
Aravind
7ee25d59d6 Fix typo in comment (#709) 2018-10-01 09:09:29 +02:00
Zachary Romero
e5b307354c Patch points of test flag leakage (#667) 2018-09-29 19:26:19 +02:00
Manu Gupta
2c3d72d52e Save http status in the outermost span (#694)
* Save http status in the outermost span

Co-authored-by: Federico Paolinelli <fedepaol@gmail.com>

* Use ochttp to use span status and store http status code as a tag

* Make the conditional oneliner
2018-09-22 00:28:18 -04:00
marpio
bcc896f226 rename example to dev and rm test config (#691) 2018-09-20 16:38:08 -04:00
marpio
29b6b6ffab Minio conf + tests refactoring (#680)
* cleanup tests and change minio port

* fix cleanup

* cleanup

* fix config test

* add comment to travis

* revert to generic minio addr

* fix test

* switch to test config

* adapt timeouts

* use example config

* fix test... again

* add new lines
2018-09-20 21:31:21 +02:00
Marwan Sulaiman
5fd3e12270 Update OpenCensus to v0.17.0 (#686)
* Update OpenCensus to v0.17.0

* fix typo

* Rename go file to match pkg name
2018-09-20 12:50:32 -04:00
Marwan Sulaiman
fe65b9703f Use config for trace url (#681)
* Use config for trace url

* pr updates

* fix typo

* fix typo
2018-09-20 12:00:13 -04:00
Marwan Sulaiman
5dcaeb8ab8 Trace stash operations (#682)
* Trace stash operations

* fix build
2018-09-20 10:59:42 -04:00
Marwan Sulaiman
8174795715 Fix module.ClearFiles operation name (#685) 2018-09-20 08:57:33 +02:00
Don McNamara
3db7a667f0 Clean up error operation names to match function names. (#614) 2018-09-19 23:43:09 -04:00
Aravind
d4941647af Refactored to remove pkg/cdn (#672) 2018-09-19 08:36:23 +02:00
Marwan Sulaiman
8cfe8b8064 Clean up Config (#673) 2018-09-19 01:44:59 -04:00
Marwan Sulaiman
4b34b1fd28 pkg/errors: add comment to exported function (#676) 2018-09-19 01:38:15 -04:00
tnextday
d4c1363d9c Add HTTP_PROXY environment variable to go_get_fetcher (#668)
* Add HTTP_PROXY env to go_get_fetcher

* gofmt
2018-09-17 20:28:42 +02:00
marpio
6af44242fb Fix list (#607)
* merge list from storage with the one from go list

* fix and rename

* move stuff to semver pkg

* fix gofmt

* move union out of semver pkg

* add tests

* fix err msg

* fix tests

* error handling

* error handling, go list refactoring

* fix list

* cleanup

* fix gofmt
2018-09-13 14:12:17 -07:00
marpio
f25c427239 Fix for Windows 10 (#648)
* ps: add cdefault conf

* init - add config

* add required env vars for win10

* update test_e2e.ps1 to use go build -mod=vendor
2018-09-13 13:57:37 -07:00
Manu Gupta
89d7b35ccd remove opentracing stuff from the code (#652) 2018-09-13 13:24:04 -04:00
Manu Gupta
57b11e8489 Shorten op names and remove spans from handlers (#647) 2018-09-11 23:20:32 -04:00
Rohan Chakravarthy
0e470d0294 Plumb config - Latest (#627)
* switch proxy to config file

pull in single flight changes

* changes for single-flight

* intermediate stage. All tests passing. pkg still has env refs

* remove all env references

* delete config/env entirely

* fix failing tests

* create the config.toml file as part of dev setup

* create config file only if it doesn't exist

* update Dockerfiles to use config file

* move composing elements to the top

* verbose parameter naming

* newline

* add flag for config file path

* update docs with config file flag

* remove unnecessary nil check

* use filepath.join

* rename redis port to address

* fix path.join

* fix issues after merge

* add vendor dir
2018-09-11 15:04:20 -07:00
Manu Gupta
bb250437da Instrumenting observability with opencensus (#604)
* Initial chanages for observability

* Fix some panics to start testing

* Export tracing properly

* First example of child spans using opencensus

* Add spans to download protocol

* Add url to traces

* Remove opentracing

* Remove gopkg.* files

* Start deprecating opentracing

* Resolve stupid build errors

* Use observability package

* Fix test errors

* Convert buffalo spans to observercontext

* change package name

* defer flush to the end of the app execution

* Change op names to the correct package

* Rename pkg/observability to pkg/observ

* Show traces for the package

* Keep tracing in the earlier way

* Add info from request headers

* Remove whitespace

* Move exporter url to env var

* Add to env file for documentation

* Remove opentracing stuff

* Use stdlib

* Shorten service name

* Add a service name to olympus as well

* Add test to test if there is a recursion or not

* Add Ops Suite

* Move around code

* Make sure the service is not instantiated if the exporter is not found
2018-09-11 10:31:20 -07:00
Tobias Klauser
ec2748ecc3 Remove unused file in pkg/module and clean up dependencies (#625)
* pkg/module: Remove unused file parser.go

PR #618 removed the last remaining user of the functions/types defined
in parser.go

* go.mod: update

The last user of github.com/sabhiram/go-gitignore was removed. Run 'go
mod tidy' to remove the dependency.

* vendor: remove unused dependency

The last user of github.com/sabhiram/go-gitignore was removed. Remove
the vendored copy as well by running 'go mod vendor'.
2018-09-07 02:04:23 -04:00
Marwan Sulaiman
ea96158f8f Athens: introduce Single Flight (#573)
* Athens: introduce singleflight

* fix ci

* small fixes

* Remove unnecessary method

* simplify folder structure

* dcoument stasher iface

* fixes

* Fix olympus

* Remove Version from Download Protocol

* fix olympus

* refactor fetcher

* consolodate wrapping styles

* fix olympus

* Further refactoring

* fix typo
2018-09-03 16:31:13 -04:00
Marwan Sulaiman
249a1a6e4e Remove unused stuff (#618) 2018-09-01 14:34:21 -04:00
Rohan Chakravarthy
7c745fb3d9 add a configuration file (#453)
* complete updated config package

* use envconfig+toml instead of viper. Add descriptions in example config file

* add unit tests

* debug gofmt on build server

* force dummy commit

* skip gofmt to validate other tests are passing

* unset env vars for example file parsing test

* cleanup tests

* test improvements

* re-enable gofmt

* naming

* PR comments

* fix failing test after olympus default endpoint change

* remove rdbms config

* set defaults in code

* add support for proxyfilteroff

* add basic auth params

* update gopkg.lock

* undo gopkg.lock changes made during merge

* remove defaults

* explicitly specify all env variables

* remove rdbms from example

* remove user and pass to disable basic auth by default

* switch to memory by default for the proxy

* fix tests after config file change
2018-08-31 14:23:41 -04:00
Rohan Chakravarthy
697a7a5073 remove olympus storage implementation and proxy worker (#513)
* remove olympus storage implementation and proxy worker

* cleanup after merge conflicts

* remove newline
2018-08-30 03:47:04 -04:00
Arthur Mello
e5ce3f7c6e Set a default value for ATHENS_MONGO_CONNECTION_STRING env var (#580)
* Set a default value for ATHENS_MONGO_CONNECTION_STRING env var

- ATHENS_MONGO_CONNECTION_STRING will default to mongodb://127.0.0.1:27017 if not set

* Fix Get function call

* Update user of env.MongoConnectionString
2018-08-28 11:56:15 +02:00
Michal Pristas
539700e598 KindNotFound for all storages (#560)
* all exc mongo

* module, version associated with each storage error

* remove confusion

* removed formatted error in favor of raw flavor

* once more
2018-08-24 20:00:00 +02:00
Marwan Sulaiman
4a618990a2 pkg/download/goget: ensure each List func gets its own GOPATH (#559)
* pkg/download/goget: ensure each List func gets its own GOPATH

* Add list tests

* fix diskref tests
2018-08-24 10:19:17 -04:00
Marwan Sulaiman
fc581c3784 pkg/storage: ensure mongo returns KindNotFound (#558) 2018-08-24 02:13:09 -04:00
marpio
76121fb5cb Fix running the athens proxy on windows (#554)
* use parent env

* rename

* use only required vars

* review feedback

* add windows env vars

* ups

* fix

* rm op
2018-08-24 01:55:55 -04:00
mydiemho
e4659ecea5 create in memory driver for cdn metadata (#498)
* create in memory driver for cdn metadata

fixes #334

* save memStorage so we're not recreating it everytime

* run gofmt
2018-08-22 23:06:09 +02:00
Andrey Dyachkov
062f24d32f Issue 427: error check for storage exists method (#458)
* The commit brings error check for storage exists

The commit opened against ticket #427, and adds error response to
checker exists method and all corresponding places. The major changes related to renaming and processing of additional errors, because the signature of Checker.Exists was changed to return error in response.

* improved error processing

* go fmt'ed files

* added check to minio storage for no key error

* moved our constant + changed check for err

* go fmt'ed

* deleted rdbms packege, which brought merge with upsteam

* process error from minio checker
2018-08-22 15:49:36 -04:00
Manu Gupta
3a8c6429e9 Removal of cdn rdbms driver and pop support from olympus (#551)
* Inital removal of cdn rdbms driver and pop support from olympus

* Update travis and other unit test scripts

* Remove postgres from Makefile

* Update comments and remove function that is not required

* Update .env file

* clean commented code

* Update docs to remove references to rdbms

* Remove pop and run dep ensure to make sure that the tests pass without pop

* Keeping GO_ENV=test for parametter logging
2018-08-22 20:10:19 +02:00
Marwan Sulaiman
a1b5060ead Proxy: add cache-control middleware (#553)
* Proxy: add cache-control middleware

* fix improt cycle + use MDN no-cache example

* remove import lines

* actually use MDN example

* remove .idea from gitignore in unrelated pr
2018-08-22 09:52:16 -04:00
Manu Gupta
c0635b748c Remove pop from Proxy & RDBMS storage getter (#511)
* Remove pop from models for now

* Remove rdbms storage from pkg/storage

* Completely remove pop

* Some improvements to tests

- Not sourcing from cmd/proxy/.env, because that doesn't export any variables
- Removing mysql support (I feel like 1 database is enough)
- Pruning networks on teardown

I'm happy to split these changes up into separate PRs - it's the end of the day for me so I wanted to get everything in 😄

* update to point to test instance

* Update databasy.yml for travis??

* Remove database.yml

* Do not use db migrations and creation

* Remove pop dependency from cdn

* add schema for olympus assuming cdn driver is part of olympus

* Move CI/CD and test scripts to point pop to olympus

* Update database.yml and keep it only for olympus

* Add migrations to travis

* Add debug logs to see what travis is doing

* Update travis to use olympus database.yml

* Moarrrr logs to see where travis is pointint the db towards

* Change env to test

* Use test as the default environment if not specified otherwise

* Check if test new rdbms storage succeeds

* Try to see which connection string rdbms test suite is using on travis

* Update db tags for cdn driver

* create db by using travis go_env

* Remove sql from database config as it is removed from docker-compose

* Remove extra logs that I missed

* Add newline

* remove models.go and test file from proxy

* removing extraneous network prune from Makefile
2018-08-21 19:21:40 -04:00
Federico Paolinelli
c8b64fd258 Proxy 'admission controller' (#492)
* The proxy middleware calls an hook if set to check if the module needs
to be filtered

* Fixed error handling

* Added a middleware func that passes the log entry down to other
middlewares

* Split the filter middleware in two

* Moved validation middleware in its own file

* Renamed filter to middleware, all the middleware stuff in a single file

* Moved default hook to empty string

* Split tests of filter middleware from validation middleware

* Installed the validation middleware only if the hook is set

* Moved filter test to middleware test

* Fixed gofmt issue caused by manual merge

* Fixed wrong logrus import

* Removed logger from filter middleware

* Moved middleware functions to own package

* Removed unused argument, fixed errors

* Fixed gofmt errors

* Made middleware func private

* Changed debuglevel from string to enum

* Added comments to now public functions

* Changed module alias to a shorter one

* Removed white space

* Split middleware in separate files, injected the hook as param

* Removed commented code
2018-08-21 22:51:26 +02:00
Michal Pristas
042cbcaddf SSL Support for Mongo (#527)
* tested mongo ssl

* Added a todo
2018-08-21 18:09:49 +02:00
Marwan Sulaiman
87c92dfe28 Proxy: add NETRC file mount (#528) 2018-08-20 13:58:28 -04:00
Marwan Sulaiman
622aba0fe1 Athens: disable buffalo logger in production (#516)
* Athens: disable buffalo logger in production

* fix gofmt

* pkg/log: pass down the level instead of environment

* pkg/log: remove nop writer

* pkg/log: use indepnedent buffalo log level
2018-08-20 13:23:27 -04:00
Marwan Sulaiman
d749465e35 pkg/download: use worker pool to limit concurrent gogets (#524)
* pkg/download: use worker pool to limit concurrent gogets

* fix test closure

* use workers env
2018-08-20 11:30:12 -04:00
marpio
a10d98d6e7 set pkg level memStorage var so we get the same instance every time (#507) 2018-08-20 09:32:38 +02:00
Marwan Sulaiman
db5eb0ef69 proxy: add path prefix (#520) 2018-08-20 09:24:51 +02:00
mydiemho
c4bf6a13eb refactor SetPointer return base on golint suggestion (#522)
fixes #521
2018-08-20 09:04:28 +02:00
Marwan Sulaiman
4c55c813d0 Proxy: make Force SSL configurable (#515)
* Proxy: make Force SSL configurable

* fix godoc
2018-08-19 15:57:55 -04:00
Marwan Sulaiman
70167dc486 Proxy: use a global switch for Filter Middleware (#501)
* Proxy: do not redirect on filter.Include

* Filter: add Off opt

* Filter: set Off to true when there's no config

* Proxy: use a global switch for filter
2018-08-17 18:06:37 -04:00
Marwan Sulaiman
231cd85728 Proxy: add basic authentication middleware (#500)
* Proxy: add basic authentication middleware

* Proxy: fix ok
2018-08-17 09:01:35 -04:00