Commit Graph

55 Commits

Author SHA1 Message Date
Marwan Sulaiman
feb17a7a61 Athens: turn off worker to avoid nil logs (#483) 2018-08-14 21:43:39 -04:00
Carolyn Van Slyck
afd01813e9 Default to in-memory storage for the proxy (#475)
* Default to in-memory storage for the proxy

This makes it easier for someone to try out or just run the proxy by
executing the binary without further environment setup.

* Remove obsolete comment
2018-08-14 17:54:26 -04:00
Michal Pristas
327fecaedd rename Private to Direct (#481) 2018-08-14 11:13:36 +02:00
Federico Paolinelli
e6d4996bad Added support for private repos (#456)
* Added filter for private repos. Non private repos get redirected to
olympus

* Changed filter to allow private modules

* Changed config file name function

* Added filter tests

* The middleware returns 200 for privates, redirects public, 404 for
disabled repos

* Removed printf

* Removed forgotten printf

* Removed logger from the middleware func, not used anymore

* Removed unused (uncommented) function

* Replaced 404 with 403 in case of banned modules

* Fixed test path / cleaned olympus endpoint suffix

* Added error operation to GetVersion

* Moved filter test in custom file, reverted injection of the filter into
the app

* Removed white line

* Clarified comment for ignoring getversion error

* Added todo comment to fill the cache and serve the request with the
cache

* Added error handling to filter

* Removed commented early version of the test
2018-08-13 16:35:17 -07:00
Michal Pristas
0ded8098ab removed unused cache miss reporter (#461) 2018-08-13 15:01:24 +02:00
Andrey Dyachkov
051f3884e1 Bring more clarity on how to run unit tests (#449)
* The commit brings more clarity on how to run unut tests.

The commit chages behaivour of `make test-unit` by adding sourcing of
env vars. Documentation is updated to provide more clarity on how to run
unit tests.

* changed list of commands to run before unit tests

* moved env vars to .env

* removed dduplicated env var

* returned conventional file name in script

* link to DEVELOPMENT.md

* fixed typo

* moved env var export to script

* removed export of tmp folder
2018-08-11 12:12:24 -07:00
Marwan Sulaiman
95cfa173e2 Storage: split storage.Getter interface (#406)
* Storage: split storage.Getter interface

* no need to clean up whats not there

* unimplement rdmb getter

* unimplement rdmb getter

* make storage tests happy

* comment out rdmbs from tests

* fix tests

* fix stuff we're removing anyway

* PR fixes

* oops

* fix gcp exists
2018-08-08 12:04:02 -04:00
Manu Gupta
23639c74d5 Add logo to proxy (#422)
* Add logo to proxy

* Add newline to css
2018-08-08 08:46:20 -07:00
Marwan Sulaiman
6b766c3f72 Download: introduce sane Entry defaults (#398) 2018-08-04 12:27:24 -04:00
Manu Gupta
7aafc2692e Change service names to specify olympus / proxy (#384) 2018-08-03 07:54:45 -07:00
Marwan Sulaiman
15326af2bb GCP: get credentials implicitly (#380)
* GCP: get credentials implicitly

* update name
2018-08-02 12:37:19 -04:00
Dinesh Kumar
85213f20b3 Goget Fetcher should error out if gobinpath is not valid (#377)
* Goget fetcher should error out if gobinpath is not valid

* Propogating the error from goget initialisation to main, to stop application

* wrapping errors with op, using exec.command(gobin).Run to verify gobin, clean up

* Inlining afero fs to goget call, and new go getfetcher

* Revert "Inlining afero fs to goget call, and new go getfetcher"

This reverts commit ae31fe6a2b.

* Fixing example test
2018-08-02 10:00:37 -04:00
Komu Wairagu
af2ecfddae rm env.GoPath (#370)
1. Gopath func doesnt seem to be used much; it is used in init func of proxy/actions/app.go which doesnt do much
2. I think relying on $GOPATH post go1.11 will lead to flaky situations
2018-07-31 10:14:20 -04:00
Aaron Schlesinger
59d5d5ffac Cleaning up unused code (#352)
* removing pkg/repo

Fixes https://github.com/gomods/athens/issues/335

* removing the gomod parser
2018-07-30 14:06:17 -07:00
Manu Gupta
afa831472d Implement context threads to be passed to storage.*.getter (#346)
* Implement context threads to be passed to storage.*.getter

* Create a background context thread on athens.proxy
2018-07-30 20:11:52 +02:00
Chad Kunde
53df0f7c6a clean up app initialization if blocks (#347)
Remove global singleton.  Create and initialize a new buffalo app on
each invocation of the constructor.
2018-07-29 22:01:30 -07:00
Marwan Sulaiman
add15e68cd download: clean up Kahona (#345) 2018-07-27 15:18:14 -07:00
Carolyn Van Slyck
fd7c1326d3 Backfill the cache on a miss (#342)
* add middlewares to fill the cache and to populate contexts with module and versions

* Carolyn tries to fix Aaron's code

* #DOINSTUFF

* updated env var

* fix all the things

* magic patch for olympus

* Add latest handler

* Remove deprecated return param

* go fmt all the things
2018-07-27 14:37:23 -07:00
Marwan Sulaiman
7b590b1885 download: add list from go cli (#336)
* download: add list from go cli

* download: include goget tests + hacky hack

* download: move dummyMod to pkg/module

* Olympus: pass dp and lggr to /list

* download: add Version to interface

* download: document Protocol
2018-07-27 11:56:32 -07:00
Marwan Sulaiman
0853f073f2 Athens: introduce pkg/errors (#320)
* Athens: introduce pkg/errors

* errors: fix Ops loop

* pkg/errors: introduce M and V types to get rid of unstable parsing

* fix build

* errors: fix text + add tests

* pr updates
2018-07-25 13:18:58 -05:00
Tyler Bui-Palsulich
27bbc3f42f storage: add context to storage.Exists and storage.Delete (#322)
* storage: add context to storage.Exists and storage.Delete

* eventlog: use context.TODO()

* storage/gcp: reuse ctx
2018-07-24 17:12:31 -05:00
Komu Wairagu
d74e0640fe remove dead code (#267)
* remove spf13/cobra and inconshreveable/mousetrap

* remove user package

* remove user package

* remove unused types, vars etc
found by using github.com/dominikh/go-tools/tree/master/cmd/unused.running  unused $(go list github.com/gomods/athens/... | grep -v /vendor/)

* rm uneeded file
2018-07-19 08:34:16 -07:00
Marwan Sulaiman
eb0768f3fe Proxy: fix init nil dereference (#282) 2018-07-16 15:21:30 -04:00
Marwan Sulaiman
084e906804 Athens: remove c.Error (#278)
* Athens: remove c.Error

* Download: return err string on VersionModuleHandler
2018-07-15 19:40:45 -07:00
Komu Wairagu
94bbcd1996 update redigo (#266)
* dep ensure -v -update github.com/gocraft/work

* rm -rf vendor/github.com/garyburd

* sed /garyburd/redigo / gomodule/redigo

* dont constrain redigo version

* update redigo ver

* rm deps

* update deps

* add vendor
2018-07-15 10:37:29 -07:00
Rob j Loranger
1aaacc7b59 Zeus: remove duplicate '/' handler assignment (#272)
* remove duplicate '/' handler declaration

* Remove homeHandler() and slow()

homeHandler is unused now and slow was a helper to add tracing with a
millisecond delay to the handler
2018-07-15 08:20:32 -07:00
Marwan Sulaiman
650577a6da Athens: introduce log package (#274)
* Athens: introduce log package

* log: add buffalo doc note

* log: add cloud runtime based formatting

* log: add log level to constructor

* log: remove nil err from fmtBuffaloErr

* log: use switch statement in buffaloFormatter.Format

* log: rearrange .go files

* log: change cloudruntime development to none

* Olympus: add buffalo logger
2018-07-15 11:13:02 -04:00
Michal Pristas
7795def18b SkipDead for workers (#256)
* skip dead

* refactor

* real redis tests of workers

* configurable maxfails
2018-07-13 12:18:24 -07:00
Tobias Kohlbau
f8fd040684 proxy: fixes missing proxy template check (#250) (#251)
Signed-off-by: Tobias Kohlbau <tobias@kohlbau.de>
2018-07-13 09:47:58 -07:00
Tobias Kohlbau
200ffe2a07 proxy: remove dangling registry templates (#252)
Signed-off-by: Tobias Kohlbau <tobias@kohlbau.de>
2018-07-12 12:35:11 +02:00
Aaron Schlesinger
d49e896d63 Adding option to disable CSRF protection (#248)
* Adding option to disable CSRF protection

This is useful at least for local development

cc/ @marwan-at-work

* using proper func name
2018-07-11 17:12:04 -07:00
Marwan Sulaiman
b823d6093e Athens: delegate worker retries to gocraft/work (#247) 2018-07-11 15:15:56 -07:00
Michal Pristas
767be5ba47 Common testing ground for storage (#215)
* added test for notfound

* 🚨 🔨

* hahaha

* test

* Olympus: default to PORT env var (#226)

* Olympus: default to PORT env var

* Athens: generalize env.Port function call

* Olympus: add PORT to .env

* test

* getset
'

* Docs

* separate package
2018-07-11 11:45:41 -07:00
Marwan Sulaiman
e342cefbd9 Olympus: implement download protocol (#238)
* Olympus: implement download protocol

* Download: change path names

* Git ignore tmp directories
2018-07-10 12:01:09 -07:00
Aaron Schlesinger
c9bb28fd6c Removing the registry renderer (#229)
The proxy should only render its own templates
2018-07-09 14:44:27 -07:00
Aaron Schlesinger
c3e3dee4d8 WIP: Removing the CLI (#227)
* Removing the CLI

Still TODO:

- Removing CLI references from docs

* removing the payloads.Upload struct

since it's not used anywhere
2018-07-09 13:44:58 -07:00
Michal Pristas
514ac49659 Support for exclude include (#212)
* almost there

* loading file

* fixed after real test

* fixed includes"

* configurable filter name

* custom error
2018-07-09 09:58:51 -07:00
Marwan Sulaiman
bcc7272020 Olympus: default port to 3001 (#225) 2018-07-09 09:54:37 -07:00
marpio
58d42a1b57 Stream zips (#200)
* storage.saver interface - use io.reader instead of []byte

* stream zip - backendconnector

* stream zip - azurecdn

* stream zip - fs

* storage - switch to io.Reader

* update callers of Save

* rm empty line

* makezip stream

* refactor mergedb

* add multierror dep

* mergedb multierror

* scope err

* update storages

* return pipereader

* fix test

* use multierror for all strorage drivers

* review feedback

* azure cdn storage - use UploadStreamToBlockBlob

* use numUpload

* use timeout ctx
2018-07-07 08:46:43 -07:00
Michal Pristas
8dc02d6b3b Env and config refactor (#209)
* reduce envy just to one package
2018-07-05 22:32:17 +02:00
Rob j Loranger
5101e1ea6d basic root dir pre check on fs storage.New (#199)
* basic root dir pre check on fs storage.New

* update uses of fs.storage.New to check for error

* use testify to check for error
2018-06-26 12:37:21 -06:00
Martin Strobel
d0a18e7dd4 Add Context to Saver Interface (#179)
* Adding context to storage.Saver parameters

* +`buffalo.Context` as first param in storage.Saver

Fixes #177

* Parallelizing Upload to Azure Blob Storage

Fixes #175

* Fixing import clausing

* go fmt ./...

* Populating default context

* CI: Echo files modified by gofmt check to stderr

Doing so allows users to see which files still need to be formatted.

* Upgrading Go, then go fmt ./...

* Addressing Feedback

* more responding to feedback

* Adding worker save Default Context

* Removing tracing, buffalo.Context usage in Storage interface

github.com/bketelsen/buffet relies on using a `buffalo.Context`
however, often a `storage.Saver` is invoked from a buffalo worker,
which has no buffalo.Context associated with it. Therefore, we had
two options: create a buffalo context for each worker's Job, or
remove the buffalo.Context dependency in the Saver interface.

* Fixing missed changes

* Review feedback
2018-06-21 10:47:31 -07:00
Brian Ketelsen
526b5f57f8 fix open tracing by setting operation name (#171) 2018-06-14 16:36:06 -04:00
Michal Pristas
6c4a0641df Background job for pulling modules from olympus (#132)
* basic structure

* added olympus endpoint for fetching log

* broken test

* added backing buffalo job

* slightly reshuffled, todo more shuffling

* reshuffle more

* breaking things up

* more breaking up

* hook to 101

* added redis to docker compose file

* working!

* a

* build fix

* first batch of changes towards new arch"

* first batch of comments resolved

* override for olympus endpoint

* styling

* osFs fix

* fixed in memory storage

* in-mem storage and test (#166)
2018-06-11 18:12:10 +02:00
marpio
a1e76e4569 in-mem storage and test (#166) 2018-06-11 09:56:04 +02:00
marpio
977d816c89 add rdbms support for cdn metadata storage (#151)
* add rdbms support for cdn metadata storage

* move cdn metadata model to the metadata pkg

* simplify get

* add mongo saver and test

* remove 'stutters'

* change default table name

* missing cdn metadata pkg rename
2018-06-06 13:59:26 -07:00
Aaron Schlesinger
f08d3138bc Making local tests runnable (#157)
* using the right DB names in the proxy database file

* using the right DB names in the olympus database file

and the right ports

* using the right DB name in the docker-compose

* adding mongo URL to the buffalo env file

* adding mongo URL to the proxy env file

* updating port for the proxy DB

* updating DB schemas for olympus and the proxy

* making the readme a little more up to date
2018-06-06 12:42:16 -07:00
Aaron Schlesinger
38a9ff3b87 Adding a DB diff handler (#153)
* Adding a DB diff handler

* Op => EventOp

to clear up confusion

* deleting and ignoring  proxy binary
2018-06-06 11:44:52 -07:00
Aaron Schlesinger
2142d9989c Clean up the athens codebase (#145)
* Making code generic between athens and olympus

* removing support in athens for "registry mode"

that is now olympus :)

* fixing compile err

* add a test target for the makefile
2018-06-06 10:55:30 -07:00
Rob j Loranger
9df207561d Add .info to upload payload (#141)
* Add .info to upload payload
everywhere else as well

+ Upload payload includes Info
+ CLI uploads `version.info` as blob to storage
+ Getter and Saver methods updated for all storages implemented
+ Migration updated to include Info column
+ Tests no longer check RevInfo, now checking for Info same as Zip
+ Updated docs

* newline in .fizz

* change RevInfo.Info to []byte

+ change to []byte instead of io.Reader
+ adjust tests to check for info byte contents directly
+ add error check for minio client GetObject `pkg/storage/minio/getter.go`
2018-05-29 08:03:01 +02:00