Commit Graph

23 Commits

Author SHA1 Message Date
Rob j Loranger
46699719a0 fix merge conflict artifact and remove corba from Gopkg.toml (#265) 2018-07-13 19:22:43 -07:00
Michal Pristas
7795def18b SkipDead for workers (#256)
* skip dead

* refactor

* real redis tests of workers

* configurable maxfails
2018-07-13 12:18:24 -07:00
marpio
9ee0478780 Stream zip up to AWS S3 storage (#254)
* add s3manager dep and remove cobra

* use env timeout

* use s3manager  for uploads

* ups
2018-07-13 09:36:16 -07:00
marpio
58d42a1b57 Stream zips (#200)
* storage.saver interface - use io.reader instead of []byte

* stream zip - backendconnector

* stream zip - azurecdn

* stream zip - fs

* storage - switch to io.Reader

* update callers of Save

* rm empty line

* makezip stream

* refactor mergedb

* add multierror dep

* mergedb multierror

* scope err

* update storages

* return pipereader

* fix test

* use multierror for all strorage drivers

* review feedback

* azure cdn storage - use UploadStreamToBlockBlob

* use numUpload

* use timeout ctx
2018-07-07 08:46:43 -07:00
Michal Pristas
5caa8a369d AWS S3 Storage (#194)
* basic aws impl

* iface client

* comments

* s3 mock

* Added test

* exclude mocks from linter

* get rid of races

* subpackaging
2018-06-29 16:02:16 -06:00
Rob j Loranger
9e76722acc GCP storage saver (#183)
* Add Dependencies for GCP

cloud.google.com/go/storage
google.golang.org/api/option // allows authetication options for storage
github.com/golang/protobuf // override to fix dependency issue

issue was with a protobuf library required by the storage library.
it was importing v1.0.0 but referencing code from v1.1.0

* WIP mostly roughed out

some small descisions to be made and testing is missing

* rename gcp.go to saver.go

* rename gcp.go to saver.go

* Requested Changes

+ closing the storage.Client is not required
+ move increment of count into 'for' line
+ add tracing span for save helper
+ removed projectID from New signiture, unused

* basic test, just testing generator so far

* Readability fixes

+ change calls to save so they send directly to the errs channel
+ return wc.Close instead of checking the error first
+ also remove failing part of tests for now, needs authentication

* Happy Path Tests

- removed use of buffalo.Context
- (storage).Save no longer accepts a context
+ includes basic happy path tests
+ private key for test project praxis-cab-207400.appspot.com

* messy gitwork

* remove keys

* ch-ch-changes🎶

+ defer close of storage.Writer, logging to be decided
+ correct check on length of errsOut
+ set inital length of errsOut to 0

* more stuff is in here

+ switch to context being passed again
+ use context with timeout instead of time.After
+ tests pull creds from env variable

missing skipping tests when no variable is set and teardown is no
implemented

* skip gcp storage tests when no key file present

* speed up tests a little

* need aetest pkg

* add aetest to Gopkg.toml explicitly

* tear down suite deletes storage artifacts

* should setup whole suite

* log error on wc.close and close errs chan

* cleaner check for env var

* use envy where appropriate and pull gcp bucket name from env var

* one more for consistency

I like the comma ok idiom better in this case but everything else is
using envy.

* don't panic in test set up and tear down

* use shared context with timeout and switch to seconds for duration

* no one needed the context.Cancel func. use `file` instead of `content` for consistency

* defer cancellation of context for uploads

* Update saver.go

+ rename `save` to `upload`
+ clarify comments
2018-06-25 15:49:56 -06:00
Aaron Schlesinger
a8e71934eb [WIP] Implementing a CDN driver for Azure (#170)
* Implementing the CDN getter and storage saver for Azure CDN

Fixes https://github.com/gomods/athens/issues/29

* Adding azure dependencies

* adding GoGet middleware into olympus

just using azure blob storage for now. will parameterize later

* cleaning up compile errs

* adding fmt.Sprintf param

* removing newline in imports

* following the rules for container names

* removing unused import
2018-06-14 21:08:49 +00:00
Michal Pristas
6c4a0641df Background job for pulling modules from olympus (#132)
* basic structure

* added olympus endpoint for fetching log

* broken test

* added backing buffalo job

* slightly reshuffled, todo more shuffling

* reshuffle more

* breaking things up

* more breaking up

* hook to 101

* added redis to docker compose file

* working!

* a

* build fix

* first batch of changes towards new arch"

* first batch of comments resolved

* override for olympus endpoint

* styling

* osFs fix

* fixed in memory storage

* in-mem storage and test (#166)
2018-06-11 18:12:10 +02:00
marpio
cde250728a Implement Minio storage (#128)
* add minio dep

* implement minio storage

* fix travis.yml

* start minio in background
2018-05-07 18:13:11 -07:00
Brian Ketelsen
e76125d3f1 Feature: Add OpenTracing (#112)
* tracing

* oops
2018-03-26 17:14:49 -04:00
Aaron Schlesinger
b7e15a845c Mongo cdn getter (#62)
* Implementing a MongoDB backed CDN getter

Fixes https://github.com/gomods/athens/issues/49

* removing the base URL from the CDN getter
2018-03-23 13:41:22 -07:00
Brian Ketelsen
cf6a6f4e3f UI: Start UI (#35)
* UI: Start UI

* link to github auth

* save point

* simplify the UI a lot

* make a separate renderer for proxy vs registry

* get rid of debug

* update deps

* remove unneeded assets
2018-03-21 17:09:45 -04:00
marpio
2694401b7e Use afero for as abstraction layer for FS (#70)
* add afero dependency

* implement memory and disk storage using afero fs

* pass the appropriate fs

* remove unnecessary interface in the fs package
2018-03-20 18:39:55 -07:00
Michal Pristas
1df7400819 not tested version (#54) 2018-03-15 20:01:54 +01:00
Brian Ketelsen
733fa62ec5 athens: add userStore interface and mongoDB implementation (#34)
* athens: save and retrieve OAUTH users from github to mongodb storage interface

* deps

* add mongo to travis

* updates based on PR feedback
2018-03-07 15:29:15 -05:00
Aaron Schlesinger
353c1e35d1 Moving to cobra for the CLI (#7)
* [WIP] Moving to cobra for the CLI

Fixes https://github.com/gomods/athens/issues/6

* dep ensure

* dep ensure

* travis ignoring vendor for 1.8

* disabled 1.8 for travis
2018-03-07 09:16:09 -05:00
Michal Pristas
7fc0e6f3b5 zip creation fixed (#30) 2018-03-05 22:43:29 +01:00
Brian Ketelsen
ef2a8230b9 basic github oauth (#23) 2018-03-02 11:22:38 -08:00
Brian Ketelsen
10bbd589b7 athens: update to buffalo 0.11.0, add cors and csrf support (#13) 2018-03-01 10:22:48 -08:00
Aaron Schlesinger
89eeef7da1 adding CLI and refactoring payload defns 2018-02-25 14:34:01 -08:00
Aaron Schlesinger
cf3d44a62c pruning
after the go-git dependency was dropped
2018-02-25 14:06:34 -08:00
Aaron Schlesinger
a7f1d27491 adding list and admin API 2018-02-25 14:05:36 -08:00
Aaron Schlesinger
5c889a06f4 more initial things 2018-02-25 12:32:51 -08:00