Commit Graph

167 Commits

Author SHA1 Message Date
Rob j Loranger
e6355881cc normalize all line endings for the repo (#269) 2018-07-14 11:32:55 -07:00
Rob j Loranger
46699719a0 fix merge conflict artifact and remove corba from Gopkg.toml (#265) 2018-07-13 19:22:43 -07:00
Michal Pristas
7795def18b SkipDead for workers (#256)
* skip dead

* refactor

* real redis tests of workers

* configurable maxfails
2018-07-13 12:18:24 -07:00
Tobias Kohlbau
f8fd040684 proxy: fixes missing proxy template check (#250) (#251)
Signed-off-by: Tobias Kohlbau <tobias@kohlbau.de>
2018-07-13 09:47:58 -07:00
marpio
9ee0478780 Stream zip up to AWS S3 storage (#254)
* add s3manager dep and remove cobra

* use env timeout

* use s3manager  for uploads

* ups
2018-07-13 09:36:16 -07:00
Tobias Kohlbau
5addd3304a repo: fixes github fetch #257 (#258)
Signed-off-by: Tobias Kohlbau <tobias@kohlbau.de>
2018-07-12 21:37:38 +02:00
Tobias Kohlbau
200ffe2a07 proxy: remove dangling registry templates (#252)
Signed-off-by: Tobias Kohlbau <tobias@kohlbau.de>
2018-07-12 12:35:11 +02:00
Tobias Kohlbau
75c7e1931f make: drop cli reference (#249)
CLI was removed and so should be the dangling reference within `Makefile`.
2018-07-12 12:31:35 +02:00
Michal Pristas
119e975c8b adding cache dir (#253) 2018-07-12 12:29:00 +02:00
Aaron Schlesinger
d49e896d63 Adding option to disable CSRF protection (#248)
* Adding option to disable CSRF protection

This is useful at least for local development

cc/ @marwan-at-work

* using proper func name
2018-07-11 17:12:04 -07:00
Aaron Schlesinger
6459512234 Adding an athens philosophy document (#246)
* Adding an athens Philosophy

* a few guiding principles

* better grammar

thanks @robjloranger!

* Adding @bketelsen's guide on contributing to a Go OSS project
2018-07-11 15:42:40 -07:00
Marwan Sulaiman
b823d6093e Athens: delegate worker retries to gocraft/work (#247) 2018-07-11 15:15:56 -07:00
Michal Pristas
767be5ba47 Common testing ground for storage (#215)
* added test for notfound

* 🚨 🔨

* hahaha

* test

* Olympus: default to PORT env var (#226)

* Olympus: default to PORT env var

* Athens: generalize env.Port function call

* Olympus: add PORT to .env

* test

* getset
'

* Docs

* separate package
2018-07-11 11:45:41 -07:00
marpio
3504476fa1 Push notification test (#239)
* refactor app to make testin easier

* add test

* add comment

* extend cache miss handler test

* use env for mongo uri

* struct for app dependencies

* eanglish iz dificult ;-)

* address review feedback
2018-07-10 12:11:50 -07:00
Marwan Sulaiman
e342cefbd9 Olympus: implement download protocol (#238)
* Olympus: implement download protocol

* Download: change path names

* Git ignore tmp directories
2018-07-10 12:01:09 -07:00
Aaron Schlesinger
4c1694f760 Removing obsolete CLI and API docs (#234) 2018-07-10 10:11:06 -07:00
Aaron Schlesinger
2bdad08f8c Update README.md 2018-07-09 16:30:51 -07:00
Aaron Schlesinger
a8730a16d2 Moving design docs from the wiki (#235)
* Adding the design doc for the proxy

Taken from the wiki page and modified slightly

* Moving the wiki page for registry to a doc

and modifying slightly to remove the "Olympus" nomenclature, since it's confusing
2018-07-09 16:29:47 -07:00
Aaron Schlesinger
349adbe58b Adding an "on the shoulders of giants" section and document (#237) 2018-07-09 16:23:47 -07:00
Aaron Schlesinger
780c7e18a1 Adding getting involved section, resources, etc... (#236) 2018-07-09 16:12:47 -07:00
Aaron Schlesinger
567aaf46fd Update README.md 2018-07-09 15:21:59 -07:00
Aaron Schlesinger
a4e39c82d8 Refactoring readme and adding a DEVELOPMENT doc (#233) 2018-07-09 15:19:44 -07:00
Aaron Schlesinger
b4392d5356 Adding code of conduct in its own section on the README (#231) 2018-07-09 15:06:22 -07:00
Aaron Schlesinger
c9bb28fd6c Removing the registry renderer (#229)
The proxy should only render its own templates
2018-07-09 14:44:27 -07:00
Aaron Schlesinger
91b4f9c876 WIP: always downloading a fixed buffalo CLI version (#230)
* always downloading a fixed buffalo CLI version

As of this writing, latest is v0.12.3

* moving the buffalo binary to a permitted place

also making the binary executable

* composing buffalo URL with tar_gz env var
2018-07-09 14:23:45 -07:00
Aaron Schlesinger
c3e3dee4d8 WIP: Removing the CLI (#227)
* Removing the CLI

Still TODO:

- Removing CLI references from docs

* removing the payloads.Upload struct

since it's not used anywhere
2018-07-09 13:44:58 -07:00
Marwan Sulaiman
f1d66d9d45 Olympus: default to PORT env var (#226)
* Olympus: default to PORT env var

* Athens: generalize env.Port function call

* Olympus: add PORT to .env
2018-07-09 13:26:43 -07:00
Michal Pristas
514ac49659 Support for exclude include (#212)
* almost there

* loading file

* fixed after real test

* fixed includes"

* configurable filter name

* custom error
2018-07-09 09:58:51 -07:00
Marwan Sulaiman
bcc7272020 Olympus: default port to 3001 (#225) 2018-07-09 09:54:37 -07:00
Marwan Sulaiman
da009db506 Zeus: check for status code in ModuleStore.Get (#224) 2018-07-09 17:22:46 +02:00
Michal Pristas
c02e9c0c23 Added configurable CDN endpoint (#214)
* added configurable cdn endpoint
2018-07-09 08:42:22 +02:00
Marwan Sulaiman
fda6ec5d4e Olympus: remove potential PROXY_URL when running vgo get (#222)
* Olympus: remove potential PROXY_URL when running vgo get

* Olympus: add PATH to vgo get

* Olympus add PATH comments
2018-07-08 16:22:10 -07:00
Marwan Sulaiman
32d5b3535c Fix upload nil pointer dereference (#217) 2018-07-08 15:50:31 -07:00
marpio
58d42a1b57 Stream zips (#200)
* storage.saver interface - use io.reader instead of []byte

* stream zip - backendconnector

* stream zip - azurecdn

* stream zip - fs

* storage - switch to io.Reader

* update callers of Save

* rm empty line

* makezip stream

* refactor mergedb

* add multierror dep

* mergedb multierror

* scope err

* update storages

* return pipereader

* fix test

* use multierror for all strorage drivers

* review feedback

* azure cdn storage - use UploadStreamToBlockBlob

* use numUpload

* use timeout ctx
2018-07-07 08:46:43 -07:00
Rob j Loranger
95b6143dc4 better docs for gcp storage (#202)
better documentation for GCP storage
2018-07-05 22:43:45 +02:00
Michal Pristas
8dc02d6b3b Env and config refactor (#209)
* reduce envy just to one package
2018-07-05 22:32:17 +02:00
Rob j Loranger
900c064c0e rename some variables for consistency (#204) 2018-06-29 15:07:11 -07:00
Michal Pristas
5caa8a369d AWS S3 Storage (#194)
* basic aws impl

* iface client

* comments

* s3 mock

* Added test

* exclude mocks from linter

* get rid of races

* subpackaging
2018-06-29 16:02:16 -06:00
Michal Pristas
b47a7f92e2 fixed (#201) 2018-06-28 11:23:11 +02:00
Rob j Loranger
5101e1ea6d basic root dir pre check on fs storage.New (#199)
* basic root dir pre check on fs storage.New

* update uses of fs.storage.New to check for error

* use testify to check for error
2018-06-26 12:37:21 -06:00
Rob j Loranger
9e76722acc GCP storage saver (#183)
* Add Dependencies for GCP

cloud.google.com/go/storage
google.golang.org/api/option // allows authetication options for storage
github.com/golang/protobuf // override to fix dependency issue

issue was with a protobuf library required by the storage library.
it was importing v1.0.0 but referencing code from v1.1.0

* WIP mostly roughed out

some small descisions to be made and testing is missing

* rename gcp.go to saver.go

* rename gcp.go to saver.go

* Requested Changes

+ closing the storage.Client is not required
+ move increment of count into 'for' line
+ add tracing span for save helper
+ removed projectID from New signiture, unused

* basic test, just testing generator so far

* Readability fixes

+ change calls to save so they send directly to the errs channel
+ return wc.Close instead of checking the error first
+ also remove failing part of tests for now, needs authentication

* Happy Path Tests

- removed use of buffalo.Context
- (storage).Save no longer accepts a context
+ includes basic happy path tests
+ private key for test project praxis-cab-207400.appspot.com

* messy gitwork

* remove keys

* ch-ch-changes🎶

+ defer close of storage.Writer, logging to be decided
+ correct check on length of errsOut
+ set inital length of errsOut to 0

* more stuff is in here

+ switch to context being passed again
+ use context with timeout instead of time.After
+ tests pull creds from env variable

missing skipping tests when no variable is set and teardown is no
implemented

* skip gcp storage tests when no key file present

* speed up tests a little

* need aetest pkg

* add aetest to Gopkg.toml explicitly

* tear down suite deletes storage artifacts

* should setup whole suite

* log error on wc.close and close errs chan

* cleaner check for env var

* use envy where appropriate and pull gcp bucket name from env var

* one more for consistency

I like the comma ok idiom better in this case but everything else is
using envy.

* don't panic in test set up and tear down

* use shared context with timeout and switch to seconds for duration

* no one needed the context.Cancel func. use `file` instead of `content` for consistency

* defer cancellation of context for uploads

* Update saver.go

+ rename `save` to `upload`
+ clarify comments
2018-06-25 15:49:56 -06:00
marpio
d0cbd9b809 [WIP] Push endpoint (#181)
* initial structure

* make it compile...

* missing comment

* fix after rename

* single worker, two handlers

* minio starting to late

* fix after merge
2018-06-25 14:03:01 -06:00
Aaron Schlesinger
2d25ccb3de spinning up required services in the makefile dev target (#192)
fixes https://github.com/gomods/athens/issues/191
2018-06-21 15:42:36 -07:00
Martin Strobel
d0a18e7dd4 Add Context to Saver Interface (#179)
* Adding context to storage.Saver parameters

* +`buffalo.Context` as first param in storage.Saver

Fixes #177

* Parallelizing Upload to Azure Blob Storage

Fixes #175

* Fixing import clausing

* go fmt ./...

* Populating default context

* CI: Echo files modified by gofmt check to stderr

Doing so allows users to see which files still need to be formatted.

* Upgrading Go, then go fmt ./...

* Addressing Feedback

* more responding to feedback

* Adding worker save Default Context

* Removing tracing, buffalo.Context usage in Storage interface

github.com/bketelsen/buffet relies on using a `buffalo.Context`
however, often a `storage.Saver` is invoked from a buffalo worker,
which has no buffalo.Context associated with it. Therefore, we had
two options: create a buffalo context for each worker's Job, or
remove the buffalo.Context dependency in the Saver interface.

* Fixing missed changes

* Review feedback
2018-06-21 10:47:31 -07:00
Aaron Schlesinger
da5b269c2d adding a "dev" target to the makefile (#189) 2018-06-20 17:17:05 -07:00
Aaron Schlesinger
312da56044 fixing olympus dockerfile (#186)
* fixing olympus dockerfile

* adding olympus docker build target

* cleaning up dockerfile
2018-06-20 13:54:11 -07:00
Michal Pristas
5765224b31 fixed (#185) 2018-06-20 09:06:27 +02:00
Aaron Schlesinger
ce5a96ddc4 s/Athens/Zeus
Ref https://github.com/gomods/athens/issues/184
2018-06-18 15:26:22 -07:00
Michal Pristas
c4ac27f585 Naive implementation of bg job processing downloads (#182)
* naive implementation

* added redis
2018-06-15 22:20:37 +02:00
Aaron Schlesinger
a8e71934eb [WIP] Implementing a CDN driver for Azure (#170)
* Implementing the CDN getter and storage saver for Azure CDN

Fixes https://github.com/gomods/athens/issues/29

* Adding azure dependencies

* adding GoGet middleware into olympus

just using azure blob storage for now. will parameterize later

* cleaning up compile errs

* adding fmt.Sprintf param

* removing newline in imports

* following the rules for container names

* removing unused import
2018-06-14 21:08:49 +00:00