Commit Graph

167 Commits

Author SHA1 Message Date
Brian Ketelsen
526b5f57f8 fix open tracing by setting operation name (#171) 2018-06-14 16:36:06 -04:00
Aaron Schlesinger
be140843e7 Update README.md 2018-06-14 13:19:51 -07:00
Aaron Schlesinger
99a00dddde Update README.md 2018-06-14 13:18:27 -07:00
Aaron Schlesinger
c0c17971d5 Update REGISTRY.md 2018-06-14 12:57:23 -07:00
Aaron Schlesinger
d99b04c5ce [WIP] Adding skeleton for doing database merges (#152)
* Adding CDN metadata functions

* Fixing compile error

* adding shim for downloading code from other CDN

* using events in the diff struct

* Implementing mergeDB

* Op => EventOp

* modifying olympus DB information

to connect to athens DB

* Op => EventOp

* making loops continue instead of bailing out

also adding logs on errors

* Adding TODO tests

* redesign
2018-06-14 19:53:10 +00:00
Brian Ketelsen
02b034eb06 enable code coverage (#169) 2018-06-14 13:34:42 -04:00
marpio
6b078e96eb Codecov & travis.yml cleanup (#167)
* add codecov

* move env vars to env section

* remove vet since it is integrated into go test

* forgot global

* covermode atomic

* move minio vars to before_script and hope for the best

* try to see if it was a random error
2018-06-14 13:32:15 -04:00
Michal Pristas
6c4a0641df Background job for pulling modules from olympus (#132)
* basic structure

* added olympus endpoint for fetching log

* broken test

* added backing buffalo job

* slightly reshuffled, todo more shuffling

* reshuffle more

* breaking things up

* more breaking up

* hook to 101

* added redis to docker compose file

* working!

* a

* build fix

* first batch of changes towards new arch"

* first batch of comments resolved

* override for olympus endpoint

* styling

* osFs fix

* fixed in memory storage

* in-mem storage and test (#166)
2018-06-11 18:12:10 +02:00
Rob j Loranger
aee9d3abb8 implement file system based pointer registry (#161)
* tests and partial implementation, failing

* Implementation complete for FS only

afero.MemMapFs is not working

* fix to allow use of MemMapFs in tests

* use encoding/json instead of gob

* no need to keep test data file

* add comment regarding use of json vs gob encoding

* Incorporate Review Changes

+ fix ordering in test require.Equal
+ make function calls that only return an error more compact
+ reduce map lookups in LookupPointer func
+ clean up old comment

* no longer defer f.Close

* defer close on open files
2018-06-11 16:27:35 +02:00
marpio
a1e76e4569 in-mem storage and test (#166) 2018-06-11 09:56:04 +02:00
marpio
5147e0a599 Add cdn metadata fs driver (#164)
* add fs driver

* rm new line

* create time upfront

* fix permissions
2018-06-08 21:37:31 +02:00
Rob j Loranger
ffff0b8172 tidy up pointer registry mongo driver (#163)
* use correct order for testify require.Equal

* make error check more compact
2018-06-08 20:38:17 +02:00
marpio
977d816c89 add rdbms support for cdn metadata storage (#151)
* add rdbms support for cdn metadata storage

* move cdn metadata model to the metadata pkg

* simplify get

* add mongo saver and test

* remove 'stutters'

* change default table name

* missing cdn metadata pkg rename
2018-06-06 13:59:26 -07:00
Aaron Schlesinger
f08d3138bc Making local tests runnable (#157)
* using the right DB names in the proxy database file

* using the right DB names in the olympus database file

and the right ports

* using the right DB name in the docker-compose

* adding mongo URL to the buffalo env file

* adding mongo URL to the proxy env file

* updating port for the proxy DB

* updating DB schemas for olympus and the proxy

* making the readme a little more up to date
2018-06-06 12:42:16 -07:00
Aaron Schlesinger
38a9ff3b87 Adding a DB diff handler (#153)
* Adding a DB diff handler

* Op => EventOp

to clear up confusion

* deleting and ignoring  proxy binary
2018-06-06 11:44:52 -07:00
Aaron Schlesinger
2142d9989c Clean up the athens codebase (#145)
* Making code generic between athens and olympus

* removing support in athens for "registry mode"

that is now olympus :)

* fixing compile err

* add a test target for the makefile
2018-06-06 10:55:30 -07:00
Rob j Loranger
947845844d add basic mongo implementation of pointer registry (#150)
* add basic mongo implementation of pointer registry

+ includes LookupPointer only for now

* complete implementation of mongo pointer registry

+ add RegiteredEventlog type for storage in registry
+ add unique index with key 'deployment' to match struct tag
+ add SetPointer implementation
+ update LookupPointer to use new struct type

* mongo pointer registry tests

+ test creation of new Registry
+ test for addition, update and missing entry

* use keyed fields for go vet
2018-06-02 19:30:21 +02:00
Rob j Loranger
6eb326b4ac update pointer registry to include method for setting current pointer (#148) 2018-06-01 19:28:32 +00:00
Aaron Schlesinger
ae6cce8cd0 Adding a Pointer registry for Athens (#144)
* Adding a pointer registry interface

For use in storing event log pointers for 1 or more Olympus deployments

* Adding a merger strict

* Adding a storage deleter

The deleter will be used in event log delete operations

Reminder: delete operations will be exceptional (i.e. security
vulnerabilities)

* Adding a reference to a deleter in the merger

* adding docs to the event log merger

* adding docs to the event log pointer registry

* Adding an ErrNotFound error to the CDN package

* Adding a CDN deleter

* Adding more docs

* Making the merger use CDN actions only

* Removing the CDN type

It's not used anywhere

* Starting to build the merger

* remove out-of-scope changes

They'll be added as appropriate in follow-ups
2018-05-31 18:34:27 +00:00
Aaron Schlesinger
c0ebc40907 Adding an empty grift for doing pull synchronization for event logs (#143)
* Adding an empty grift for doing pull synchronization for event logs

Ref https://github.com/gomods/athens/issues/107

* Fixing typo
2018-05-31 13:21:19 +00:00
Rob j Loranger
c2887e8c7c remove dos line endings (#142) 2018-05-30 18:52:41 +02:00
Rob j Loranger
9df207561d Add .info to upload payload (#141)
* Add .info to upload payload
everywhere else as well

+ Upload payload includes Info
+ CLI uploads `version.info` as blob to storage
+ Getter and Saver methods updated for all storages implemented
+ Migration updated to include Info column
+ Tests no longer check RevInfo, now checking for Info same as Zip
+ Updated docs

* newline in .fizz

* change RevInfo.Info to []byte

+ change to []byte instead of io.Reader
+ adjust tests to check for info byte contents directly
+ add error check for minio client GetObject `pkg/storage/minio/getter.go`
2018-05-29 08:03:01 +02:00
Rob j Loranger
8cce792e55 Env readme (#140)
* remove carriage returns

* add environment variables and database set up

* make exported variable persistent
2018-05-20 20:48:18 +02:00
marpio
42ce1780b2 endpoint for cache misses from proxies (#137)
* endpoint for cache misses from proxies

* upgrade mongo to mitigate docker/for-linux/issues/58
allow running tests against postgres

* add test for cache miss route
2018-05-17 16:06:27 -07:00
marpio
d78252d527 Enpoint for fetching event log (#133)
* Enpoint for fetching event log

* sort listed module version

* review changes

* unify mongo env var name
2018-05-14 14:02:23 -07:00
Paul Burlumi
656d4b2549 Update README.md (#135)
Fixed a minor typo
2018-05-14 12:37:42 +02:00
marpio
8ceda38ad7 replace fs operations with afero (#123)
* replace fs operations with afero

* fix goget test

* fix test

* fix test again...

* remove empty lines

* fix merge

setupTmp - return 3, not 2 values

* just to start new travis build

got The command "go get github.com/golang/lint/golint" failed and exited with 1 during . I hope it was a temp problem.
2018-05-13 12:40:09 -07:00
Michal Pristas
7cd41ef236 Added support for eventlog (#129)
Eventlog is append-only log. 
Will be used in
- Olympus as a log of backed modules
- Proxy as a log of backed modules
- Proxy as a log of registered cache misses (clearable after read)

Multiple backing storages should be supported, for now, mongo is implemented. 
Preparation for Olympus reader, which is remote reader of Olympus event log, is in place without implementation, this implementation is for proxies to read logs from Olympus, and Olympus instances between each other (Pull synchronization)
2018-05-08 07:54:37 +02:00
marpio
cde250728a Implement Minio storage (#128)
* add minio dep

* implement minio storage

* fix travis.yml

* start minio in background
2018-05-07 18:13:11 -07:00
Chris Pickard
00ac591082 changed to using path.Join to build url (#122)
* added base-url option for cli

* changed to using path.Join to build url

also, fixed linting issue
2018-04-16 12:48:56 -04:00
Michal Pristas
ab89139c5d Generic vgo fetcher (#114)
Generic (service invariant) fetcher of go sources.
2018-04-03 20:36:06 +02:00
Chris Pickard
8fd44cac61 fixed a couple paths in docs (#115) 2018-03-30 16:25:45 -07:00
Michal Pristas
02b64609d4 fixed git fetcher (#109)
fixed git fetcher
2018-03-30 09:44:22 +02:00
lcd1232
700d6ee7af Add rdbms backend (#74)
* Add rdbms backend

* Remove unused imports

* Fix ci

* Fix problems

* Now test works

* Fix lint errors

* Change name to more descriptive

* Add logic to support rdbms storage

* Update to new newStorage logic
2018-03-29 14:23:08 -07:00
Michal Pristas
9c14b1fd12 Proposal: move buffalo app to cmd (#91)
* buffalo app moved to cmd

* ooops:

* added copy of proxy as olympus

* removed buffalo test

* added blank endpoint for feed

* forgot db config file and storage setup

* resolved comments

* newlines

* resolved conficts
2018-03-28 10:30:56 -04:00
Brian Ketelsen
e76125d3f1 Feature: Add OpenTracing (#112)
* tracing

* oops
2018-03-26 17:14:49 -04:00
marpio
0a3eb4c4ea fix golint issues (#110)
* fix stutters

* add comments

* add -set_exit_status flag to golint

* missing renames

* renames again
2018-03-26 11:18:18 -07:00
Aaron Schlesinger
b7e15a845c Mongo cdn getter (#62)
* Implementing a MongoDB backed CDN getter

Fixes https://github.com/gomods/athens/issues/49

* removing the base URL from the CDN getter
2018-03-23 13:41:22 -07:00
Aaron Schlesinger
a70ca0564c Fixing conflicts in the travis yaml file (#89)
* Fixing conflicts in the travis yaml file

* Removing dupes
2018-03-23 13:38:43 -07:00
Brian Ketelsen
cccaaab741 merge travis 2018-03-23 15:21:47 -04:00
Brian Ketelsen
8eb6fc9764 remove bad line endings 2018-03-23 15:21:16 -04:00
Aaron Schlesinger
cba1c45003 Submitting documentation that describes how to contribute (#86) 2018-03-22 18:03:21 -07:00
marpio
769f833999 add gofmt, golint and go vet to ci (#85)
* add gofmt, golint and go vet to ci

* result of gofmt -w .
2018-03-22 14:46:59 -07:00
Aaron Schlesinger
a6721f7505 Remove the base URL (#60)
* Removing concept of a baseURL

Just using a single ‘module’ and ‘version’ concept is enough

* go build => buffalo build
2018-03-21 16:03:44 -07:00
Brian Ketelsen
cf6a6f4e3f UI: Start UI (#35)
* UI: Start UI

* link to github auth

* save point

* simplify the UI a lot

* make a separate renderer for proxy vs registry

* get rid of debug

* update deps

* remove unneeded assets
2018-03-21 17:09:45 -04:00
Alexey Makhov
6220035bf6 gomod parsers tests (#72)
* gomod parsers tests

* gomod parsers tests. combining with #71

* gomod parsers tests. make it simpler

* gomod parsers tests. go fmt

* gomod parsers tests. add new case

* gomod parsers tests. add new case
2018-03-21 17:08:39 -04:00
marpio
2694401b7e Use afero for as abstraction layer for FS (#70)
* add afero dependency

* implement memory and disk storage using afero fs

* pass the appropriate fs

* remove unnecessary interface in the fs package
2018-03-20 18:39:55 -07:00
Michal Pristas
0a6e4407a1 Implemented backing mongo storage for modules (#64)
* mongo storage initial impl

* added tests

* mongo tests passing
2018-03-20 11:28:42 -07:00
Alexey Makhov
23f469ac18 fix assets manifest path (#67) 2018-03-19 11:20:08 -07:00
Alexey Makhov
d9d9fe1aed fix cdn saver interface (#66) 2018-03-18 22:39:15 -07:00