* Add support for Redis Username and Password configuration
Introduced Redis master authentication parameters (username and password) to the Redis Sentinel setup. This enhances compatibility with Redis environments that require authentication for both sentinel and master nodes.
* Add support for protected Redis Sentinel configuration and related unit tests
* gcp/saver: Only return errors.KindAlreadyExists if all three exist
In #1124, a GCP lock type was added as a singleflight backend. As part of this work, the GCP backend's Save() was made serial, likely because moduploader.Upload requires a call to Exists() before it, rendering the GCP lock less useful, by doubling the calls to GCS.
However, by doing this, the existence check was now only checking the existence of the mod file, and not the info or zip. This meant that if during a Save, the zip or info uploads failed, on subsequent rquests, that when using the GCP singleflight backend, Athens would assume everything had been stashed and saved properly, and then fail to serve up the info or zip that had failed upload, meaning the cache was in an unhealable broklen state, requiring a manual intervention.
To fix this, without breaking the singleflight behavior, introduce a metadata key that is set on the mod file during its initial upload, indicating that a Stash is still in progress on subsequent files, which gets removed once all three files are uploaded successfully, which can be checked if it it is determined that the mod file already exists. That way we can return a errors.KindAlreadyExists if a Stash is in progress, but also properly return it when a Stash is *not* currently in progress if and only if all three files exist on GCS, which prevents the cache from becoming permanently poisoned.
One note is that it is possible the GCS call to remove the metadata key fails, which would mean it is left on the mod object forever. To avoid this, consider it stale after 2 minutes.
---------
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
Co-authored-by: Matt <matt.ouille@protonmail.com>
A default homepage is baked into the server that uses the request host address, or in HTTP 2 the authority. This includes ports. It also checks for schema. The values are used to indicate to users how to configure their go env
Of course, this won't work on all installations - especially enterprise ones. For that, we've introduced ATHENS_HOME_TEMPLATE_PATH as an environment variable along with HomeTemplatePath in the config. This value defaults to /var/lib/athens/home.html but can be configured to any location that Athens can reliably read from. This is a Go HTML template so it should use Go HTML template formatting and logic.
* read redis lock options from config to support custom TTL & timeout
* fix test
* fix typo
* downgrade to bsm/redislock@v0.7.2 to prevent usage of beta go-redis version
* revert test changes
* return error for invalid lock config
* update config parsing test
* udpate docs to include redis lock config
* fix test
* set default max retries to 10
* reduce default redis lock timeout to 15s
* update default TTL to 15mins
Co-authored-by: Manu Gupta <manugupt1@gmail.com>
* Reverting PR 1650
https://github.com/gomods/athens/pull/1650 was a big change to the
authentication/authorization code, which we have decided to pull
out and potentially move into a separate process/project
Signed-off-by: Aaron Schlesinger <aaron@ecomaz.net>
* removing commented, unused code
Signed-off-by: Aaron Schlesinger <aaron@ecomaz.net>
* removing more commented, unused code
Signed-off-by: Aaron Schlesinger <aaron@ecomaz.net>
* removing more unused code
Signed-off-by: Aaron Schlesinger <aaron@ecomaz.net>
* Fix sumdb/* paths when config.PathPrefix is set
http.StripPrefix will look at the entire request path when called,
if we do not include config.PathPrefix then the StripPrefix call
will never receive a valid path from the application and the user
will always get a 404 error.
There were no test where I could easily check this regression so
I also added a few endpoint tests, the last test will fail with
a 404 instead of 403 if this change in not applied.
* Update cmd/proxy/actions/app_proxy.go
Co-authored-by: Marwan Sulaiman <marwan-at-work@github.com>
* Update cmd/proxy/actions/app_proxy_test.go
Co-authored-by: Marwan Sulaiman <marwan-at-work@github.com>
* Update cmd/proxy/actions/app_proxy_test.go
Co-authored-by: Marwan Sulaiman <marwan-at-work@github.com>
* Update cmd/proxy/actions/app_proxy_test.go
Co-authored-by: Marwan Sulaiman <marwan-at-work@github.com>
* Removed unneeded import of logrus
Co-authored-by: Marwan Sulaiman <marwan-at-work@github.com>
* add config options for redis password
* redis password test and failure test
* changed redis name and made a minimal redis.conf
add drone volume
volume fix
remove volumes from drone.
repoint redis path
Co-authored-by: Aaron Schlesinger <70865+arschles@users.noreply.github.com>
Co-authored-by: Marwan Sulaiman <marwan.sameer@gmail.com>
* cmd/proxy/actions: pass NoSumPatterns to GoGetFetcher
* add gosum unit tests
* upgrade appveyor to 1.13
* fix test typo
* check appveyor version
* AppVeyor has no 1.13
* use logger
* actions: support proxying sum db urls
* remove proxy prefix
* add docs for checksum db
* more docs
* typo
* typo
* typo
* typo
* typo
* typo
* typo
* typo
* move checksum db into its own section
* add azblob sf stasher
* use http status const
* rm line
* cleanup, comments
* add test
* fix test
* skip test it no account key provided
* fix stash
* introduce stash timeout
* Adding readiness endpoint for readinessProbe in k8s
* liveness and readiness probes added to deployment template
* Changing endpoint to /readyz, onelining checks
* Removing DNS check as that will be covered by storage.List() call
* merge list from storage with the one from go list
* fix and rename
* move stuff to semver pkg
* fix gofmt
* move union out of semver pkg
* add tests
* fix err msg
* fix tests
* error handling
* error handling, go list refactoring
* fix list
* cleanup
* fix gofmt
* switch proxy to config file
pull in single flight changes
* changes for single-flight
* intermediate stage. All tests passing. pkg still has env refs
* remove all env references
* delete config/env entirely
* fix failing tests
* create the config.toml file as part of dev setup
* create config file only if it doesn't exist
* update Dockerfiles to use config file
* move composing elements to the top
* verbose parameter naming
* newline
* add flag for config file path
* update docs with config file flag
* remove unnecessary nil check
* use filepath.join
* rename redis port to address
* fix path.join
* fix issues after merge
* add vendor dir
* Goget fetcher should error out if gobinpath is not valid
* Propogating the error from goget initialisation to main, to stop application
* wrapping errors with op, using exec.command(gobin).Run to verify gobin, clean up
* Inlining afero fs to goget call, and new go getfetcher
* Revert "Inlining afero fs to goget call, and new go getfetcher"
This reverts commit ae31fe6a2b.
* Fixing example test
* add middlewares to fill the cache and to populate contexts with module and versions
* Carolyn tries to fix Aaron's code
* #DOINSTUFF
* updated env var
* fix all the things
* magic patch for olympus
* Add latest handler
* Remove deprecated return param
* go fmt all the things
* download: add list from go cli
* download: include goget tests + hacky hack
* download: move dummyMod to pkg/module
* Olympus: pass dp and lggr to /list
* download: add Version to interface
* download: document Protocol
* Athens: introduce pkg/errors
* errors: fix Ops loop
* pkg/errors: introduce M and V types to get rid of unstable parsing
* fix build
* errors: fix text + add tests
* pr updates
