By default `ListObjectsV2()` returns the first 1000 objects matching the
list parameters. Normally this is fine, as it supports up to 333
versions (1000 / 3 files in proxy-triplet). For modules with more
versions, this is insufficient and must be upgraded to paginate.
Upgrades to the Go 1.22 toolchain. Upgrades golangci-lint to 1.61.0 and disables some new linters.
---------
Co-authored-by: Nicholas Wiersma <nick@wiersma.co.za>
Uses the new Azure Identity package for current support. Support automated Azure storage token refresh. Sets the default value of Athens storage account key and Azure managed identity resource id should be empty.
* gcp/saver: Only return errors.KindAlreadyExists if all three exist
In #1124, a GCP lock type was added as a singleflight backend. As part of this work, the GCP backend's Save() was made serial, likely because moduploader.Upload requires a call to Exists() before it, rendering the GCP lock less useful, by doubling the calls to GCS.
However, by doing this, the existence check was now only checking the existence of the mod file, and not the info or zip. This meant that if during a Save, the zip or info uploads failed, on subsequent rquests, that when using the GCP singleflight backend, Athens would assume everything had been stashed and saved properly, and then fail to serve up the info or zip that had failed upload, meaning the cache was in an unhealable broklen state, requiring a manual intervention.
To fix this, without breaking the singleflight behavior, introduce a metadata key that is set on the mod file during its initial upload, indicating that a Stash is still in progress on subsequent files, which gets removed once all three files are uploaded successfully, which can be checked if it it is determined that the mod file already exists. That way we can return a errors.KindAlreadyExists if a Stash is in progress, but also properly return it when a Stash is *not* currently in progress if and only if all three files exist on GCS, which prevents the cache from becoming permanently poisoned.
One note is that it is possible the GCS call to remove the metadata key fails, which would mean it is left on the mod object forever. To avoid this, consider it stale after 2 minutes.
---------
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
Co-authored-by: Matt <matt.ouille@protonmail.com>
* Using HeadObject to check the existence of a module
* fixing build
* calling cancel before closing channel
* fixing test
* using waitgroup
* calling cancel
* calling wg.Done
* only return the first error
* Revert "only return the first error"
This reverts commit c0aa18b522.
* clean err if not exist
S3 API sometimes returns duplicate keys, causing the number of file for a specific module greater than 3. This PR makes Athens handle that situation gracefully and log a warning message instead of treating the module as not found.
logging string content
Co-authored-by: Manu Gupta <manugupt1@gmail.com>
Co-authored-by: Ashish Ranjan <ashishranjan2912@gmail.com>
When using Minio storage, memory usage constantly increases. After
profiling, it appears that memory usage is building in the minio client,
and not being released.
Finding a similar issue listed in the minio/warp project, I found that
they fixed the issue by closing the *minio.Object instances. In the
athens usage, the minio.Object instances are never closed after reading
the data from them.
This adds deferred 'Close' calls in the 'storageImpl.Info' and
'storageImpl.GoMod' functions so that the objects may be released and
garbage collected. The 'storageImpl.Zip' function does not use defer,
since it has the object embedded into the return value, so the 'Close'
is only called on error.
* Fixing the build failures from a missing GCS Key
Accidentally introduced in https://github.com/gomods/athens/pull/1428
* Adding a TODO to the drone build
* adding a step to test the Athens server startup
* Adding the JSONKey back to the default config
* use temporary image
* Switching back to the old env var for the GCS JSON Key
cc/ @marwan-at-work
* removing the ServiceAccount. we want to stick with JSONKey
* reverting to original key fetching code
* fixing build err
* bumping chart ver
* Add GCS to helm chart
* base64 encode serviceAccount in the chart
* Standardize GCP Service Account references
* pipe storage.gcp.serviceAccount to quote
* Unescape key before attempting to split module and version
* PathUnescape key in minio/lister
* use unescaped path for module
* Compare againse escaped key
* Exctract common key parsing from cataloger and lister
* Add working image for minio
Between RELEASE.2019-09-26T19-42-35Z and RELEASE.2019-10-02T21-19-38Z
there's an issue with continuation tokens that's caused #1414 and #1413
to fail.
Signed-off-by: Chris M <me@christophermills.co.uk>
* Undo docker changes, update minio to v6
* Narrow issue down to catalog compliance test
Signed-off-by: Chris M <me@christophermills.co.uk>
* fix ignoring error
* fix operator
* fix minio error handling
* fix get url to return error if failed parsing
* add error handling
* position of statement to get redirect url was wrong.
* Updated mongo init to use db name env variable.
* Updated mongo init to use coll name env var.
* Fixed merge conflict.
* Updated formatting in mongo_test.
