mirror of
https://github.com/gomods/athens
synced 2026-02-03 12:10:32 +00:00
Link to docs (#440)
* Add link to our new docs site at the top of the readme * Update links to use docs site, not raw markdown
This commit is contained in:
Carolyn Van Slyck
committed by
Michal Pristas
Michal Pristas
parent
e44782fe0c
commit
e0a983ed38
+2
-2
@@ -55,8 +55,8 @@ Note that `make dev` only runs the minimum amount of dependencies needed for thi
|
||||
As you know from reading the [README](./README.md) (if you didn't read the whole thing, that's ok. Just read the
|
||||
introduction), the Athens project is made up of two components:
|
||||
|
||||
1. [Package Registry](./REGISTRY.md)
|
||||
2. [Edge Proxy](./PROXY.md)
|
||||
1. [Package Registry](https://docs.gomods.io/design/registry/)
|
||||
2. [Edge Proxy](https://docs.gomods.io/design/proxy/)
|
||||
|
||||
To run the registry:
|
||||
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
# The Athens Proxy
|
||||
|
||||
The Athens project has two components, the [central registry](./REGISTRY.md) and edge proxies.
|
||||
This document details the latter.
|
||||
|
||||
# The Role of the Proxy
|
||||
|
||||
We intend proxies to be deployed primarily inside of enterprises to:
|
||||
|
||||
- Host private modules
|
||||
- Exclude access to public modules
|
||||
- Cache public modules
|
||||
|
||||
Importantly, a proxy is not intended to be a complete _mirror_ of an upstream registry. For public modules, its role is to cache and provide access control.
|
||||
|
||||
# Proxy Details
|
||||
|
||||
First and foremost, a proxy exposes the same vgo download protocol as the registry. Since it doesn't have the multi-cloud requirements as the registry does, it supports simpler backend data storage mechanisms. We plan to release a proxy with several backends including:
|
||||
|
||||
- In-memory
|
||||
- Disk
|
||||
- RDBMS
|
||||
- Cloud blob storage
|
||||
|
||||
Users who want to target a proxy configure their `vgo` CLI to point to the proxy, and then execute commands as normal.
|
||||
|
||||
# Cache Misses
|
||||
|
||||
When a user requests a module `MxV1` from a proxy and the proxy doesn't have `MxV1` in its cache, it first determines whether `MxV1` is private or not private.
|
||||
|
||||
If it's private, it immediately does a cache fill operation from the internal VCS.
|
||||
|
||||
If it's not private, the proxy consults its exclude list for non-private modules (see below). If `MxV1` is on the exclude list, the proxy returns 404 and does nothing else. If `MxV1` is not on the exclude list, the proxy executes the following algorithm:
|
||||
|
||||
```
|
||||
registryDetails := lookupOnRegistry(MxV1)
|
||||
if registryDetails == nil {
|
||||
return 404 // if the registry doesn't have the thing, just bail out
|
||||
}
|
||||
return registryDetails.baseURL
|
||||
```
|
||||
|
||||
The important part of this algorithm is `lookupOnRegistry`. That function queries an endpoint on the registry that either:
|
||||
|
||||
- Returns 404 if it has `MxV1` in the registry
|
||||
- Returns the base URL for MxV1 if it has `MxV1` in the registry
|
||||
|
||||
Finally, if `MxV1` is fetched from a registry server, a background job will be created to periodically check `MxV1` for deletions and/or deprecations. In the event that one happens, the proxy will delete it from the cache.
|
||||
|
||||
_In a later version of the project, we may implement an event stream on the registry that the proxy can subscribe to and listen for deletions/deprecations on modules that it cares about_
|
||||
|
||||
# Exclude Lists and Private Module Filters
|
||||
|
||||
To accommodate private (i.e. enterprise) deployments, the proxy maintains two important access control mechanisms:
|
||||
|
||||
- Private module filters
|
||||
- Exclude lists for public modules
|
||||
|
||||
## Private Module Filters
|
||||
|
||||
Private module filters are string globs that tell the proxy what is a private module. For example, the string `github.internal.com/**` tells the proxy:
|
||||
|
||||
- To never make requests to the public internet (i.e. to the registry) regarding this module
|
||||
- To download module code (in its cache filling mechanism) from the VCS at `github.internal.com`
|
||||
|
||||
## Exclude Lists for Public Modules
|
||||
|
||||
Exclude lists for public modules are also globs that tell the proxy what modules it should never download from the registry. For example, the string `github.com/arschles/**` tells the proxy to always return `404 Not Found` to clients.
|
||||
@@ -4,10 +4,10 @@
|
||||
|
||||
Welcome to the Athens project! We're building all things Go package repository in here.
|
||||
|
||||
1. [Package Registry](./REGISTRY.md)
|
||||
2. [Edge Proxy](./PROXY.md)
|
||||
1. [Package Registry](https://docs.gomods.io/design/registry/)
|
||||
2. [Edge Proxy](https://docs.gomods.io/design/proxy/)
|
||||
|
||||
If you want more of a tl;dr on the project, check out [this quick introduction](https://medium.com/@arschles/project-athens-c80606497ce1)
|
||||
See our documentation site [https://docs.gomods.io](https://docs.gomods.io) for more details on the project.
|
||||
|
||||
# Project Status
|
||||
|
||||
@@ -67,8 +67,8 @@ If you're not ready to contribute code yet, there are plenty of other great ways
|
||||
- Get familiar with the system. There's lots to read about. Here are some places to start:
|
||||
- [Gentle Introduction to the Project](https://medium.com/@arschles/project-athens-c80606497ce1) - the basics of why we started this project
|
||||
- [The Download Protocol](https://medium.com/@arschles/project-athens-the-download-protocol-2b346926a818) - the core API that the registry and proxies implement and CLIs use to download packages
|
||||
- [Registry Design](./REGISTRY.md) - what the registry is and how it works
|
||||
- [Proxy Design](./PROXY.md) - what the proxy is and how it works
|
||||
- [Registry Design](https://docs.gomods.io/design/registry/) - what the registry is and how it works
|
||||
- [Proxy Design](https://docs.gomods.io/design/proxy/) - what the proxy is and how it works
|
||||
- [vgo wiki](https://github.com/golang/go/wiki/vgo) - context and details on how Go dependency management works in general
|
||||
- ["Go and Versioning"](https://research.swtch.com/vgo) - long papers on Go dependency management details, internals, etc...
|
||||
|
||||
|
||||
-214
@@ -1,214 +0,0 @@
|
||||
# The Athens Registry
|
||||
|
||||
Written by:
|
||||
|
||||
- [Aaron Schlesinger](https://github.com/arschles)
|
||||
- [Michal Pristas](https://github.com/michalpristas)
|
||||
- [Brian Ketelsen](https://github.com/bketelsen)
|
||||
|
||||
The Athens registry is a Go package registry service that is hosted globally across multiple cloud providers. The **global deployment** will have a DNS name (i.e. `registry.golang.org`) that round-robins across each **cloud deployment**. We will use the following **cloud deployments** for _example only_ in this document:
|
||||
|
||||
- Microsoft Azure (hosted at `microsoft.registry.golang.org`)
|
||||
- Google Cloud (hosted at `google.registry.golang.org`)
|
||||
- Amazon AWS (hosted at `amazon.registry.golang.org`)
|
||||
|
||||
Regardless of which **cloud deployment** is routed to, the **global deployment** must provide up-to-date (precise definition below) module metadata & code.
|
||||
|
||||
We intend to create a foundation (the TBD foundation) that manages **global deployment** logistics and governs how each **cloud deployment** participates.
|
||||
|
||||
# Glossary
|
||||
|
||||
In this document, we will use the following keywords and symbols:
|
||||
|
||||
- `OA` - the registry **cloud deployment** hosted on Amazon AWS
|
||||
- `OG` - the registry **cloud deployment** hosted on Google Cloud
|
||||
- `OM` - the registry **cloud deployment** hosted on Microsoft Azure
|
||||
- `MxVy` - the module `x` at version `y`
|
||||
|
||||
# Properties of the Registry
|
||||
|
||||
The registry should obey the following invariants:
|
||||
|
||||
- No existing module or version should ever be deleted or modified
|
||||
- Except for exceptional cases, like a DMCA takedown (more below)
|
||||
- Module metadata & code may be eventually consistent across **cloud deployments**
|
||||
|
||||
These properties are both important to design the **global deployment** and to ensure repeatable builds in the Go community as much as is possible.
|
||||
|
||||
# Technical Challenges
|
||||
|
||||
A registry **cloud deployment** has two major concerns:
|
||||
|
||||
- Sharing module metadata & code
|
||||
- Staying current with what other registry **cloud deployment**s are available
|
||||
|
||||
For the rest of this document, we’ll refer to these concerns as **data exchange** and **membership**, respectively.
|
||||
|
||||
Registries will use separate protocols to do **data exchange** and **membership**.
|
||||
|
||||
# Data Exchange
|
||||
|
||||
The overall design of the **global deployment** should ensure the following:
|
||||
|
||||
- Module metadata and code is fetched from the appropriate source (i.e. a VCS)
|
||||
- Module metadata and code is replicated across all **cloud deployment**s. As previously stated, replication may be eventually consistent.
|
||||
|
||||
Each **cloud deployment** holds:
|
||||
|
||||
- A module metadata database
|
||||
- A log of actions it has taken on the database (used to version the module database)
|
||||
- Actual module source code and metadata
|
||||
- This is what vgo requests
|
||||
- Likely stored in a CDN
|
||||
|
||||
The module database holds metadata and code for all modules that the cloud deployment is aware of, and the log records all the operations the cloud deployment has done in its lifetime.
|
||||
|
||||
# The Module Database
|
||||
|
||||
The module database is made up of two components:
|
||||
|
||||
- A blob storage system (usually a CDN) that holds module metadata and source code
|
||||
- This is called the module CDN
|
||||
- A key/value store that indicates whether and where a module MxV1 exists in the **cloud deployment**'s blob storage
|
||||
- This is called the module metadata database, or key/value storage
|
||||
|
||||
If a **cloud deployment** OM holds modules `MxV1`, `MxV2` and `MyV1`, its module metadata database would look like the following:
|
||||
|
||||
```
|
||||
Mx: {baseLocation: mycdn.com/Mx}
|
||||
My: {baseLocation: mycdn.com/My}
|
||||
```
|
||||
|
||||
Note that `baseLocation` is intended for use in the `<meta>` redirect response passed to vgo. As a result, it may point to other **cloud deployment** blob storage systems. More information on that in the synching sections below.
|
||||
|
||||
# The Log
|
||||
|
||||
The log is an append-only record of actions that a **cloud deployment** OM has taken on its module database. The log exists only to facilitate module replication between **cloud deployment**s (more on how replication below).
|
||||
|
||||
Below is an example event log:
|
||||
|
||||
```
|
||||
ADD MxV1 ID1
|
||||
ADD MxV2 ID2
|
||||
ADD MyV1.5 ID3
|
||||
```
|
||||
|
||||
This log corresponds to a database that looks like the following:
|
||||
|
||||
```
|
||||
Mx: {baseLocation: mycdn.com/Mx}
|
||||
My: {baseLocation: mycdn.com/My}
|
||||
```
|
||||
|
||||
And blob storage that holds versions 1 and 2 of Mx and version 1.5 of My.
|
||||
|
||||
## Log IDs
|
||||
|
||||
Note that each event log line holds ID data (`ID1`, `ID2`, etc...). These IDs are used to by other **cloud deployment**s as database versions. Details on how these IDs are used are below in the pull sync section.
|
||||
|
||||
# Cache Misses
|
||||
|
||||
If an individual **cloud deployment** OM gets a request for a module MxV1 that is not in its database, it returns a "not found" (i.e. HTTP 404) response to vgo. Then, the following happens:
|
||||
|
||||
- OM starts a background cache fill operation to look for MxV1 on OA and OG
|
||||
- If OA and OG both report a miss, OM does a cache fill operation from the VCS and does a push synchronization (see below)
|
||||
- vgo downloads code directly from the VCS on the client's machine
|
||||
|
||||
# Pull Sync
|
||||
|
||||
Each **cloud deployment** will actively sync its database with the others. Every timer tick `T a **cloud deployment** OM will query another **cloud deployment** OA for all the modules that changed or were added since the last time OM synched with OA.
|
||||
|
||||
## Query Mechanism
|
||||
|
||||
The query obviously relies on OA being able to provide deltas of its database over logical time. Logical time is communicated between OM and OA with log IDs (described above). The query algorithm is approximately:
|
||||
|
||||
```
|
||||
lastID := getLastQueriedID(OA)
|
||||
newDB, newID := query(OA, lastID) // get the new operations that happened on OA's database since lastID
|
||||
mergeDB(newDB) // merge newDB into my own DB
|
||||
storeLastQueriedID(OA, newID) // after this, getLastQueriedID(OA) will return newID
|
||||
```
|
||||
|
||||
The two most important parts of this algorithm are the `newDB` response and the `mergeDB` function.
|
||||
|
||||
### Database Diffs
|
||||
|
||||
OA uses its database log to construct a database diff starting from the `lastID` value that it receives from OM. It then sends the diff to OM in JSON that looks like the following:
|
||||
|
||||
```json
|
||||
{
|
||||
"added": ["MxV1", "MxV2", "MyV1"],
|
||||
"deleted": ["MaB1", "MbV2"],
|
||||
"deprecated": ["MdB1"]
|
||||
}
|
||||
```
|
||||
|
||||
Explicitly, this structure indicates that:
|
||||
|
||||
- `MxV1`, `MxV2` and `MyV1` were added since `lastID`
|
||||
- `MaB1` and `MbV2` were deleted since `lastID`
|
||||
- `MdB1` was deprecated since `lastID`
|
||||
|
||||
### Database Merging
|
||||
|
||||
The `mergeDB` algorithm above receives a database diff and merges the new entries into its own database. It follows a few rules:
|
||||
|
||||
- Deletes insert a tombstone into the database
|
||||
- If a module `MdV1` is tombstoned, all future operations that come via database diffs are sent to `/dev/null`
|
||||
- If module `MdV2` is deprecated, future add or deprecation diffs for `MdV2` are sent to `/dev/null`. Future delete operations can still tombstone
|
||||
|
||||
The approximate algorithm for `mergeDB` is this:
|
||||
|
||||
```
|
||||
func mergeDB(newDB) {
|
||||
for added in newDB.added {
|
||||
fromDB := lookup(added)
|
||||
if fromDB != nil {
|
||||
break // the module already exists (it may be deprecated or tombstoned), bail out
|
||||
}
|
||||
addToDB(added) // this adds the module to the module db's key/value store, but points baseLocation to the other cloud deployment's blob storage
|
||||
go downloadCode(added) // this downloads the module to local blob storage, then updates the key/value store's baseLocation accordingly
|
||||
}
|
||||
for deprecated in newDB.deprecated {
|
||||
fromDB := lookup(deprecated)
|
||||
if fromDB.deleted() {
|
||||
break // can't deprecated something that's already deleted
|
||||
}
|
||||
deprecateInDB(deprecated) // importantly, this function inserts a deprecation record into the DB even if the module wasn't already present!
|
||||
}
|
||||
for deleted in newDB.deleted {
|
||||
deleteInDB(deleted) // importantly, this function inserts a tombstone into the DB even if the module wasn't already present!
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
# Push Sync
|
||||
|
||||
If a **cloud deployment** OM has a cache miss on a module MxV1, does a cache fill operation and discovers that no other **cloud deployment** OG or OA have MxV1, it fills from the VCS. After it finishes the fill operation, it saves the module code and metadata to its module database and adds a log entry for it. The algorithm look like the following:
|
||||
|
||||
```
|
||||
newCode := fillFromVCS(MxV1)
|
||||
storeInDB(newCode)
|
||||
storeInLog(newCode)
|
||||
pushTo(OA, newCode) // retry and give up after N failures
|
||||
pushTo(OG, newCode) // retry and give up after N failures
|
||||
```
|
||||
|
||||
The `pushTo` function is most important in this algorithm. It _only_ sends the existence of a new module, but no event log metadata (i.e. `lastID`):
|
||||
|
||||
```
|
||||
func pushTo(OA, newCode) {
|
||||
http.POST(OA, newCode.moduleName, newCode.moduleVersion, "https://OM.com/fetch")
|
||||
}
|
||||
```
|
||||
|
||||
The endpoint in OA that receives the HTTP `POST` request in turn does the following:
|
||||
|
||||
```
|
||||
func receive(moduleName, moduleVersion, fetchURL) {
|
||||
addToDB(moduleName, moduleVersion, OM) // stores moduleName and moduleVersion in the key/value store, with baseLocation pointing to OM
|
||||
go downloadCode(added) // this downloads the module to local blob storage, then updates the key/value store's baseLocation accordingly
|
||||
```
|
||||
|
||||
Note again that `lastID` is not sent. Future pull syncs that OA does from OM will receive moduleName/moduleVersion in the 'added' section, and OA will properly do nothing because it already has moduleName/moduleVersion.
|
||||
@@ -5,7 +5,7 @@ date: 2018-02-11T15:59:56-05:00
|
||||
|
||||
## The Athens Proxy
|
||||
|
||||
The Athens project has two components, the [central registry](./REGISTRY.md) and edge proxies.
|
||||
The Athens project has two components, the [central registry](/design/registry/) and edge proxies.
|
||||
This document details the latter.
|
||||
|
||||
## The Role of the Proxy
|
||||
|
||||
Reference in New Issue
Block a user