[docs registries] Add reference documentation (microsoft#17672)

* using registries reference documentation

* start work on "creating registries" document

also fix minor issue in "using registries" doc

* Robert's comments, continue working

* finish creating registries docs

* add links to example registries

* aupopa cr

.gitignore

@@ -10,6 +10,7 @@
 *.user
 *.userosscache
 *.sln.docstates
+/vcpkg-configuration.json
 
 # fuzzing
 sync_dir*

docs/README.md

@@ -23,6 +23,7 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too
 - [Usage with Android](users/android.md)
 - [Usage with Mingw-w64](users/mingw.md)
 - [Host Dependencies](users/host-dependencies.md)
+- [Using Registries](users/registries.md)
 
 ### Maintainer Help
 
@@ -31,6 +32,7 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too
 - [Portfile functions](maintainers/portfile-functions.md)
 - [Common CMake definitions](maintainers/vcpkg_common_definitions.md)
 - [Maintainer Guidelines](maintainers/maintainer-guide.md)
+- [Creating Registries](maintainers/registries.md)
 
 ### [Vcpkg-Tool](https://github.com/microsoft/vcpkg-tool) Maintainer Help
 

docs/maintainers/registries.md

@@ -0,0 +1,357 @@
+# Creating Registries
+
+**The latest version of this documentation is available on [GitHub](https://github.com/Microsoft/vcpkg/tree/master/docs/maintainers/registries.md).**
+
+There are two parts to using registries; this documents the creation side of
+the relationship. In order to learn more about using registries that others
+have created, please read [this documentation](../users/registries.md).
+## Table of Contents
+
+- [Creating Registries](#creating-registries)
+  - [Table of Contents](#table-of-contents)
+  - [Overview](#overview)
+    - [Git Registries](#git-registries)
+      - [Adding a New Version](#adding-a-new-version)
+    - [Filesystem Registries](#filesystem-registries)
+      - [Adding a New Version](#adding-a-new-version-1)
+
+## Overview
+
+Registries are collections of ports and their versions. There are two major
+choices of implementation for registries, if you want to create your own -
+git registries, and filesystem registries.
+
+Git registries are simple git repositories, and can be shared publicly or
+privately via normal mechanisms for git repositories. The vcpkg repository at
+<https://github.com/microsoft/vcpkg>, for example, is a git registry.
+
+Filesystem registries are designed as more of a testing ground. Given that they
+literally live on your filesystem, the only way to share them is via shared
+directories. However, filesystem registries can be useful as a way to represent
+registries held in non-git version control systems, assuming one has some way
+to get the registry onto the disk.
+
+Note that we expect the set of registry types to grow over time; if you would
+like support for registries built in your favorite public version control
+system, don't hesitate to open a PR.
+
+The basic structure of a registry is:
+
+- The set of versions that are considered "latest" at certain times in history,
+  known as the "baseline".
+- The set of all the versions of all the ports, and where to find each of
+  these in the registry.
+
+### Git Registries
+
+As you're following along with this documentation, it may be helpful to have
+a working example to refer to. We've written one and put it here:
+<https://github.com/northwindtraders/vcpkg-registry>.
+
+All git registries must have a `versions/baseline.json` file. This file
+contains the set of "latest versions" at a certain commit. It is laid out as
+a top-level object containing only the `"default"` field. This field should
+contain an object mapping port names to the version which is currently the
+latest.
+
+Here's an example of a valid baseline.json:
+
+```json
+{
+  "default": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  }
+}
+```
+
+The `versions` directory contains all the information about which versions of
+which packages are contained in the registry, along with where those versions
+are stored. The rest of the registry just acts as a backing store, as far as
+vcpkg is concerned: only things inside the `versions` directory will be used
+to direct how your registry is seen by vcpkg.
+
+Each port in a registry should exist in the versions directory as
+`<first letter of port>-/<name of port>.json`; in other words, the
+information about the `kitten` port would be located in
+`versions/k-/kitten.json`. This should be a top-level object with only a
+single field: `"versions"`. This field should contain an array of version 
+objects:
+
+- The version of the port in question; should be exactly the same as the
+  `vcpkg.json` file, including the version fields and `"port-version"`.
+- The `"git-tree"` field, which is a git tree; in other words, what you get
+  when you write `git rev-parse COMMIT-ID:path/to/port`.
+
+Note that the version fields for ports with `CONTROL` files, is 
+`"version-string"`; we do not recommend using `CONTROL` files in new
+registries, however.
+
+_WARNING_: One very important part of registries is that versions should
+_never_ be changed. Updating to a later ref should never remove or change an
+existing version. It must always be safe to update a registry.
+
+Here's an example of a valid version database for a `kitten` port with one 
+version:
+
+```json
+{
+  "versions": [
+    {
+      "version": "2.6.2",
+      "port-version": 0,
+      "git-tree": "67d60699c271b7716279fdea5a5c6543929eb90e"
+    }
+  ]
+}
+```
+
+In general, it's not important where you place port directories. However, the
+idiom in vcpkg is to follow what the built in vcpkg registry does: your 
+`kitten` port should be placed in `ports/kitten`.
+
+_WARNING_: One other thing to keep in mind is that when you update a registry,
+all previous versions should also be accessible. Since your user will set their
+baseline to a commit ID, that commit ID must always exist, and be accessible
+from your HEAD commit, which is what is actually fetched. This means that your
+HEAD commit should be a child of all previous HEAD commits.
+
+#### Adding a New Version
+
+There is some git trickery involved in creating a new version of a port. The
+first thing to do is make some changes, update the `"port-version"` and regular
+version field as you need to, and then test with `overlay-ports`:
+`vcpkg install kitten --overlay-ports=ports/kitten`.
+
+Once you've finished your testing, you'll need to make sure that the directory
+as it is is under git's purview. You'll do this by creating a temporary commit:
+
+```pwsh
+> git add ports/kitten
+> git commit -m 'temporary commit'
+```
+
+Then, get the git tree ID of the directory:
+
+```pwsh
+> git rev-parse HEAD:ports/kitten
+73ad3c823ef701c37421b450a34271d6beaf7b07
+```
+
+Then, you can add this version to the versions database. At the top of your
+`versions/k-/kitten.json`, you can add (assuming you're adding version
+`2.6.3#0`):
+
+```json
+{
+  "versions": [
+    {
+      "version": "2.6.3",
+      "port-version": 0,
+      "git-tree": "73ad3c823ef701c37421b450a34271d6beaf7b07"
+    },
+    {
+      "version": "2.6.2",
+      "port-version": 0,
+      "git-tree": "67d60699c271b7716279fdea5a5c6543929eb90e"
+    }
+  ]
+}
+```
+
+then, you'll want to modify your `versions/baseline.json` with your new version 
+as well:
+
+```json
+{
+  "default": {
+    "kitten": {
+      "baseline": "2.6.3",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  }
+}
+```
+
+and amend your current commit:
+
+```pwsh
+> git commit --amend
+```
+
+then share away!
+
+### Filesystem Registries
+
+As you're following along with this documentation, it may be helpful to have
+a working example to refer to. We've written one and put it here:
+<https://github.com/vcpkg/example-filesystem-registry>.
+
+All filesystem registries must have a `versions/baseline.json` file. This file
+contains the set of "latest versions" for a certain version of the registry.
+It is laid out as a top-level object containing a map from version name to
+"baseline objects", which map port names to the version which is considered
+"latest" for that version of the registry.
+
+Filesystem registries need to decide on a versioning scheme. Unlike git 
+registries, which have the implicit versioning scheme of refs, filesystem
+registries can't rely on the version control system here. One possible option
+is to do a daily release, and have your "versions" be dates.
+
+_WARNING_: A baseline must always refer to the same set of versions. If you
+want to add new versions, you need to create a new version of the registry in
+the `baseline.json` file.
+
+Here's an example of a valid `baseline.json`, for a registry that has decided
+upon dates for their versions:
+
+```json
+{
+  "2021-04-16": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  },
+  "2021-04-15": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 1
+    }
+  }
+}
+```
+
+The `versions` directory contains all the information about which versions of
+which packages are contained in the registry, along with where those versions
+are stored. The rest of the registry just acts as a backing store, as far as
+vcpkg is concerned: only things inside the `versions` directory will be used
+to direct how your registry is seen by vcpkg.
+
+Each port in a registry should exist in the versions directory as
+`<first letter of port>-/<name of port>.json`; in other words, the
+information about the `kitten` port would be located in
+`versions/k-/kitten.json`. This should be a top-level object with only a
+single field: `"versions"`. This field should contain an array of version 
+objects:
+
+- The version of the port in question; should be exactly the same as the
+  `vcpkg.json` file, including the version fields and `"port-version"`.
+- The `"path"` field: a relative directory, rooted at the base of the registry
+  (in other words, the directory where `versions` is located), to the port 
+  directory. It should look something like `"$/path/to/port/dir`"
+
+Note that the version fields for ports with `CONTROL` files, is 
+`"version-string"`; we do not recommend using `CONTROL` files in new
+registries, however.
+
+In general, it's not important where you place port directories. However, the
+idiom in vcpkg is to follow somewhat closely to what the built in vcpkg
+registry does: your `kitten` port at version `x.y.z` should be placed in
+`ports/kitten/x.y.z`, with port versions appended as you see fit (although
+since `#` is not a good character to use for file names, perhaps use `_`).
+
+_WARNING_: One very important part of registries is that versions should
+_never_ be changed. One should never remove or change an existing version.
+Your changes to your registry shouldn't change behavior to downstream users.
+
+Here's an example of a valid version database for a `kitten` port with one 
+version:
+
+```json
+{
+  "versions": [
+    {
+      "version": "2.6.2",
+      "port-version": 0,
+      "git-tree": "$/ports/kitten/2.6.2_0"
+    }
+  ]
+}
+```
+
+#### Adding a New Version
+
+Unlike git registries, adding a new version to a filesystem registry mostly
+involves a lot of copying. The first thing to do is to copy the latest
+version of your port into a new version directory, update the version and
+`"port-version"` fields as you need to, and then test with `overlay-ports`:
+`vcpkg install kitten --overlay-ports=ports/kitten/new-version`.
+
+Once you've finished your testing, you can add this new version to the top of
+your `versions/k-/kitten.json`:
+
+```json
+{
+  "versions": [
+    {
+      "version": "2.6.3",
+      "port-version": 0,
+      "git-tree": "$/ports/kitten/2.6.3_0"
+    },
+    {
+      "version": "2.6.2",
+      "port-version": 0,
+      "git-tree": "$/ports/kitten/2.6.2_0"
+    }
+  ]
+}
+```
+
+then, you'll want to modify your `versions/baseline.json` with your new version 
+as well (remember not to modify existing baselines):
+
+```json
+{
+  "2021-04-17": {
+    "kitten": {
+      "baseline": "2.6.3",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  },
+  "2021-04-16": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  },
+  "2021-04-15": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 1
+    }
+  }
+}
+```
+
+and you're done!