[Docs] Add developer guide section (conda#11041)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jannis Leidel <[email protected]>

.github/workflows/ci-images.yml

@@ -5,12 +5,12 @@ on:
     - cron:  '0 0 * * 0'
   pull_request:
     paths:
-      - 'ci/Dockerfile'
+      - 'Dockerfile'
       - '.github/workflows/ci-images.yml'
       - 'tests/requirements.txt'
   push:
     paths:
-      - 'ci/Dockerfile'
+      - 'Dockerfile'
       - '.github/workflows/ci-images.yml'
       - 'tests/requirements.txt'
   workflow_dispatch:
@@ -50,8 +50,8 @@ jobs:
       - name: Build & push Python ${{ matrix.python }}
         uses: docker/build-push-action@v2
         with:
-          file: ./ci/Dockerfile
-          build-args: PYTHON_VERSION=${{ matrix.python }}
+          file: ./Dockerfile
+          build-args: python_version=${{ matrix.python }}
           labels: ${{ steps.meta.outputs.labels }}
           tags: ${{ steps.meta.outputs.tags }}
           push: ${{ github.ref == 'refs/heads/master' }}

.github/workflows/ci.yml

@@ -42,12 +42,12 @@ jobs:
     - name: Setup environment
       shell: cmd
       run: |
-        call .\ci\scripts\windows\setup.bat
+        call .\dev\windows\setup.bat
 
     - name: Python ${{ matrix.python-version }}, ${{ matrix.conda-subdir }} ${{ matrix.test-type }} tests, group ${{ matrix.test-group }}
       shell: cmd
       run: |
-        call .\ci\scripts\windows\${{ matrix.test-type }}.bat
+        call .\dev\windows\${{ matrix.test-type }}.bat
 
     - uses: codecov/codecov-action@v2
       with:
@@ -79,7 +79,7 @@ jobs:
           -e TEST_SPLITS
           -e TEST_GROUP
           ghcr.io/conda/conda-ci:master-linux-python${{ matrix.python-version }}
-          /opt/conda-src/ci/scripts/linux/${{ matrix.test-type }}.sh
+          /opt/conda-src/dev/linux/${{ matrix.test-type }}.sh
 
       - uses: codecov/codecov-action@v2
         with:

CONTRIBUTING.md

@@ -8,8 +8,7 @@ If your issue is a bug report or feature request for:
 * **anaconda.org**: please file it at <https://anaconda.org/contact/report>
 * **repo.anaconda.com**: please file it at <https://github.com/ContinuumIO/anaconda-issues/issues>
 * **commands under `conda build`**: please file it at <https://github.com/conda/conda-build/issues>
-* **commands under `conda env`**: please file it here!
-* **all other conda commands**: please file it here!
+* **commands under `conda env` and all other conda commands**: please file it at <https://github.com/conda/conda/issues>
 
 ## Code of Conduct
 
@@ -58,7 +57,7 @@ The conda organization adheres to the [NumFOCUS Code of Conduct](https://www.num
    > git remote add upstream [email protected]:conda/conda
    ```
 
-3. Create a local development environment and activate that environment
+3. One option is to create a local development environment and activate that environment
 
    **Bash (macOS, Linux, Windows)**
 
@@ -81,6 +80,54 @@ The conda organization adheres to the [NumFOCUS Code of Conduct](https://www.num
    directory, look at the value of `conda location:` in the output of
    `conda info --all`.
 
+4. Alternatively, for Linux development only, you can use the same Docker
+   image the CI pipelines use. Note that you can run this from all three
+   operating systems! We are using `docker compose`, which provides three
+   actions for you:
+
+   - `unit-tests`: Run all unit tests.
+   - `integration-tests`: Run all integration tests.
+   - `interactive`: You are dropped in a pre-initialized Bash session,
+     where you can run all your `pytest` commands as required.
+
+   Use them with `docker compose run <action>`. For example:
+
+
+   **Any shell (macOS, Linux, Windows)**
+
+   ```bash
+   $ docker compose run unit-tests
+   ```
+
+   This builds the same Docker image as used in continuous
+   integration from the [Github Container Registry](https://github.com/conda/conda/pkgs/container/conda-ci)
+   and starts `bash` with the conda development mode already enabled.
+   By default, it will use Python 3.9 installation.
+
+   If you need a different Python version, set a `CONDA_DOCKER_PYTHON`
+   environment variable like this to rebuild the image. You might need
+   to add `--no-cache` to make sure the image is rebuilt.
+
+   **Bash (macOS, Linux, Windows)**
+
+   ```bash
+   $ CONDA_DOCKER_PYTHON=3.8 docker compose build --no-cache unit-tests
+   ```
+
+   **cmd.exe (Windows)**
+
+   ```batch
+   > set CONDA_DOCKER_PYTHON=3.8 && docker compose build --no-cache unit-tests && set "CONDA_DOCKER_PYTHON="
+   ```
+
+   The next time you run `docker compose run <task>` you will use the new image.
+   If you want to revert to the version you were previously using, you need to rebuild
+   the image again.
+
+>  The `conda` repository will be mounted to `/opt/conda-src`, so all changes
+   done in your editor will be reflected live while the Docker container is
+   running.
+
 ## Static Code Analysis
 
 This project is configured with [pre-commit](https://pre-commit.com/) to
@@ -101,6 +148,8 @@ what you need to get started below:
 ```bash
 # reuse the development environment created above
 $ source ./dev/start
+# or start the Docker image in interactive mode
+# $ docker compose run interactive
 
 # install pre-commit hooks for conda
 $ cd "$CONDA_PROJECT_ROOT"
@@ -119,6 +168,8 @@ $ git commit
 ```batch
 :: reuse the development environment created above
 > .\dev\start.bat
+:: or start the Docker image in interactive mode
+:: > docker compose run interactive
 
 :: install pre-commit hooks for conda
 > cd "%CONDA_PROJECT_ROOT%"
@@ -156,6 +207,8 @@ but generally speaking all you need is the following:
 ```bash
 # reuse the development environment created above
 $ source ./dev/start
+# or start the Docker image in interactive mode
+# $ docker compose run interactive
 
 # run conda's unit tests using GNU make
 $ make unit
@@ -172,6 +225,8 @@ $ pytest tests/test_create.py -k create_install_update_remove_smoketest
 ```batch
 :: reuse the development environment created above
 > .\dev\start.bat
+:: or start the Docker image in interactive mode
+:: > docker compose run interactive
 
 :: run conda's unit tests with pytest
 > pytest -m "not integration" conda tests
@@ -180,6 +235,20 @@ $ pytest tests/test_create.py -k create_install_update_remove_smoketest
 > pytest tests\test_create.py -k create_install_update_remove_smoketest
 ```
 
+Note: Some integration tests require you build a package with conda-build beforehand.
+This is taking care of if you run `docker compose run integration-tests`, but you need
+to do it manually in other modes:
+
+**Bash (macOS, Linux, Windows)**
+
+```bash
+$ conda install conda-build
+$ conda-build tests/test-recipes/activate_deactivate_package
+```
+
+Check `dev/linux/integration.sh` and `dev\windows\integration.bat` for more details.
+
+
 ## Conda Contributor License Agreement
 
 In case you're new to CLAs, this is rather standard procedure for larger
@@ -212,14 +281,18 @@ everywhere to actually perform the release. So it is customary to run
 
 The standard workflow is thus:
 
-   rever check
-   rever 1.2.3
+```bash
+$ rever check
+$ rever 1.2.3
+```
 
 If for some reason a release fails partway through, or you want to claw back a
 release that you have made, rever allows you to undo activities. If you find yourself
-in this pickle, you can pass the ``--undo`` option a comma-separated list of
+in this pickle, you can pass the `--undo` option a comma-separated list of
 activities you'd like to undo. For example:
 
-   rever --undo tag,changelog,authors 1.2.3
+```bash
+$ rever --undo tag,changelog,authors 1.2.3
+```
 
 Happy releasing!

ci/Dockerfile → Dockerfile

@@ -1,4 +1,4 @@
-FROM debian:buster-slim AS buildbase
+FROM --platform=linux/amd64 debian:buster-slim AS buildbase
 
 ARG MINICONDA_URL=https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
 
@@ -11,23 +11,23 @@ RUN wget --quiet $MINICONDA_URL -O ~/miniconda.sh && \
     rm ~/miniconda.sh && \
     /opt/conda/bin/conda clean --all --yes
 
-FROM debian:buster-slim
+FROM --platform=linux/amd64 debian:buster-slim
 
 ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
 ENV PATH /opt/conda/bin:$PATH
 
-COPY ci/scripts/linux/setup.sh /tmp
+COPY dev/linux/setup.sh /tmp
 RUN bash /tmp/setup.sh
 
 COPY --from=buildbase /opt/conda /opt/conda
 
-ARG PYTHON_VERSION=3.8
+ARG python_version=3.9
 
-COPY tests/requirements.txt /tmp
+COPY ./tests/requirements.txt /tmp
 
 # conda and test dependencies
 RUN /opt/conda/bin/conda install --update-all -y -c defaults \
-    python=$PYTHON_VERSION \
+    python=$python_version \
     --file /tmp/requirements.txt && \
     /opt/conda/bin/conda clean --all --yes
 

dev/docker.bashrc → dev/linux/bashrc.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 echo "Initializing conda in dev mode..."
 echo "Factory config is:"

ci/scripts/linux/setup.sh → dev/linux/setup.sh

@@ -4,9 +4,9 @@ set -o errtrace -o pipefail -o errexit
 
 apt-get update --fix-missing
 apt-get install -y --no-install-recommends \
-    tini wget build-essential bzip2  ca-certificates \
+    tini wget build-essential bzip2 ca-certificates \
     libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion \
-    sudo htop less nano man
+    sudo htop less nano man grep
 apt-get clean
 rm -rf /var/lib/apt/lists/*
 

docker-compose.yml

@@ -0,0 +1,25 @@
+version: "3.9"
+
+x-base: &base
+  build:
+    context: .
+    # cache_from:
+    #   - debian:buster-slim
+    args:
+      - python_version=${CONDA_DOCKER_PYTHON:-3.9}
+  volumes:
+    - .:/opt/conda-src
+  container_name: conda-linux-python-${CONDA_DOCKER_PYTHON:-3.9}
+
+services:
+  interactive:
+    <<: *base
+    command: ["bash", "--rcfile", "/opt/conda-src/dev/linux/bashrc.sh"]
+
+  integration-tests:
+    <<: *base
+    command: ["bash", "/opt/conda-src/dev/linux/integration.sh"]
+
+  unit-tests:
+    <<: *base
+    command: ["bash", "/opt/conda-src/dev/linux/unit.sh"]

docs/source/dev-guide/contributing.md

@@ -0,0 +1,2 @@
+```{include} ../../../CONTRIBUTING.md
+```

docs/source/dev-guide/index.rst

@@ -0,0 +1,9 @@
+===============
+Developer guide
+===============
+
+.. toctree::
+   :maxdepth: 2
+
+   contributing
+   ../architecture

docs/source/index.rst

@@ -56,9 +56,9 @@ that approach may not be as up to date.
    user-guide/index
    configuration
    api/index
-   release-notes
    commands
-   architecture
    glossary
+   dev-guide/index
+   release-notes
 
 .. |reg|    unicode:: U+000AE .. REGISTERED SIGN