This repository houses the configuration files and scripts for building a TU Delft Conda-based standalone offline installer that includes annually requested software packages. This is suitable for installing on TU Delft's lab computers including those used for digital examinations.
Current:
- Jason K. Moore, ME, [email protected]
- Jan-Maarten Brockhoff, ICT, [email protected]
- Kevin Geboers, Workplace Support Education, [email protected]
Past:
- Bart Gerritsen, EEMCS, [email protected]
Managing a university-wide software distribution takes a large effort. At the minimum, we ask that you contribute by opening issues in this repository to report bugs and software/feature requests. At the maximum, help to collaboratively improve the distribution through merge requests and testing the build and installation. We will do our best to address issues and merge requests in a timely manner and will also provide commit rights to those that would like to help maintain this repository. Reach out if that interests you.
Version numbering schema: v<year>.<two digit integer>.
| version | changes |
|---|---|
| v2024.04 | final release for 2024-2025 academic year |
| v2024.03 | pre-final release for 2024-2025 academic year |
| v2024.02 | second testing release for 2024-2025 academic year |
| v2024.01 | first testing release for 2024-2025 academic year |
| v2023.01 | 2023-2024 update |
| v2022.01 | 2022-2023 update |
| v2021.01 | 2021-2022 update |
| v2020.01 | initial make |
- Make open source scientific software available in TU Delft computer labs
- Make open source scientific software available for use during TU Delft computer exams
- Software should be at the latest versions (latest within ~18 months) and mutually compatible
- Installation must primarily support Windows (exam computers), but should also install on Linux and Mac OSX
- Installation must be scriptable on Windows
- Software installation must occur on an air-gapped computer (no internet)
We create a Windows executable installer for a collection of Conda packages using Conda Constructor and script its installation on Windows. We install a set of software packages from Conda Forge and make them available via the Conda package manager.
The goal of agreeing on a joint installation of a conda-based distribution is multifold:
- to avoid a cluttering of (slightly) different Python environments across the infrastructure within the TU Delft
- to avoid intricate and disturbing differences between Python environments on the one hand and the DigitalExam python-environment during exams
- to mitigate the burden of maintaining the configurations involved
This installation is revised and updated in a yearly cycle. ICT-WPS ("Werkplekbeheer") receives an installation instruction, drafted on behalf of all faculties by the team that runs this repository, taking care of the actual implementation of a prepared package. This package may then be deployed through https://software.tudelft.nl and is at the basis of the implementation of the DigitalExam-environment.
The (annual) process flow is roughly as follows:
| Date | Action |
|---|---|
| Mar | review the issues brought up in the running academic year |
| Apr | set out a master plan with ICT Werkplekbeheer |
| May | send around communication to users community to express new needs and wishes |
| Jun | draft an installation specification and work plan and send to ICT |
| Jul 1 | package version number freeze |
| Jul | alpha testing new installation and repair apparent errors |
| Jul | tweak package version pins and create any necessary conda environments |
| Aug | do beta (field) testing in faculties |
| Sep 1 | new environment packaged, in DEE, and published on software.tudelft.nl |
| Sep+ | new issues opened on Gitlab |
If you need a conda package that is not available in Conda Forge installation, you can request inclusion of the package. The first step is to check if your package is available in Conda Forge. Search this list to check (note that package names may differ from PyPi, CRAN, etc.). If your package is not present in Conda Forge, then you should initiate a new package build following the instructions to add a recipe.
Once the package is available on Conda Forge for Windows 64 bit systems, open a
merge request on this repository adding your package name to the
construct.yaml file. Test building the installer locally on Windows and
report your success (or failures) in the pull request.
My software is not available on the exam computers. How do I make it available?
The first step is to ensure that your desired package can be installed from Conda Forge. Search https://conda-forge.org/feedstock-outputs/ to see if your package is already available and if not you will need to add a recipe following the instructions here: https://conda-forge.org/#contribute. Once the package is available on Conda Forge, open a merge request on this repository adding your package to the
construct.yamlfile. Make sure to submit a merge request here months before the freeze deadline for the new academic year.
The package I need is missing from Conda Forge and adding it is beyond my skills.
Open an issue here and we will coach and help you get the package built on Conda Forge.
Is it possible to have different conda environments other than the base environment?
We can add new environments using Conda Constructor and make them selectable by end users, but the current limitation is the 2GB maximum for our exe installer on Windows. It is difficult to keep the installer under 2GB with even one additional environment if it includes the Anaconda distribution. Switching to only Conda Forge packages may allow us to reduce the installer size and thus support a limited number of additional environments.
Why can't my package be installed with PyPi/Cargo/NPM/CRAN?
Software building and installing ranges from simple to extremely complex, even for a single package. This is further complicated by trying to have a mutually compatible set of hundreds or thousands of packages installed together. We manage a monolithic installation of packages that come from a variety of languages (Python, C++, C, R, Julia, CUDA, etc.). Firstly, many of the packages cannot be installed from a single languages' repository, e.g. PyPi only hosts Python packages. Secondly, it is quite difficult, if not impossible, to create a compatible set of packages if relying on PyPi (see https://pypackaging-native.github.io/ for detailed explanations). A tenable solution that does not require excessive amounts of time and complexity for this team is to rely on Conda Forge for solving these problems (with our help!). The binaries available on Conda Forge are guaranteed to provide a compatible set by the nature of its design. Even though it may seem simple to install your package on your computer, it may not be so here. Every special case we add, costs us more time. So we default to Conda Forge packages with rare exceptions that depend on available volunteer time. This provides a minimal headache way to deliver a working software environments to our students.
Why can't students pip install or conda install packages while on the
computers?
The computers are not connected to the internet so these commands will not work in general. It may be possible to host our own copies of Conda Forge or PyPi behind our firewall in the future, but this is a large undertaking. We also do not give students write access to the locations that either of these tools install packages to, so the default install will not work.
Why are there no version specifications in the construct.yaml file? How do we
know what version will be installed?
We install the latest Conda Forge version of all packages up to the July 1 version freeze date. Build the installer to see what versions are installed. We are working on a way to publish regular version lists.
This is a list of people who have requested packages in the past or expressed interest in doing so.
| Name | Faculty | Requests | |
|---|---|---|---|
| Artur Schweidtmann | [email protected] | ||
| Bart Gerritsen | [email protected] | ||
| Coen de Visser | [email protected] | ||
| Cornel Thill | [email protected] | ||
| Erik Ulijn (ME) | [email protected] | ||
| Ferdinand Grozema | [email protected] | ||
| Ferdinand Postema (LR) | [email protected] | ||
| Frank Mulder | [email protected] | ||
| Gary Steele | [email protected] | ||
| Heike Vallery | [email protected] | ||
| Helma Torkamaan | [email protected] | ||
| Iulia lefter | [email protected] | ||
| Jacco Hoekstra | [email protected] | ||
| Jason K. Moore | [email protected] | ME | sckits.odes |
| Jeroen Kalkman | [email protected] | ||
| Joost Ellerbroek | [email protected] | ||
| Jordan Boyle | [email protected] | IDE | pygame, pymunk |
| Ludolf Meester | [email protected] | ||
| Marcel Sluiter | [email protected] | ||
| Marcel van den Broek | [email protected] | ||
| Margreet Docter | [email protected] | TNW | nb2214-2023 |
| Mario Negrello | [email protected] | nbgrader, numba, notebook extensions, TOC2 | |
| Mark Bakker | [email protected] | ||
| Max Theisen | [email protected] | pytorch, pytorch_geometric, torchvision-cpu, rdkit | |
| Miriam Coenders | [email protected] | ||
| Omar Kammouh | [email protected] | mesa, ffmpeg | |
| Özge Okur | [email protected] | ||
| Peter Somhorst | [email protected] | ME | |
| Peter Wilders | [email protected] | ||
| Peter van Nieuwenhuizen | [email protected] | ||
| Petra Heijnen (TBM) | [email protected] | more_itertools, pyvisgraph, shapely | |
| Rebeca Gonzalez Cabaleiro | [email protected] | ||
| Regine Vroom | [email protected] | ||
| Remko Uijlenhoet | [email protected] | ||
| Rene van Paassen (LR) | [email protected] | control, slycot | |
| Ronald Ligteringen | [email protected] | ||
| Ruud van der Ent | [email protected] | ||
| Sander van Cranenburgh | [email protected] | biogeme, biogeme-optimization, cython-biogeme | |
| Tom Viering | [email protected] | ||
| Valeri Markine | [email protected] | ||
| Wouter van der Wal | [email protected] | ||
| Requested by Artur Schweidtmann | [email protected] |
You will need conda installed on Windows 10. You can use conda from Anaconda, Miniconda, Miniforge, or any other conda-based installation. For example, you can install the latest of the mentioned three installers from:
- Anaconda: https://www.anaconda.com/download/
- Miniconda: https://docs.anaconda.com/free/miniconda/miniconda-install/
- Miniforge: https://conda-forge.org/miniforge/
The latest Miniconda Windows installer is at:
https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe
Install Git, either:
- Follow general installation instructions for your Operating System. For example, Github provides this guidance: https://github.com/git-guides/install-git
- or, install Git with conda (open the Anaconda/Conda prompt in Windows or a Terminal in OSX or Linux) and type:
conda install -c conda-forge gitIf you want to build a specific version of the installer then download the release files from:
https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/releases
If you want to work with the development version, then use one of the following methods to obtain the files:
If you have SSH setup for the EWI Gitlab use:
git clone [email protected]:bhmgerritsen/anaconda-dee-config.git
cd anaconda-dee-configelse use:
git clone https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config.git
cd anaconda-dee-configIf you don't want to use Git you can download a zip file of the latest version of this repository from:
This makes sure conda is up-to-date and is using the recently added libmamba solver. The libmamba solver is faster, uses less memory, gives better error messages, and can solve harder sets of constraints.
conda update -n base condaYou will build the installer with constructor, which is the same tool used to build the Anaconda installer. Create an environment with only constructor and activate it.
conda create -n constructor constructor
conda activate constructorconstructor will read the contents of the construct.yaml file, solve for the
compatible set of packages and build a .exe installer that contains them.
On Windows, to build the installer, run:
set "CONDA_CHANNEL_PRIORITY=strict"
constructor .
or on Mac or Linux, run:
CONDA_CHANNEL_PRIORITY=strict constructor .
in the directory with the construct.yaml file. This can take 10-30 minutes
depending on your computer's speed and download speeds. After it finishes there
will be tudelft-anaconda-<version>-Windows-x86_64.exe in the directory. This
installer can be executed on any computer that does not have Anaconda
installed.
Once you have built the installer exe, you can use it to install the TU Delft
Anaconda on other computers.
Download tudelft-anaconda-<version>-Windows-x86_64.exe to the Windows 10
computer you want to install it on.
We build Windows installers in this Github CI action:
https://github.com/moorepants/anaconda-dee-config/actions
The files can be downloaded up to 90 days after the build. Click the workflow run (likely the latest one) and then download from the "Artifacts" section:
Double click on the tudelft-anaconda installer and follow the prompts. By
default this will install the TU Delft Anaconda in C:\Program Files\tudelft-anaconda.
The full list of packages and their versions can be created with:
conda list --verbose --show-channel-urls -n base > base-pkgs.lst
Various approaches exist to create a beta testing environment that matches the exam computers. The method used in recent years:
- assign a test machine and test using RemoteDesktop (Citrix)
- assign access rights to a beta tester
Request these from WPS.