Open-CE Builder

This repository contains the tools needed to build the Open-CE project.

The open-ce tool allows a user to build collections of conda recipes described within a collection of feedstocks. It also provides tools for validating potential conda environments and running tests for feedstocks.

GETTING STARTED

Requirements

• conda
  • The conda tool can either be installed through Anaconda or Miniconda.
• conda-build >= 3.22
  • Once conda is installed, conda-build can be installed with the command: conda install conda-build
• networkx >= 2.5
• python >= 3.8
• junit-xml >= 1.9
• docker >= 1.13 or podman >= 2.0.5
  • docker or podman required only when building within a container (see below).
• matplotlib == 3.7.1
  • Required only when exporting the dependency graph.

CUDA Requirements

Currently CUDA 11.8 and 12.2 are supported by the recipes in Open-CE. Please see doc/README.cuda_support.md for details on setting up a proper build enviornment for CUDA support.

Open-CE expects the CUDA_HOME environment variable to be set to the location of the CUDA installation. Note that not all recipes work when CUDA_HOME references a non-standard CUDA installation location. Reference the cuda README for more information.

When building packages that use CUDA, a tar package of TensorRT for the intended CUDA version will need to be downloaded ahead of time. The downloaded file should be placed in a new local directory called local_files. The cuda README has more information.

Installing the Open-CE Build Tools

To get the Open-CE build tools, one can either install them via conda from the Open-CE channel, install them via pip from github or clone the source code from github as below -

# Conda install from the open-ce channel
conda install -c open-ce open-ce-builder

OR

# Pip install from the main branch
pip install git+https://github.com/open-ce/open-ce-builder.git@main

OR

# Clone Open-CE from GitHub
git clone https://github.com/open-ce/open-ce-builder.git
cd open-ce-builder
pip install -e .

Open-CE compatibility with Open-CE Builder

Open-CE version	Open-CE Builder version
All releases upto 1.5.2	<=9.0.0
1.5.7 P10	9.0.1
1.5.7	10.0.3
1.6.1	10.0.3
1.7.10	12.0.2
1.8.1	12.0.2
1.9.4	12.0.3
1.10.0	13.0.0

Building a Collection of Packages

To build an entire integrated and functional conda channel using Open-CE, start by installing the needed tools in the Requirements section above. The open-ce build env command can then be used to build a collection of Open-CE packages. An Open-CE environment file needs to be passed in as input. A selection of environment files are provided within the open-ce repo for different frameworks such as TensorFlow and PyTorch. The output from running open-ce build env will be a local conda channel (by default called condabuild) and one or more conda environment file(s) in the output folder depending on the selected build configuration. For more details on open-ce build env, please see doc/README.open_ce_build.md.

The following commands will use the opence-env.yaml Open-CE environment file to build all of the Open-CE packages for Python 3.10 (the default), including CUDA builds and cpu-only builds (also the default). The commands should be run from within the same directory that contains local_files.

# Clone Open-CE from GitHub
git clone https://github.com/open-ce/open-ce.git
# Build packages
open-ce build env ./open-ce/envs/opence-env.yaml

The open-ce tool will also automatically look for environment files within the open-ce repo's env directory if an environment file isn't found locally.

The following commands will build the opence-env.yaml environment file:

# Build packages
open-ce build env opence-env

A specific version of an environment file from the open-ce repo can be built using the --git_tag flag.

The following commands will build version 1.1.4 of the open-ce environment file provided within the open-ce repo:

# Build packages
open-ce build env --git_tag open-ce-v1.1.4 opence-env

The following commands will use the opence-env.yaml Open-CE environment file from a specific Open-CE release to build all of the Open-CE packages for Python 3.9, 3.10 and 3.11, including only CUDA builds. The commands should be run from within the same directory that contains local_files.

# Build packages
open-ce build env --python_versions 3.9,3.10,3.11 --build_types cuda opence-env

Note that having conda-forge in your channel list may sometime cause conflicts or unexpected errors due to dependencies' versions mismatch. So, it is recommended to avoid mixing the channels during the build as well as at runtime.

Power10 MMA Optimization

Building Packages

One can build the major Open-CE libraries like TensorFlow, Pytorch, Xgboost, etc. with Power10 MMA optimization. For details, please see doc/README.open_ce_build.md.

Running Packages

These packages will work on Power9 or Power10, but not on Power8.

FIPS Compliant Packages

Building FIPS Compliant Packages

One can build Open-CE packages that are FIPS compliant by providing --fips option to open-ce build env command. When this option is provided, Open-CE Builder builds envs/openssl-env.yaml internally and then builds the provided Open-CE environment file for which the build is requested. FIPS compliant openssl and cryptography conda packages are built by openssl-env.yaml and then other Open-CE packages are built using these two packages.

Installing FIPS Compliant Packages

The system where FIPS compliant packages are to be installed must have Redhat's openssl and openssl-libs packages installed. These can be installed using yum install openssl openssl-libs openssl-devel.

Once FIPS complaint Open-CE packages are installed in a conda environment, ensure that the conda environment contains openssl and cryptography also from the same conda channel.

FFMPEG with reduced codecs and openssl

One can build Open-CE packages with ffmpeg in two ways:

• Provide --build-ffmpeg option to the open-ce build env command. When this option is provided, Open-CE Builder builds envs/ffmpeg-env.yaml internally and then builds the provided Open-CE environment file for which the build is requested.

• Provide --fips option to the open-ce build env command. This will use the openssl-env.yaml, which includes ffmpeg by default.

Building within a container

Passing the --container_build argument to the open-ce build env command will create a container image and perform the actual build inside of a container based on that image. This will provide a "clean" environment for the builds and make builds more system independent. It is recommended to build with this option as opposed to running on a bare metal machine. For more information on the --container_build option, please see doc/README.open_ce_build.md.

Building a Single Feedstock

The open-ce build feedstock command can be used to build a single feedstock (which could produce one or more conda packages). The output from running open-ce build feedstock will be a local conda channel (by default called condabuild). For more details on open-ce build feedstock, please see doc/README.open_ce_build.md.

The following commands will build all of the packages within a feedstock named MY_FEEDSTOCK.

# Clone Open-CE Environments from GitHub
git clone https://github.com/open-ce/open-ce.git
# Clone MY_FEEDSTOCK from GitHub
git clone https://github.com/open-ce/MY_FEEDSTOCK-feedstock.git
# Build packages
cd MY_FEEDSTOCK-feedstock
open-ce build feedstock --conda_build_config ../open-ce/envs/conda_build_config.yaml

Installing Packages

After performing a build, a local conda channel will be created. By default, this will be within a folder called condabuild (it can be changed using the --output_folder argument). After the build, packages can be installed within a conda environment from this local channel. If the packages are built using open-ce build env script, then a conda environment file will also be generated which can be used to generate a conda environment with the built packages installed in it. See conda's documentation for more information on conda environments.

The following command will install a package named PACKAGE from the local conda channel condabuild into the currently active conda environment.

conda install -c ./condabuild PACKAGE

The following command can be used to create a conda environment using a conda environment file.

conda env create -f <conda_environment_file>

Testing Packages

After performing the build using the open-ce build env tool, the open-ce test tool can be used to either test a package or a collection of packages. For more details on open-ce test, please see doc/README.open_ce_test.md.

Creating Container Image with Open-CE Packages installed

After performing the build using open-ce build env, the open-ce build image command can be used to create a runtime container image containing the newly created conda channel, as well as a conda environment with the newly build Open-CE packages. For more details on open-ce build image, please see doc/README.open_ce_build.md.

Contributions

For contribution information, please see the CONTRIBUTING.md page.