| layout | title |
|---|---|
docu |
Building DuckDB from Source |
DuckDB binaries are available for stable and nightly builds on the installation page. You should only build DuckDB under specific circumstances, such as when running on a specific architecture or building an unmerged pull request.
DuckDB needs CMake and a C++11-compliant compiler (e.g., GCC, Apple-Clang, MSVC). Additionally, we recommend using the Ninja build system.
Install the required packages with the package manager of your distribution.
Fedora, CentOS, and Red Hat:
sudo yum install -y git g++ cmake ninja-buildUbuntu and Debian:
sudo apt-get update
sudo apt-get install -y git g++ cmake ninja-buildAlpine Linux:
apk add g++ git make cmake ninjaInstall Xcode and Homebrew. Then, install the required packages with:
brew install cmake ninjaConsult the Windows CI workflow for a list of packages used to build DuckDB on Windows.
The DuckDB Python package requires the Microsoft Visual C++ Redistributable package to be built and to run.
To build DuckDB we use a Makefile which in turn calls into CMake. We also advise using Ninja as the generator for CMake.
GEN=ninja makeIt is not advised to directly call CMake, as the Makefile sets certain variables that are crucial to properly building the package.
DuckDB can be built in many different settings, most of these correspond directly to CMake but not all of them.
This build has been stripped of all the assertions and debug symbols and code, optimized for performance.
This build runs with all the debug information, including symbols, assertions and DEBUG blocks.
The special debug defines are not automatically set for this build however.
This build does not trigger the #ifdef DEBUG code blocks, but still has debug symbols that make it possible to step through the execution with line number information and D_ASSERT lines are still checked in this build.
This build is similar to relassert in many ways, only assertions are also stripped in this build.
This build is a shorthand for release with BUILD_BENCHMARK=1 set.
This creates a build and then runs Clang-Tidy to check for issues or style violations through static analysis.
The CI will also run this check, causing it to fail if this check fails.
This doesn't actually create a build, but uses the following format checkers to check for style issues:
- clang-format to fix format issues in the code.
- cmake-format to fix format issues in the CMakeLists.txt files.
The CI will also run this check, causing it to fail if this check fails.
For every package that is maintained by core DuckDB, there exists a flag in the Makefile to enable building the package.
These can be enabled by either setting them in the current env, through set up files like bashrc or zshrc, or by setting them before the call to make, for example:
BUILD_PYTHON=1 make debugWhen this flag is set, the Python package is built.
When this flag is set, the CLI is built, this is usually enabled by default.
When this flag is set, our in-house Benchmark testing suite is built.
More information about this can be found here.
When this flag is set, the Java package is built.
When this flag is set, the ODBC package is built.
For every in-tree extension that is maintained by core DuckDB there exists a flag to enable building and statically linking the extension into the build.
When this flag is set, the autocomplete extension is built.
When this flag is set, the icu extension is built.
When this flag is set, the tpch extension is built, this enables TPCH-H data generation and query support using dbgen.
When this flag is set, the tpcds extension is built, this enables TPC-DS data generation and query support using dsdgen.
When this flag is set, the TPCE extension is built, unlike TPC-H and TPC-DS this does not enable data generation and query support, but does enable tests for TPC-E through our test suite.
When this flag is set, the fts (full text search) extension is built.
When this flag is set, the httpfs extension is built.
When this flag is set, the json extension is built.
When this flag is set, the inet extension is built.
When this flag is set, the SQLSmith extension is built.
D_ASSERT(condition) is used all throughout the code, these will throw an InternalException in debug builds.
With this flag enabled, when the assertion triggers it will instead directly cause a crash.
In our execution format string_t has the feature to "inline" strings that are under a certain length (12 bytes), this means they don't require a separate allocation.
When this flag is set, we disable this and don't inline small strings.
Our data structures that are used extensively throughout the non-performance-critical code have extra checks to ensure memory safety, these checks include:
- Making sure
nullptris never dereferenced. - Making sure index out of bounds accesses don't trigger a crash.
With this flag enabled we remove these checks, this is mostly done to check that the performance hit of these checks is negligible.
When previously pinned blocks in the BufferManager are unpinned, with this flag enabled we destroy them instantly to make sure that there aren't situations where this memory is still being used, despite not being pinned.
When a crash or assertion hit occurs in a test, print a stack trace.
This is useful when debugging a crash that is hard to pinpoint with a debugger attached.
To improve compilation time, we use Unity Build to combine translation units.
This can however hide include bugs, this flag disables using the unity build so these errors can be detected.
In some situations, running an executable that has been built with sanitizers enabled is not support / can cause problems. Julia is an example of this.
With this flag enabled, the sanitizers are disabled for the build.
By default, R compiles packages using a single thread.
To parallelize the compilation, create or edit the ~/.R/Makevars file, and add the following content:
MAKEFLAGS = -j8
Building the R package on Linux running on an ARM64 architecture (AArch64) may result in the following error message:
/usr/bin/ld: /usr/include/c++/10/bits/basic_string.tcc:206: warning: too many GOT entries for -fpic, please recompile with -fPIC
To work around this, create or edit the ~/.R/Makevars file:
ALL_CXXFLAGS = $(PKG_CXXFLAGS) -fPIC $(SHLIB_CXXFLAGS) $(CXXFLAGS)
Problem: The build fails on macOS when both the httpfs extension and the Python package are included:
GEN=ninja BUILD_PYTHON=1 BUILD_HTTPFS=1 makeld: library not found for -lcrypto
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang++' failed with exit code 1
ninja: build stopped: subcommand failed.
make: *** [release] Error 1
Solution:
In general, we recommended using the nightly builds, available under GitHub main (Bleeding Edge) on the installation page.
If you would like to build DuckDB from source, avoid using the BUILD_PYTHON=1 flag unless you are actively developing the Python library.
Instead, first build the httpfs extension (if required), then build and install the Python package separately using pip:
GEN=ninja BUILD_HTTPFS=1 makeIf the next line complains about pybind11 being missing, or --use-pep517 not being supported, make sure you're using a modern version of pip and setuptools.
python3-pip on your OS may not be modern, so you may need to run python3 -m pip install pip -U first.
python3 -m pip install tools/pythonpkg --use-pep517 --userProblem: When building the httpfs extension on Linux, the build may fail with the following error.
CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
OPENSSL_INCLUDE_DIR)
Solution: Install the libssl-dev library.
sudo apt-get install -y libssl-dev
GEN=ninja BUILD_HTTPFS=1 make