InfluxDB IOx (short for Iron Oxide, pronounced InfluxDB "eye-ox") is the future core of InfluxDB, an open source time series database. The name is in homage to Rust, the language this project is written in. It is built using Apache Arrow and DataFusion among other things. InfluxDB IOx aims to be:
- The future core of InfluxDB; supporting industry standard SQL, InfluxQL, and Flux
- An in-memory columnar store using object storage for persistence
- A fast analytic database for structured and semi-structured events (like logs and tracing data)
- A system for defining replication (synchronous, asynchronous, push and pull) and partitioning rules for InfluxDB time series data and tabular analytics data
- A system supporting real-time subscriptions
- A processor that can transform and do arbitrary computation on time series and event data as it arrives
- An analytic database built for data science, supporting Apache Arrow Flight for fast data transfer
Persistence is through Parquet files in object storage. It is a design goal to support integration with other big data systems through object storage and Parquet specifically.
For more details on the motivation behind the project and some of our goals, read through the InfluxDB IOx announcement blog post. If you prefer a video that covers a little bit of InfluxDB history and high level goals for InfluxDB IOx you can watch Paul Dix's announcement talk from InfluxDays NA 2020. For more details on the motivation behind the selection of Apache Arrow, Flight and Parquet, read this.
As we commit to support platforms they will be added here. Our current goal is that the following platforms will be able to run InfluxDB IOx.
- Linux x86 (
x86_64-unknown-linux-gnu
) - Darwin x86 (
x86_64-apple-darwin
) - Darwin arm (
aarch64-apple-darwin
)
This list is very unlikely to be complete; we will add more platforms based on our ability to support them effectively.
This project is very early and in active development. It isn't yet ready for testing, which is why we're not producing builds or documentation yet.
If you would like contact the InfluxDB IOx developers, join the InfluxData Community Slack and look for the #influxdb_iox channel.
We're also hosting monthly tech talks and community office hours on the project on the 2nd Wednesday of the month at 8:30 AM Pacific Time.
- Install dependencies
- Clone the repository
- Configure the server
- Compiling and Running (You can also build a Docker image to run InfluxDB IOx.)
- Write and read data
- Use the CLI
- Use InfluxDB 2.0 API compatibility
- Run health checks
- Manually call the gRPC API
To compile and run InfluxDB IOx from source, you'll need the following:
The easiest way to install Rust is to use rustup
, a Rust version manager.
Follow the instructions for your operating system on the rustup
site.
rustup
will check the rust-toolchain
file and automatically install and use the correct Rust version for you.
Building InfluxDB IOx requires clang
(for the croaring
dependency).
Check for clang
by running clang --version
.
clang --version
Apple clang version 12.0.0 (clang-1200.0.32.27)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
If clang
is not already present, it can typically be installed with the system package manager.
If you are building InfluxDB IOx on Linux then you will need to ensure you have installed the lld
LLVM linker.
Check if you have already installed it by running lld -version
.
lld -version
lld is a generic driver.
Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead
If lld
is not already present, it can typically be installed with the system package manager.
If you are building InfluxDB IOx on Apple Silicon you may find that the build fails with an error containing:
failed to invoke protoc (hint: https://docs.rs/prost-build/#sourcing-protoc): Bad CPU type in executable (os error 86)
Prost bundles a protoc
binary, which it uses if it cannot find a system alternative.
Prior to version 0.9, the binary it chooses with the above error is an x86
one, which won't work if you do not have Rosetta installed on your system.
You can install Rosetta by running:
softwareupdate --install-rosetta
An alternative to installing Rosetta is to point Prost at an arm
build of protoc
.
First, install protoc
, e.g., via Homebrew:
brew update && brew install protobuf
Then set the following environment variables to point Prost at your system install:
PROTOC=/opt/homebrew/bin/protoc
PROTOC_INCLUDE=/opt/homebrew/include
IOx should then build correctly.
The catalog is stored in Postgres (unless you're running in ephemeral mode). Postgres can be installed via Homebrew:
brew install postgres
then follow the instructions for starting Postgres either at system startup or on-demand.
Clone this repository using git
.
If you use the git
command line, this looks like:
git clone [email protected]:influxdata/influxdb_iox.git
Then change into the directory containing the code:
cd influxdb_iox
The rest of these instructions assume you are in this directory.
InfluxDB IOx can be configured using either environment variables or a configuration file, making it suitable for deployment in containerized environments.
For a list of configuration options, run influxdb_iox --help
.
For configuration options for specific subcommands, run influxdb_iox <subcommand> --help
.
To use a configuration file, use a .env
file in the working directory.
See the provided example configuration file.
To use the example configuration file, run:
cp docs/env.example .env
InfluxDB IOx is built using Cargo, Rust's package manager and build tool.
To compile for development, run:
cargo build
This creates a binary at target/debug/influxdb_iox
.
Building the Docker image requires:
- Docker 18.09+
- BuildKit
To enable BuildKit by default, set { "features": { "buildkit": true } }
in the Docker engine configuration,
or run docker build
withDOCKER_BUILDKIT=1
To build the Docker image:
DOCKER_BUILDKIT=1 docker build .
To start InfluxDB IOx and store data in memory, after you've compiled for development, run:
./target/debug/influxdb_iox run all-in-one
By default this runs an "all-in-one" server with HTTP server on port 8080
, router gRPC server on port 8081
and querier gRPC server on port 8082
. When the server is stopped all data lost.
To start InfluxDB IOx and store the catalog in Postgres and data in the local filesystem to persist data across restarts, after you've compiled for development, run:
./target/debug/influxdb_iox run all-in-one --catalog-dsn postgres:///iox_shared --data-dir=~/iox_data
where --catalog-dsn
is a connection URL to the Postgres database you wish to use, and
--data-dir
is the directory you wish to use.
Note that when the server is stopped all data that has not yet been written to parquet files will be lost.
Rather than building and running the binary in target
, you can also compile and run with one
command:
cargo run -- run all-in-one
To compile for performance testing, build in release mode then use the binary in target/release
:
cargo build --release
./target/release/influxdb_iox run all-in-one
You can also compile and run in release mode with one step:
cargo run --release -- run all-in-one
You can run tests using:
cargo test --all
See [docs/testing.md] for more information
Data can be written to InfluxDB IOx by sending line protocol format to the /api/v2/write
endpoint or using the CLI.
For example, assuming you are running in local mode, this command will send data in the test_fixtures/lineproto/metrics.lp
file to the company_sensors
database.
./target/debug/influxdb_iox -vv write company_sensors test_fixtures/lineproto/metrics.lp --host http://localhost:8081
Note that --host http://localhost:8081
is required because the router and query services run on different gRPC ports and the CLI defaults to the querier's port, 8082
.
To query the data stored in the company_sensors
database:
./target/debug/influxdb_iox query company_sensors "SELECT * FROM cpu LIMIT 10"
InfluxDB IOx is packaged as a binary with commands to start the IOx server, as well as a CLI interface for interacting with and configuring such servers.
The CLI itself is documented via built-in help which you can access by running influxdb_iox --help
InfluxDB IOx allows seamless interoperability with InfluxDB 2.0.
Where InfluxDB 2.0 stores data in organizations and buckets,
InfluxDB IOx stores data in namespaces.
IOx maps organization
and bucket
pairs to namespaces with the two parts separated by an underscore (_
):
organization_bucket
.
Here's an example using curl
to send data into the company_sensors
namespace using the InfluxDB 2.0 /api/v2/write
API:
curl -v "http://127.0.0.1:8080/api/v2/write?org=company&bucket=sensors" --data-binary @test_fixtures/lineproto/metrics.lp
The HTTP API exposes a healthcheck endpoint at /health
$ curl http://127.0.0.1:8080/health
OK
The gRPC API implements the gRPC Health Checking Protocol.
This can be tested with grpc-health-probe
:
$ grpc_health_probe -addr 127.0.0.1:8082 -service influxdata.platform.storage.Storage
status: SERVING
To manually invoke one of the gRPC APIs, use a gRPC CLI client such as grpcurl.
Tonic (the gRPC server library we're using) currently doesn't have support for gRPC reflection, hence you must pass all .proto
files to your client.
You can find a convenient grpcurl
wrapper that does that in the scripts
directory:
$ ./scripts/grpcurl -plaintext 127.0.0.1:8082 list
grpc.health.v1.Health
influxdata.iox.management.v1.ManagementService
influxdata.platform.storage.IOxTesting
influxdata.platform.storage.Storage
$ ./scripts/grpcurl -plaintext 127.0.0.1:8082 influxdata.iox.management.v1.ManagementService.ListDatabases
{
"names": [
"foobar_weather"
]
}
We welcome community contributions from anyone!
Read our Contributing Guide for instructions on how to run tests and how to make your first contribution.
There are a variety of technical documents describing various parts of IOx in the docs directory.