Skip to content

Latest commit

 

History

History
188 lines (147 loc) · 5.38 KB

README.adoc

File metadata and controls

188 lines (147 loc) · 5.38 KB
Raw

Kudu Docker Developer Documentation

Note
 All of this work is experimental and subject to change or removal.

Getting Started

  • Install docker following the instructions here

Building images

Note
 These sample commands assume running from the project root directory.

Build all the images for the default OS:

$ ./docker/docker-build.py
Note
 You can pass --help to the build script to see usage details and all the available parameters.

Running an image

Run an image with a bash prompt and remove it on exit:

$ docker run -it --rm apache/kudu:latest /bin/bash

Running a Kudu cluster with docker-compose

Below is a brief example of using Kudu with docker compose. For more advanced guidance please see the docker-compose documentation here.

Start a Kudu cluster with the following command: (This cluster has 3 masters and 3 tablet servers)

docker-compose -f docker/docker-compose.yml up --scale kudu-tserver=3 -d

Find the web UI url for kudu-master-1:

docker ps -q --filter=name=.*kudu-master-1.* | xargs -L1 docker port \
  | grep "^8051" | cut -d":" -f2 | sed 's/.*/http\:\/\/localhost\:&/'

You can adjust the number of tablet servers up or down by adjusting the scale:

docker-compose -f docker/docker-compose.yml up --scale kudu-tserver=<scale> -d

Shutdown the cluster:

docker-compose -f docker/docker-compose.yml down

Copying container files to the host

It could be useful to copy files from a pre-built container to your host. For example, pre-built thirdparty or kudu binaries.

$ SOURCE=`docker create kudu:thirdparty-xenial-latest`
$ docker cp $SOURCE:/kudu/thirdparty /local/kudu/thirdparty

Images

apache/kudu:[OS]-[VERSION]

A runtime image with the Kudu binaries and clients pre-installed and an entrypoint script that enables easily starting Kudu masters and tservers along with executing other commands. Copies the built artifacts and files from the kudu:build image. Uses the kudu:runtime image as a base.

apache/kudu:build-[OS]-[VERSION]

An image that has the Kudu source code pre-built. Uses the kudu:thirdparty image as a base.

apache/kudu:thirdparty-[OS]-[VERSION]

An image that has Kudu’s thirdparty dependencies built. Uses the kudu:dev image as a base.

apache/kudu:dev-[OS]-[VERSION]

A base image that has all the development libraries for Kudu pre-installed.

apache/kudu:runtime-[OS]-[VERSION]

A base image that has all the runtime libraries for Kudu pre-installed.

apache/kudu:kudu-python-[VERSION]

Builds a runtime image with the Kudu python client pre-installed.

Tips and Troubleshooting

Ensure enough resources are allocated to docker

If your docker build fails with an unusual or unclear error a good first step is to ensure docker has enough resources. A good place to start is 4 CPUs, 4 GiB of memory, and 32 GiB of disk.

Free up disk space

If you don’t clean up your environment periodically, you will likely run out of disk space. Use the following commands clean up after builds.

Clean all stopped containers, all dangling images, all dangling build cache and all unused networks:

$ docker system prune

Remove all stopped containers:

$ docker container prune

Remove all dangling images:

$ docker image prune

Remove all Kudu images:

$ docker rmi -f $(docker images -q apache/kudu)

Remove specific images:

$ docker rmi [IMAGE...]

Remove images by tag pattern:

$ TAG_PATTERN="apache/kudu:*"
$ docker rmi $(docker images -q "$TAG_PATTERN" --format "{{.Repository}}:{{.Tag}}")

View and remove cache mounts (ccache and Gradle cache):

$ docker system df -v | grep exec.cachemount
$ docker builder prune --filter type=exec.cachemount

Using the cache from pre-built images

You can tell docker to considered remote or local images in your build as cache sources. This can be especially useful when the base or thirdparty images have not changed as those builds can take a very long time.

To do this use the --cache-from flag when executing docker build commands.

Use multiple workspaces

The docker builds can take a long time to run. During that time you may want to continue other work in the git repo. Use git worktree to specify a worktree in another directory and run the docker build from there. Read more about git worktree here.