forked from apache/flink
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[FLINK-4118] Update docker image to 1.0.3 and remove unneeded deps
Some of the changes include: - Remove unneeded dependencies (nano, wget) - Remove apt lists to reduce image size - Reduce number of layers on the docker image (best docker practice) - Remove useless variables and base the code in generic ones e.g. FLINK_HOME - Change the default JDK from oracle to openjdk-8-jre-headless, based on two reasons: 1. You cannot legally repackage the oracle jdk in docker images 2. The open-jdk headless is more appropriate for a server image (no GUI stuff) - Return port assignation to the standard FLINK one: Variable: docker-flink -> flink taskmanager.rpc.port: 6121 -> 6122 taskmanager.data.port: 6122 -> 6121 jobmanager.web.port: 8080 -> 8081 This closes apache#2176
- Loading branch information
Showing
13 changed files
with
91 additions
and
304 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,80 +1,81 @@ | ||
#Apache Flink cluster deployment on Docker using Docker-Compose | ||
Apache Flink cluster deployment on docker using docker-compose | ||
|
||
##Installation | ||
###Install Docker | ||
# Installation | ||
|
||
Install the most recent stable version of docker | ||
https://docs.docker.com/installation/ | ||
|
||
if you have issues with Docker-Compose versions incompatible with your version of Docker try | ||
Install the most recent stable version of docker-compose | ||
https://docs.docker.com/compose/install/ | ||
|
||
`curl -sSL https://get.docker.com/ubuntu/ | sudo sh` | ||
# Build | ||
|
||
###Install Docker-Compose | ||
Images are based on the official Java Alpine (OpenJDK 8) image and run | ||
supervisord to stay alive when running containers. If you want to build the | ||
flink image run: | ||
|
||
``` | ||
curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose | ||
sh build.sh | ||
|
||
chmod +x /usr/local/bin/docker-compose | ||
``` | ||
|
||
###Get the repo | ||
|
||
###Build the images | ||
or | ||
|
||
Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers. | ||
docker build -t flink . | ||
|
||
The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config. | ||
If you want to build the container for a specific version of flink/hadoop/scala | ||
you can configure it in the respective args: | ||
|
||
- Run `./build.sh` | ||
docker build --build-arg FLINK_VERSION=1.0.3 --build-arg HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t "flink:1.0.3-hadoop2.6-scala_2.10" flink | ||
|
||
###Deploy | ||
# Deploy | ||
|
||
- Deploy cluster and see config/setup log output (best run in a screen session) | ||
|
||
`docker-compose up` | ||
docker-compose up | ||
|
||
- Deploy as a daemon (and return) | ||
|
||
`docker-compose up -d` | ||
docker-compose up -d | ||
|
||
- Scale the cluster up or down to *N* TaskManagers | ||
|
||
`docker-compose scale taskmanager=<N>` | ||
docker-compose scale taskmanager=<N> | ||
|
||
- Access the JobManager node with SSH (exposed on Port 220) | ||
- Access the Job Manager container | ||
|
||
`ssh root@localhost -p 220` | ||
docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh | ||
|
||
or on Mac OS X with boot2docker | ||
- Kill the cluster | ||
|
||
`ssh root@$(boot2docker ip) -p 220` | ||
docker-compose kill | ||
|
||
The password is 'secret' | ||
- Upload jar to the cluster | ||
|
||
- Kill the cluster | ||
docker cp <your_jar> $(docker ps --filter name=flink_jobmanager --format={{.ID}}):/<your_path> | ||
|
||
`docker-compose kill` | ||
- Copy file to all the nodes in the cluster | ||
|
||
- Upload a jar to the cluster | ||
|
||
`scp -P 220 <your_jar> root@localhost:/<your_path>` | ||
for i in $(docker ps --filter name=flink --format={{.ID}}); do | ||
docker cp <your_file> $i:/<your_path> | ||
done | ||
|
||
- Run a topology | ||
|
||
`ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c <your_class> <your_jar> <your_params>` | ||
From the jobmanager: | ||
|
||
docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params> | ||
|
||
If you have a local flink installation: | ||
|
||
$FLINK_HOME/bin/flink run -m <jobmanager:port> <your_jar> | ||
|
||
or | ||
|
||
ssh to the job manager and run the topology from there. | ||
$FLINK_HOME/bin/flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params> | ||
|
||
###Ports | ||
### Ports | ||
|
||
- The Web Dashboard is on port `48080` | ||
- The Web Client is on port `48081` | ||
- JobManager RPC port `6123` (default, not exposed to host) | ||
- TaskManagers RPC port `6121` (default, not exposed to host) | ||
- TaskManagers Data port `6122` (default, not exposed to host) | ||
- JobManager SSH `220` | ||
- TaskManagers SSH: randomly assigned port, check wih `docker ps` | ||
- TaskManagers RPC port `6122` (default, not exposed to host) | ||
- TaskManagers Data port `6121` (default, not exposed to host) | ||
|
||
Edit the `docker-compose.yml` file to edit port settings. |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.