Local Docker environment for Hadoop

This repository is a fork from BDE

Changes

• Use alpine linux as base image (adoptopenjdk/openjdk8:alpine)
• Use Hadoop 3.3.1
• Enabled K8S commands (in development)
• Manually download hadoop binary from Mirror and save on alpine folder

Build images on local host

docker build -t hadoop-base:3.3.1-jdk8-alpine .\alpine\
docker build -t hadoop-namenode:3.3.1-jdk8-alpine .\namenode\
docker build -t hadoop-datanode:3.3.1-jdk8-alpine .\datanode\
docker build -t hadoop-resourcemanager:3.3.1-jdk8-alpine .\resourcemanager\
docker build -t hadoop-historyserver:3.3.1-jdk8-alpine .\historyserver\
docker build -t hadoop-nodemanager:3.3.1-jdk8-alpine .\nodemanager\

Start services

docker-compose up --build -d

docker-compose creates a docker network that can be found by running docker network list, e.g. dockerhadoop_default.

Run docker network inspect on the network (e.g. dockerhadoop_default) to find the IP the hadoop interfaces are published on. Access these interfaces with the following URLs:

• Namenode: http://127.0.0.1:9870/dfshealth.html#tab-overview
• History server: http://127.0.0.1:8188/applicationhistory
• Datanode: http://127.0.0.1:9864/
• Nodemanager: http://127.0.0.1:8042/node
• Resource manager: http://127.0.0.1:8088/

Configure Environment Variables

The configuration parameters can be specified in the hadoop.env file or as environmental variables for specific services (e.g. namenode, datanode etc.):

  CORE_CONF_fs_defaultFS=hdfs://namenode:8020

CORE_CONF corresponds to core-site.xml. fs_defaultFS=hdfs://namenode:8020 will be transformed into:

  <property><name>fs.defaultFS</name><value>hdfs://namenode:8020</value></property>

To define dash inside a configuration parameter, use triple underscore, such as YARNCONF_yarn_log**_aggregation**enable=true (yarn-site.xml):

  <property><name>yarn.log-aggregation-enable</name><value>true</value></property>

The available configurations are:

• /etc/hadoop/core-site.xml CORE_CONF
• /etc/hadoop/hdfs-site.xml HDFS_CONF
• /etc/hadoop/yarn-site.xml YARN_CONF
• /etc/hadoop/httpfs-site.xml HTTPFS_CONF
• /etc/hadoop/kms-site.xml KMS_CONF
• /etc/hadoop/mapred-site.xml MAPRED_CONF

If you need to extend some other configuration file, refer to base/entrypoint.sh bash script.