Deploying with Docker
Deploy with Docker
This guide is about how to create a running image of Apache Doris using a Dockerfile. It allows for the quick pulling of an Apache Doris image to create and run a cluster, either with container orchestration tools or for a quick test.
Prerequisites
Overview
Prepare the build machine before creating the Docker image. The platform architecture of this machine determines the platform architecture for which the Docker image will be applicable. For example, if you use an X86_64 machine, you should download the X86_64 Doris binary program, because the resulting image will only run on X86_64 platforms. The same applies to ARM64 platforms.
Hardware
Recommended configuration: 4 cores, 16GB memory.
Software
Docker version: 20.10 or later.
Build image
Preparation
Dockerfile script writing
- Choose a Docker Hub-certified OpenJDK official image as the base parent image. The recommended base parent image is: openjdk:8u342-jdk (JDK 1.8 version).
- Use embed scripts for FE startup, multi-FE registration, FE status checks, BE startup, BE registration in FE, and BE status checks.
- Avoid using the
--daemon
option to start the application within Docker, as it may cause issues during deployment with orchestration tools like Kubernetes (K8s).
Methods to build
About the Dockerfile script used for compiling the Docker image, there are two ways to load the Apache Doris binary package:
- Use
wget
orcurl
commands to download the package during compilation and then complete the Docker Build process. - Pre-download the binary package to the build directory and load it into the Docker Build process using the
ADD
orCOPY
command.
The former method produces a Docker image of smaller size, but if the build fails, the download operation may be repeated, leading to longer build times. The latter method is suitable for environments with unstable network conditions. Here, we will provide an example using the second method.
Build FE Image
- Environment directory for building the FE image:
The build environment directory is as follows:
└── docker-build // Root directory
└── fe // FE directory
├── dockerfile // Dockerfile script
└── resource // Resource directory
├── init_fe.sh // Startup and registration script
└── apache-doris-2.0.3-bin.tar.gz // Binary package
- Download binary package
Download the official binary package or the compiled binary package, and replace the apache-doris package in ./docker-build/fe/resource
with it.
- Write Dockerfile
# Choose a base image
FROM openjdk:8u342-jdk
# Set environment variables
ENV JAVA_HOME="/usr/local/openjdk-8/"
ENV PATH="/opt/apache-doris/fe/bin:$PATH"
# Download the software into the Docker image
ADD ./resource/apache-doris-2.0.3-bin.tar.gz /opt/
RUN apt-get update && \
apt-get install -y default-mysql-client && \
apt-get clean && \
mkdir /opt/apache-doris && \
cd /opt && \
mv apache-doris-2.0.3-bin/fe /opt/apache-doris/
ADD ./resource/init_fe.sh /opt/apache-doris/fe/bin
RUN chmod 755 /opt/apache-doris/fe/bin/init_fe.sh
ENTRYPOINT ["/opt/apache-doris/fe/bin/init_fe.sh"]
- Rename the file as
Dockerfile
and save it to the./docker-build/fe
directory. - For the FE execution script init_fe.sh, refer to init_fe.sh in the Doris source code library.
- Perform build
Note that ${tagName}
should be replaced with the tag you need, such as: apache-doris:2.0.3-fe
.
cd ./docker-build/fe
docker build . -t ${fe-tagName}
Build BE image
- Environment directory for building the BE image:
└── docker-build // Root directory
└── be // BE directory
├── dockerfile // Dockerfile script
└── resource // Resource directory
├── init_be.sh // Startup and registration script
└── apache-doris-2.0.3-bin.tar.gz // Binary package
- Write Dockerfile
# Choose a base image
FROM openjdk:8u342-jdk
# Set environment variables
ENV JAVA_HOME="/usr/local/openjdk-8/"
ENV PATH="/opt/apache-doris/be/bin:$PATH"
# 下Download the software into the Docker image
ADD ./resource/apache-doris-2.0.3-bin.tar.gz /opt/
RUN apt-get update && \
apt-get install -y default-mysql-client && \
apt-get clean && \
mkdir /opt/apache-doris && \
cd /opt && \
mv apache-doris-2.0.3-bin/be /opt/apache-doris/
ADD ./resource/init_be.sh /opt/apache-doris/be/bin
RUN chmod 755 /opt/apache-doris/be/bin/init_be.sh
ENTRYPOINT ["/opt/apache-doris/be/bin/init_be.sh"]
- Rename the file as
Dockerfile
and save it to the./docker-build/be
directory. - For the BE execution script init_be.sh, refer to init_be.sh.
- Perform build
Note that ${tagName}
should be replaced with the tag you need, such as: apache-doris:2.0.3-be
.
cd ./docker-build/be
docker build . -t ${be-tagName}
Push the image to DockerHub or a private repository
Log in to DockerHub
docker login
Upon successful login, a "Success" prompt will be displayed. After that, you can push the image.
docker push ${tagName}
Deploy Docker cluster
The following is a brief overview of how to quickly create a complete Doris testing cluster using the docker run
or docker-compose up
commands.
It is advisable to avoid containerized solutions for Doris deployment in production environments. Instead, when deploying Doris on Kubernetes (K8s), it is recommended to utilize the Doris Operator for deployment.
Prerequisite
Software
Software | Version |
---|---|
Docker | 20.0 and later |
docker-compose | 20.1 and later |
Hardware
Configuration | Hardware | Maximum Running Cluster Size |
---|---|---|
Minimum | 2C 4G | 1FE 1BE |
Recommended | 4C 16G | 3FE 3BE |
Execute the following command in the host machine:
sysctl -w vm.max_map_count=2000000
Docker Compose
The required image varies depending on the platform. The following takes the X86_64
platform as an example.
Network mode
Doris Docker supports two network modes:
- The HOST mode is suitable for deploying across multiple nodes, with one FE and one BE per node.
- The subnet bridge mode is suitable for deploying multiple Doris processes on a single node (recommended). If you want to deploy across multiple nodes, additional component deployments are required (not recommended).
For demonstration purposes, this section will only show scripts written for the subnet bridge mode.
Interface description
Since Apache Doris 2.0.3 Docker Image
, the interface list for each process image is as follows:
Process | Interface | Interface Definition | Interface Example |
---|---|---|---|
FE | BE | BROKER | FE_SERVERS |
FE | FE_ID | FE node ID | 1 |
BE | BE_ADDR | BE node informatioin | 172.20.80.5:9050 |
BE | NODE_ROLE | BE node type | computation |
Note that the above interfaces must be specified with relevant information; otherwise, the process will not start.
The FE_SERVERS interface follows the rule:
FE_NAME:FE_HOST:FE_EDIT_LOG_PORT[,FE_NAME:FE_HOST:FE_EDIT_LOG_PORT]
The FE_ID interface should be an integer from
1
to9
, where1
represents the Master node.The BE_ADDR interface follows the rule:
BE_HOST:BE_HEARTBEAT_SERVICE_PORT
The NODE_ROLE interface should be
computation
or empty. When it is empty or any other value, it indicates amix
node.The BROKER_ADDR interface follows the rule:
BROKER_HOST:BROKER_IPC_PORT
Script template
Docker Run command
1 FE & 1 BE command template
Note that you should replace ${INTERNAL_IP_OF_CURRENT_MACHINE}
with the internal IP of your current machine.
docker run -itd \
--name=fe \
--env FE_SERVERS="fe1:${INTERNAL_IP_OF_CURRENT_MACHINE}:9010" \
--env FE_ID=1 \
-p 8030:8030 \
-p 9030:9030 \
-v /data/fe/doris-meta:/opt/apache-doris/fe/doris-meta \
-v /data/fe/log:/opt/apache-doris/fe/log \
--net=host \
apache/doris:2.0.3-fe-x86_64
docker run -itd \
--name=be \
--env FE_SERVERS="fe1:${INTERNAL_IP_OF_CURRENT_MACHINE}:9010" \
--env BE_ADDR="${INTERNAL_IP_OF_CURRENT_MACHINE}:9050" \
-p 8040:8040 \
-v /data/be/storage:/opt/apache-doris/be/storage \
-v /data/be/log:/opt/apache-doris/be/log \
--net=host \
apache/doris:2.0.3-be-x86_64
Download the Docker Run command template for 3 FE & 3 BE from here if needed.
Docker Compose scripte
1 FE & 1 BE template
Note that you should replace ${INTERNAL_IP_OF_CURRENT_MACHINE}
with the internal IP of your current machine.
version: "3"
services:
fe:
image: apache/doris:2.0.3-fe-x86_64
hostname: fe
environment:
- FE_SERVERS=fe1:${INTERNAL_IP_OF_CURRENT_MACHINE}:9010
- FE_ID=1
volumes:
- /data/fe/doris-meta/:/opt/apache-doris/fe/doris-meta/
- /data/fe/log/:/opt/apache-doris/fe/log/
network_mode: host
be:
image: apache/doris:2.0.3-be-x86_64
hostname: be
environment:
- FE_SERVERS=fe1:${INTERNAL_IP_OF_CURRENT_MACHINE}:9010
- BE_ADDR=${INTERNAL_IP_OF_CURRENT_MACHINE}:9050
volumes:
- /data/be/storage/:/opt/apache-doris/be/storage/
- /data/be/script/:/docker-entrypoint-initdb.d/
depends_on:
- fe
network_mode: host
Download the Docker Compose command template for 3 FE & 3 BE from here if needed.
Deploy Doris Docker
Choose one of the following deployment methods:
- Execute the
docker run
command to create the cluster. - Save the
docker-compose.yaml
script and execute thedocker-compose up -d
command in the same directory to create the cluster.