Doris 部署说明

Doris 集群的部署说明。


官网地址

安装包下载地址

https://cloud.baidu.com/doc/DORIS/s/bkn5owgit

官方文档地址

http://doris.apache.org/master/zh-CN/installing/install-deploy.html#%E9%9B%86%E7%BE%A4%E9%83%A8%E7%BD%B2

路径相关

安装路径

/opt/doris

手动创建文件夹

1
2
mkdir -p fe/doris-meta
mkdir -p be/storage

FE 主节点启动

1
./fe/bin/start_fe.sh --daemon

登录 Doris (使用MySQL控制台登录)

1
mysql -h10.0.15.131 -P9030 -uroot

设置密码

1
SET PASSWORD FOR 'root' = PASSWORD('root');

使用密码登录 Doris

1
mysql -h10.0.15.131 -P9030 -uroot -proot

FE 子节点注册、启动

FE 子节点注册

1
2
mysql> ALTER SYSTEM ADD FOLLOWER "10.0.15.132:19010";
mysql> ALTER SYSTEM ADD FOLLOWER "10.0.15.135:19010";

FE 集群启动

注意ip、端口为主节点对应的ip、端口

1
./bin/start_fe.sh --helper 10.0.15.131:19010 --daemon

FE 状态查看

1
mysql> SHOW PROC '/frontends'\G

BE 节点注册、启动

1
2
3
ALTER SYSTEM ADD BACKEND "10.0.15.131:9050";
ALTER SYSTEM ADD BACKEND "10.0.15.132:9050";
ALTER SYSTEM ADD BACKEND "10.0.15.135:9050";

BE 启动

1
./be/bin/start_be.sh --daemon

BE 状态查看

1
SHOW PROC '/backends'\G

Broker 启动、注册

Broker 启动

1
sh bin/start_broker.sh --daemon

Broker 节点注册

1
ALTER SYSTEM ADD BROKER broker_name "10.0.15.131:8000","10.0.15.132:8000";

Tips

前端报错修复方法

1
2
3
4
enable_http_server_v2
是否启用的 V2 版本的 HTTP Server 实现。新的 HTTP Server 采用 SpringBoot 实现。
并且实现了前后端分离。 只有当开启后,才能使用 ui/ 目录下的新版 UI 界面。
默认为 false。

修改可打开文件数

1
ulimit -n 65535

FE 集群启动失败的处理

1
因为IP冲突、启动命令使用错误引起的 FE 字节点启动失败,需要在下次启动前重建fe/doris-meta路径。

其他命令

删除 FE 节点
1
ALTER SYSTEM DROP FOLLOWER "10.0.15.132:9010";
删除 Broker 节点
1
ALTER SYSTEM DROP BROKER broker_name "10.0.15.131:8000","10.0.15.132:8000";

配置文件

FE.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

#####################################################################
## The uppercase properties are read and exported by bin/start_fe.sh.
## To see all Frontend configurations,
## see fe/src/org/apache/doris/common/Config.java
#####################################################################

# the output dir of stderr and stdout
LOG_DIR = ${DORIS_HOME}/log

DATE = `date +%Y%m%d-%H%M%S`
JAVA_OPTS="-Xmx4096m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Xmx4096m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"

##
## the lowercase properties are read by main program.
##

# INFO, WARN, ERROR, FATAL
sys_log_level = INFO

# store metadata, must be created before start FE.
# Default value is ${DORIS_HOME}/doris-meta
# meta_dir = ${DORIS_HOME}/doris-meta

http_port = 18030
rpc_port = 9020
query_port = 9030
edit_log_port = 19010
mysql_service_nio_enabled = true

# Choose one if there are more than one ip except loopback address.
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24
# Default value is empty.
priority_networks = 10.10.110.236/24

# Advanced configurations
# log_roll_size_mb = 1024
# sys_log_dir = ${DORIS_HOME}/log
# sys_log_roll_num = 10
# sys_log_verbose_modules =
# audit_log_dir = ${DORIS_HOME}/log
# audit_log_modules = slow_query, query
# audit_log_roll_num = 10
# meta_delay_toleration_second = 10
# qe_max_connection = 1024
# max_conn_per_user = 100
# qe_query_timeout_second = 300
# qe_slow_log_ms = 5000

enable_http_server_v2 = true

BE.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

PPROF_TMPDIR="$DORIS_HOME/log/"

# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO

# ports for admin, web, heartbeat service
be_port = 9060
be_rpc_port = 9070
webserver_port = 18040
heartbeat_service_port = 9050
brpc_port = 18060

# Choose one if there are more than one ip except loopback address.
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24
# Default value is empty.
priority_networks = 10.10.110.236/24

# data root path, separate by ';'
# you can specify the storage medium of each root path, HDD or SSD
# you can add capacity limit at the end of each root path, seperate by ','
# eg:
# storage_root_path = /home/disk1/doris.HDD,50;/home/disk2/doris.SSD,1;/home/disk2/doris
# /home/disk1/doris.HDD, capacity limit is 50GB, HDD;
# /home/disk2/doris.SSD, capacity limit is 1GB, SSD;
# /home/disk2/doris, capacity limit is disk capacity, HDD(default)
#
# you also can specify the properties by setting '<property>:<value>', seperate by ','
# property 'medium' has a higher priority than the extension of path
#
# Default value is ${DORIS_HOME}/storage, you should create it by hand.
# storage_root_path = ${DORIS_HOME}/storage

# Advanced configurations
# sys_log_dir = ${DORIS_HOME}/log
# sys_log_roll_mode = SIZE-MB-1024
# sys_log_roll_num = 10
# sys_log_verbose_modules = *
# log_buffer_level = -1
# palo_cgroups

apache_hdfs_broker.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# the thrift rpc port
broker_ipc_port=8000

# client session will be deleted if not receive ping after this time
client_expire_seconds=300
1
2
3
需要在conf路径增加以下两个文件,需要将集群对应文件复制到该路径
core-site.xml
hdfs-site.xml