微众平台部署

以下是微众平台的部署流程。

基础环境

环境变量配置
环境变量 ~/.bashrc
变量名 变量路径
JAVA_HOME /usr/java/jdk1.8.0_281
HADOOP_CONF_DIR /opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop
HIVE_CONF_DIR /opt/cloudera/parcels/CDH/lib/hive/conf
SPARK_HOME /opt/cloudera/parcels/CDH/lib/spark
SPARK_CONF_DIR /opt/cloudera/parcels/CDH/lib/spark/conf
环境组件版本
基础组件版本
组件 版本
Python 2.7.5
JDK 1.8.0_281
MySQL 5.7.22
Nginx 1.16.1
Hadoop 3.0.0-cdh6.3.2
Hive 2.1.1-cdh6.3.2
Spark 2.4.0-cdh6.3.2
Scala 2.11.12
以下组件需要确认本地是否能够正常通讯
环境组件连通性确认
组件 确认命令
MySQL mysql -h10.0.. -P3306 -uusername -ppassword
Hadoop hadoop fs -ls /
Hive hive
Spark spark-shell
微众组件使用版本
微众组件使用版本
组件 版本
wedatasphere-dss 0.9.0
wedatasphere-dss-web 0.9.0
wedatasphere-linkis 0.11.0
wedatasphere-qualitis 0.8.0
linkis-jobtype 0.9.0
azkaban-exec-server 3.51.0
azkaban-web-server 3.51.0
Visualis 0.3.0-beta.8

组件编译

Linkis编译
pom文件修改

properties修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<properties>
<!-- 修改 hadoop、hive、hbase、spark 版本为CDH集群版本 -->
<hadoop.version>3.0.0-cdh6.3.2</hadoop.version>
<hive.version>2.1.1-cdh6.3.2</hive.version>
<hbase.version>2.1.0-cdh6.3.2</hbase.version>
<spark.version>2.4.0-cdh6.3.2</spark.version>
<presto.version>0.234</presto.version>
<spring.eureka.version>1.4.4.RELEASE</spring.eureka.version>
<spring.boot.version>2.0.3.RELEASE</spring.boot.version>
<gson.version>2.8.5</gson.version>
<fasterxml.jackson.version>2.10.0</fasterxml.jackson.version>
<scala.version>2.11.8</scala.version>
<jdk.compile.version>1.8</jdk.compile.version>
<plugin.scala.version>2.15.2</plugin.scala.version>
<scala.binary.version>2.11</scala.binary.version>
<jersey.version>2.16</jersey.version>
<jersey.servlet.version>2.23.1</jersey.servlet.version>
<jetty.version>9.4.11.v20180605</jetty.version>
<httpclient.version>4.5.4</httpclient.version>
<httpmime.version>4.5.4</httpmime.version>
<slf4j.version>1.7.12</slf4j.version>
<maven.version>3.3.3</maven.version>
<xstream.core.version>1.4.11.1</xstream.core.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<dispatch.version>0.11.2</dispatch.version>
<jsoup.version>1.8.2</jsoup.version>
<harbor.reposity>wedatasphere/linkis</harbor.reposity>
</properties>

profiles修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
<profiles>
<profile>
<id>product</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<project.release.version>0.11.0</project.release.version>
<netty.version>4.1.17.Final</netty.version>
<json4s.version>3.5.3</json4s.version>
</properties>
</profile>
<!-- 修改 spark版本为spark2.4.0.cloudera1 -->
<profile>
<id>spark2.4.0.cloudera1</id>
<properties>
<project.release.version>0.11.0</project.release.version>
<netty.version>4.1.17.Final</netty.version>
<json4s.version>3.2.11</json4s.version>
</properties>
</profile>
<profile>
<id>spark2.2-2.0</id>
<properties>
<project.release.version>0.11.0</project.release.version>
<netty.version>4.0.47.Final</netty.version>
<json4s.version>3.2.11</json4s.version>
</properties>
</profile>

<profile>
<id>release</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<attach>true</attach>
</configuration>
<executions>
<execution>
<id>create-source-jar</id>
<goals>
<goal>jar-no-fork</goal>
<goal>test-jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>3.0.0-M1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>

</plugins>
</build>
</profile>
</profiles>
编译时Maven setting.xml 配置信息

需要注意cloudera相关的镜像配置,避免出现找不到cloudera源的情况

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
<?xml version="1.0" encoding="UTF-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

<!--
| This is the configuration file for Maven. It can be specified at two levels:
|
| 1. User Level. This settings.xml file provides configuration for a single user,
| and is normally provided in ${user.home}/.m2/settings.xml.
|
| NOTE: This location can be overridden with the CLI option:
|
| -s /path/to/user/settings.xml
|
| 2. Global Level. This settings.xml file provides configuration for all Maven
| users on a machine (assuming they're all using the same Maven
| installation). It's normally provided in
| ${maven.conf}/settings.xml.
|
| NOTE: This location can be overridden with the CLI option:
|
| -gs /path/to/global/settings.xml
|
| The sections in this sample file are intended to give you a running start at
| getting the most out of your Maven installation. Where appropriate, the default
| values (values used when the setting is not specified) are provided.
|
|-->
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<!-- localRepository
| The path to the local repository maven will use to store artifacts.
|
| Default: ${user.home}/.m2/repository
<localRepository>/path/to/local/repo</localRepository>
-->

<!-- interactiveMode
| This will determine whether maven prompts you when it needs input. If set to false,
| maven will use a sensible default value, perhaps based on some other setting, for
| the parameter in question.
|
| Default: true
<interactiveMode>true</interactiveMode>
-->

<!-- offline
| Determines whether maven should attempt to connect to the network when executing a build.
| This will have an effect on artifact downloads, artifact deployment, and others.
|
| Default: false
<offline>false</offline>
-->

<!-- pluginGroups
| This is a list of additional group identifiers that will be searched when resolving plugins by their prefix, i.e.
| when invoking a command line like "mvn prefix:goal". Maven will automatically add the group identifiers
| "org.apache.maven.plugins" and "org.codehaus.mojo" if these are not already contained in the list.
|-->
<pluginGroups>
<!-- pluginGroup
| Specifies a further group identifier to use for plugin lookup.
<pluginGroup>com.your.plugins</pluginGroup>
-->
</pluginGroups>

<!-- proxies
| This is a list of proxies which can be used on this machine to connect to the network.
| Unless otherwise specified (by system property or command-line switch), the first proxy
| specification in this list marked as active will be used.
|-->
<proxies>
<!-- proxy
| Specification for one proxy, to be used in connecting to the network.
|
<proxy>
<id>optional</id>
<active>true</active>
<protocol>http</protocol>
<username>proxyuser</username>
<password>proxypass</password>
<host>proxy.host.net</host>
<port>80</port>
<nonProxyHosts>local.net|some.host.com</nonProxyHosts>
</proxy>
-->
</proxies>

<!-- servers
| This is a list of authentication profiles, keyed by the server-id used within the system.
| Authentication profiles can be used whenever maven must make a connection to a remote server.
|-->
<servers>
<!-- server
| Specifies the authentication information to use when connecting to a particular server, identified by
| a unique name within the system (referred to by the 'id' attribute below).
|
| NOTE: You should either specify username/password OR privateKey/passphrase, since these pairings are
| used together.
|
<server>
<id>deploymentRepo</id>
<username>repouser</username>
<password>repopwd</password>
</server>
-->

<!-- Another sample, using keys to authenticate.
<server>
<id>siteServer</id>
<privateKey>/path/to/private/key</privateKey>
<passphrase>optional; leave empty if not used.</passphrase>
</server>
-->
</servers>

<!-- mirrors
| This is a list of mirrors to be used in downloading artifacts from remote repositories.
|
| It works like this: a POM may declare a repository to use in resolving certain artifacts.
| However, this repository may have problems with heavy traffic at times, so people have mirrored
| it to several places.
|
| That repository definition will have a unique id, so we can create a mirror reference for that
| repository, to be used as an alternate download site. The mirror site will be the preferred
| server for that repository.
|-->
<mirrors>
<!-- mirror
| Specifies a repository mirror site to use instead of a given repository. The repository that
| this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
| for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
|
<mirror>
<id>mirrorId</id>
<mirrorOf>repositoryId</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://my.repository.com/repo/path</url>
</mirror>
-->
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<mirrorOf>*,!cloudera</mirrorOf>
<url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
</mirror>

<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<mirrorOf>*,!cloudera</mirrorOf>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
</mirror>

<mirror>
<id>jFog</id>
<name>bintray</name>
<url>http://jcenter.bintray.com/</url>
<mirrorOf>*,!cloudera</mirrorOf>
</mirror>

<mirror>
<id>repository.jboss.org-public</id>
<name>JBoss repository</name>
<url>https://repository.jboss.org/nexus/content/groups/public</url>
<mirrorOf>*,!cloudera</mirrorOf>
</mirror>
</mirrors>

<!-- profiles
| This is a list of profiles which can be activated in a variety of ways, and which can modify
| the build process. Profiles provided in the settings.xml are intended to provide local machine-
| specific paths and repository locations which allow the build to work in the local environment.
|
| For example, if you have an integration testing plugin - like cactus - that needs to know where
| your Tomcat instance is installed, you can provide a variable here such that the variable is
| dereferenced during the build process to configure the cactus plugin.
|
| As noted above, profiles can be activated in a variety of ways. One way - the activeProfiles
| section of this document (settings.xml) - will be discussed later. Another way essentially
| relies on the detection of a system property, either matching a particular value for the property,
| or merely testing its existence. Profiles can also be activated by JDK version prefix, where a
| value of '1.4' might activate a profile when the build is executed on a JDK version of '1.4.2_07'.
| Finally, the list of active profiles can be specified directly from the command line.
|
| NOTE: For profiles defined in the settings.xml, you are restricted to specifying only artifact
| repositories, plugin repositories, and free-form properties to be used as configuration
| variables for plugins in the POM.
|
|-->
<profiles>
<profile>
<id>boundlessgeo</id>
<repositories>
<repository>
<id>boundlessgeo</id>
<url>https://repo.boundlessgeo.com/main/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</snapshots>
</repository>
</repositories>
</profile>
<profile>
<id>aliyun</id>
<repositories>
<repository>
<id>aliyun</id>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</snapshots>
</repository>
</repositories>
</profile>
<profile>
<id>maven-central</id>
<repositories>
<repository>
<id>maven-central</id>
<url>http://central.maven.org/maven2/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</snapshots>
</repository>
</repositories>
</profile>
<profile>
<id>cloudera</id>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</snapshots>
</repository>
</repositories>
</profile>
</profiles>

<!-- activeProfiles
| List of profiles that are active for all builds.
|
<activeProfiles>
<activeProfile>alwaysActiveProfile</activeProfile>
<activeProfile>anotherAlwaysActiveProfile</activeProfile>
</activeProfiles>
-->
<activeProfiles>
<activeProfile>boundlessgeo</activeProfile>
<activeProfile>aliyun</activeProfile>
<activeProfile>maven-central</activeProfile>
<activeProfile>cloudera</activeProfile>
</activeProfiles>
</settings>
编译命令
1
2
3
4
5
# 在Linkis pom.xml同级路径执行一下命令
mvn -N install
mvn clean install
# 获取安装包,在工程的assembly -> target目录下:
wedatasphere-linkis-0.11.0-dist.tar.gz

Qualitis编译
编译命令
1
2
# 在Qualitis build.gradle同级路径执行一下命令
gradle clean distZip
DSS编译
pom文件修改

properties修改

1
2
3
4
5
6
7
<properties>
<dss.version>0.9.0</dss.version>
<linkis.version>0.11.0</linkis.version>
<scala.version>2.11.8</scala.version>
<jdk.compile.version>1.8</jdk.compile.version>
<maven.version>3.3.3</maven.version>
</properties>
编译命令
1
2
3
# 在DSS pom.xml同级路径执行一下命令
mvn -N install
mvn clean install

组件部署

Nginx部署
1
2
3
4
5
6
7
8
9
10
11
12
# First update the system software packages to the latest version.
yum -y update
# Next, install Nginx
yum install epel-release
yum install nginx
# Once Nginx web server installed, you can start it first time and enable it to start automatically at system boot.
systemctl start nginx
systemctl enable nginx
systemctl status nginx
# By default, CentOS 7 built-in firewall is set to block Nginx traffic.
firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --zone=public --permanent --add-service=https

DSS_Linkis部署

由于DataSphere Studio依赖于Linkis,官方文档提供了两种部署方式:

  1. DSS & Linkis 一键部署
    该模式适合于DSS和Linkis都没有安装的情况。

    进入DSS & Linkis安装环境准备

  2. DSS 一键部署
    该模式适合于Linkis已经安装,需要安装DSS的情况。

    由于官方的一键部署不完美,Azkaban和Qualitis自动安装集成会有很多问题,很深的坑。

    本次选择的是第一种方式安装大部分组件,Azkaban和Qualitis手动部署的方式。

1
2
3
4
5
6
7
8
9
10
├── dss_linkis # 一键部署主目录
├── backup # 用于兼容Linkis老版本的安装启动脚本
├── bin # 用于一键安装启动DSS+Linkis
├── conf # 一键部署的配置文件
├── azkaban-solo-server-x.x.x.tar.gz # Azkaban安装包 Ps:本次安装使用的是分布式的Azkaban,单独与Linkis进行了对接。
├── linkis-jobtype-x.x.x.zip # Linkis jobtype安装包
├── wedatasphere-dss-x.x.x-dist.tar.gz # DSS后台安装包
├── wedatasphere-dss-web-x.x.x-dist.zip # DSS前端安装包
├── wedatasphere-linkis-x.x.x-dist.tar.gz # Linkis安装包
├── wedatasphere-qualitis-x.x.x.zip # Qualitis安装包 Ps:本次安装单独对Qualitis进行了部署,单独与Linkis进行了对接。
一键部署配置文件修改
修改config.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# 将conf目录下的config.sh.lite.template,修改为config.sh
cp conf/config.sh.lite.template conf/config.sh
vi conf/config.sh

# ssh默认端口
SSH_PORT=22
# 默认获取当前用户为部署用户
deployUser="`whoami`"
# 工作空间路径,默认为本地路径,尽量提前创建并授于写权限
WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis/
# 结果集路径,默认为本地路径,尽量提前创建并授于写权限
RESULT_SET_ROOT_PATH=file:///tmp/linkis
DSS_NGINX_IP=127.0.0.1 #DSS Nginx访问IP
DSS_WEB_PORT=8088 #DSS Web页面访问端口

# azkaban address for check
AZKABAN_ADRESS_IP=10.0.*.*
AZKABAN_ADRESS_PORT=8082

# qualitis address for check
QUALITIS_ADRESS_IP=10.0.*.*
QUALITIS_ADRESS_PORT=8099

# hive metastore
HIVE_META_URL=jdbc:mysql://10.0.*.*/metastore?useUnicode=true
HIVE_META_USER=database_username
HIVE_META_PASSWORD=database_password

# HADOOP CONF DIR
HADOOP_CONF_DIR=/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop

# HIVE CONF DIR
HIVE_CONF_DIR=/opt/cloudera/parcels/CDH/lib/hive/conf

# SPARK CONF DIR
SPARK_CONF_DIR=/opt/cloudera/parcels/CDH/lib/spark/conf

##########################################################################################
####The following parameters can be modified by the user as required, but not necessary###
##########################################################################################

# java application default jvm memory
export SERVER_HEAP_SIZE="512M"

LINKIS_VERSION=0.11.0

DSS_VERSION=0.9.0
修改db.sh
1
2
3
4
5
6
7
8
9
# 设置数据库的连接信息
MYSQL_HOST=10.0.*.*
MYSQL_PORT=3306
MYSQL_DB=webank
MYSQL_USER=database_username
MYSQL_PASSWORD=database_password

# 说明:此为必须配置参数,并确保可以从本机进行访问,验证方式:
mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD
增加环境变量
1
2
3
4
5
6
7
8
9
10
11
12
13
# 修改安装用户的.bash_rc,命令如下:
vi ~/.bash_rc

# 增加环境变量如下:
#JDK
export JAVA_HOME=/usr/java/jdk1.8.0_281
#HADOOP
export HADOOP_CONF_DIR=/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop
#Hive
export HIVE_CONF_DIR=/opt/cloudera/parcels/CDH/lib/hive/conf
#Spark
export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export SPARK_CONF_DIR=/opt/cloudera/parcels/CDH/lib/spark/conf
安装前必要指令确认
指令确认
1
该安装脚本会检查各项集成环境命令,如果没有请按照提示进行安装,以下命令为必须项: yum java mysql unzip expect telnet tar sed dos2unix nginx
alias 去掉cp、mv、rm的别名
1
2
3
4
5
6
安装过程如果有很多cp 命令提示您是否覆盖安装,说明您的系统配置有别名,输入alias,如果有cp、mv、rm的别名,如果有可以去掉,就不会再有大量提示
vi ~/.bashrc
注释掉以下三行
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
开始一键部署
1
2
3
4
5
6
7
8
9
10
1. 执行安装脚本:
sh bin/install.sh
注意:安装脚本有两处是相对路径,为了正确安装,请按照以上命令执行。

2. 安装步骤
install.sh脚本会询问您安装模式。 安装模式分为精简版、标准版,请根据您准备的环境情况,选择合适的安装模式。 本次选择的是标准版
install.sh脚本会询问您是否需要初始化数据库并导入元数据,linkis和dss 均会询问。 本次选择的是初始化数据库并导入元数据
第一次安装必须选是。

ps.为了数据库、元数据不出现问题,多次重复安装时,都选择初始化数据库并导入元数据。
登录Eureka页面确认服务启动情况

以下是正常运行时的Eureka服务

登录DSS Web页面确认

http://DSS_NGINX_IP:DSS_WEB_PORT 启动日志会打印此访问地址。

登陆时管理员的用户名和密码均为部署用户名,

如部署用户为root,则管理员的用户名/密码为:root/root。

至次部署完成

可能遇到的问题

DSS前端界面报错 [ Unexpected token < in JSON at position 0 ]

解决方法:

  1. 检查前端配置文件中linkis gateway的url配置

    确认到网关端口为9002

  1. 确认Nginx配置

    修改Nginx配置,后端Linkis的地址端口与网关9002一致

  2. Nginx reload 配置

1
nginx -s reload
DSS工作流创建失败 [ /api/rest_j/v1/bml/upload request failed! ]

修改如下配置,重启后问题解决

DSS创建工作流时报错 [ No FileSystem for scheme: hdfs问题 ]

该问题是linkis/linkis-bml报错,缺少FileSystem类。

在以下路径找到hadoop-common-*.jar

/opt/dss_linkis/linkis/linkis-bml/lib

修改jar包中的pom.xml文件,增加以下依赖。

1
2
3
4
5
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
<description>The FileSystem for hdfs: uris.</description>
</property>

找到与环境匹配的依赖类的jar包 hadoop-hdfs-client-*.jar

本次使用的是hadoop-hdfs-client-3.0.0-cdh6.3.2.jar

ps:jar包在这个位置有 /opt/cloudera/parcels/CDH/jars

将修改后的jar包和依赖的jar包(包名如下)放到 /opt/dss_linkis/linkis/linkis-bml/lib

hadoop-common-3.0.0-cdh6.3.2.jar

hadoop-hdfs-client-3.0.0-cdh6.3.2.jar

重启后问题解决

DSS工作流保存失败问题 [ /api/rest_j/v1/bml/upload request failed! ]

解决方法:

因为将修改网关端口为9002,而public service源码中网关端口为9001,所以需要在配置文件中覆盖端口配置。

修改如下配置,重启后问题解决

Scriptis数据库读取不到信息的问题

解决方法:

修改linkis-metadata配置文件

配置元数据库配置,重启linkis-metadata后该问题解决。

Scriptis查询完成后可视化报错

与webank官方确认,是Linkis0.11.0的Bug,在1.0版本修复。

Qualitis执行失败 [ NoSuchDatabaseException: Database ‘*****_ind’ not found; ]

解决方法:

需要在hive里面建立一个username_ind的数据库,然后授权给登录用户:username

Qualitis执行失败 [ java.lang.ClassNotFoundException: com.mysql.jdbc.Driver ]

解决方法:

在Spark 各个节点jars路径下增加mysql-connector-java-8.0.17.jar

Spark jars路径:/opt/cloudera/parcels/CDH/lib/spark/jars

工作流发布失败、Azkaban登录失败

Azkaban需要创建与DSS对应的用户,并赋权限。

创建完成后重启Azkaban后问题解决。

工作流Sendemail功能增加

微众平台工作流有邮件发送功能模块,但是开源代码中没有整合进来,需要自行修改源码增加这个功能。

源码地址:https://github.com/WeBankFinTech/DataSphereStudio

编译后,需要将jar包放置到指定的位置。 jar包需要放置到dss-server和linkis-appjoint-entrance两个微服务中,

以linkis-appjoint-entrance 为例(dss-server与linkis-appjoint-entrance一致), 在linkis-appjont-entrance下面的lib的同级目录有一个appjoints目录,目录层次如图所示

依赖包和已经实现的jar包统一放置到lib目录中。

lib依赖包清单如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
-rw-r--r-- 1 root root 62983 Mar 10 16:43 activation-1.1.jar
-rw-r--r-- 1 root root 3482 Mar 10 16:43 animal-sniffer-annotations-1.14.jar
-rw-r--r-- 1 root root 74557 Mar 10 16:43 annotations-2.0.1.jar
-rw-r--r-- 1 root root 445288 Mar 10 16:43 antlr-2.7.7.jar
-rw-r--r-- 1 root root 164368 Mar 10 16:43 antlr-runtime-3.4.jar
-rw-r--r-- 1 root root 4467 Mar 10 16:43 aopalliance-1.0.jar
-rw-r--r-- 1 root root 14765 Mar 10 16:43 aopalliance-repackaged-2.4.0-b31.jar
-rw-r--r-- 1 root root 44925 Mar 10 16:43 apacheds-i18n-2.0.0-M15.jar
-rw-r--r-- 1 root root 691479 Mar 10 16:43 apacheds-kerberos-codec-2.0.0-M15.jar
-rw-r--r-- 1 root root 243137 Mar 10 16:43 apache-el-8.5.24.2.jar
-rw-r--r-- 1 root root 16560 Mar 10 16:43 api-asn1-api-1.0.0-M20.jar
-rw-r--r-- 1 root root 79912 Mar 10 16:43 api-util-1.0.0-M20.jar
-rw-r--r-- 1 root root 139611 Mar 10 16:43 archaius-core-0.7.5.jar
-rw-r--r-- 1 root root 43581 Mar 10 16:43 asm-3.3.1.jar
-rw-r--r-- 1 root root 398782 Mar 10 16:43 asm-all-repackaged-2.4.0-b34.jar
-rw-r--r-- 1 root root 32041 Mar 10 16:43 asm-analysis-6.2.jar
-rw-r--r-- 1 root root 78919 Mar 10 16:43 asm-commons-6.2.jar
-rw-r--r-- 1 root root 50370 Mar 10 16:43 asm-tree-6.2.jar
-rw-r--r-- 1 root root 1930381 Mar 10 16:43 aspectjweaver-1.8.13.jar
-rw-r--r-- 1 root root 685403 Mar 10 16:43 bcpkix-jdk15on-1.56.jar
-rw-r--r-- 1 root root 3448507 Mar 10 16:43 bcprov-jdk15on-1.56.jar
-rw-r--r-- 1 root root 813288 Mar 10 16:43 bean-validator-2.4.0-b34.jar
-rw-r--r-- 1 root root 287192 Mar 10 16:43 cglib-2.2.2.jar
-rw-r--r-- 1 root root 343222 Mar 10 16:43 checker-qual-2.0.0.jar
-rw-r--r-- 1 root root 88035 Mar 10 16:43 class-model-2.4.0-b34.jar
-rw-r--r-- 1 root root 188671 Mar 10 16:43 commons-beanutils-1.7.0.jar
-rw-r--r-- 1 root root 206035 Mar 10 16:43 commons-beanutils-core-1.8.0.jar
-rw-r--r-- 1 root root 41123 Mar 10 16:43 commons-cli-1.2.jar
-rw-r--r-- 1 root root 284184 Mar 10 16:43 commons-codec-1.10.jar
-rw-r--r-- 1 root root 588337 Mar 10 16:43 commons-collections-3.2.2.jar
-rw-r--r-- 1 root root 751238 Mar 10 16:43 commons-collections4-4.1.jar
-rw-r--r-- 1 root root 241367 Mar 10 16:43 commons-compress-1.4.1.jar
-rw-r--r-- 1 root root 298829 Mar 10 16:43 commons-configuration-1.6.jar
-rw-r--r-- 1 root root 24239 Mar 10 16:43 commons-daemon-1.0.13.jar
-rw-r--r-- 1 root root 160519 Mar 10 16:43 commons-dbcp-1.4.jar
-rw-r--r-- 1 root root 143602 Mar 10 16:43 commons-digester-1.8.jar
-rw-r--r-- 1 root root 305001 Mar 10 16:43 commons-httpclient-3.1.jar
-rw-r--r-- 1 root root 185140 Mar 10 16:43 commons-io-2.4.jar
-rw-r--r-- 1 root root 299994 Mar 10 16:43 commons-jxpath-1.3.jar
-rw-r--r-- 1 root root 284220 Mar 10 16:43 commons-lang-2.6.jar
-rw-r--r-- 1 root root 61829 Mar 10 16:43 commons-logging-1.2.jar
-rw-r--r-- 1 root root 988514 Mar 10 16:43 commons-math-2.2.jar
-rw-r--r-- 1 root root 1599627 Mar 10 16:43 commons-math3-3.1.1.jar
-rw-r--r-- 1 root root 273370 Mar 10 16:43 commons-net-3.1.jar
-rw-r--r-- 1 root root 96221 Mar 10 16:43 commons-pool-1.5.4.jar
-rw-r--r-- 1 root root 8460 Mar 10 16:43 config-types-2.4.0-b34.jar
-rw-r--r-- 1 root root 69500 Mar 10 16:43 curator-client-2.7.1.jar
-rw-r--r-- 1 root root 186273 Mar 10 16:43 curator-framework-2.7.1.jar
-rw-r--r-- 1 root root 270342 Mar 10 16:43 curator-recipes-2.7.1.jar
-rw-r--r-- 1 root root 98365 Mar 10 16:43 curvesapi-1.04.jar
-rw-r--r-- 1 root root 50858 Mar 10 16:43 dss-appjoint-core-0.9.0.jar
-rw-r--r-- 1 root root 39775 Mar 10 16:43 dss-common-0.9.0.jar
-rw-r--r-- 1 root root 112806 Mar 10 16:43 dss-sendmail-appjoint-core-0.9.0.jar
-rw-r--r-- 1 root root 13704 Mar 10 16:43 error_prone_annotations-2.1.3.jar
-rw-r--r-- 1 root root 409687 Mar 10 16:42 eureka-client-1.7.2.jar
-rw-r--r-- 1 root root 252899 Mar 10 16:42 eureka-core-1.7.2.jar
-rw-r--r-- 1 root root 241622 Mar 10 16:42 gson-2.8.5.jar
-rw-r--r-- 1 root root 2734339 Mar 10 16:42 guava-25.1-jre.jar
-rw-r--r-- 1 root root 674028 Mar 10 16:42 guice-4.1.0.jar
-rw-r--r-- 1 root root 17385 Mar 10 16:42 hadoop-annotations-2.7.2.jar
-rw-r--r-- 1 root root 70685 Mar 10 16:42 hadoop-auth-2.7.2.jar
-rw-r--r-- 1 root root 3443040 Mar 10 16:42 hadoop-common-2.7.2.jar
-rw-r--r-- 1 root root 8268375 Mar 10 16:42 hadoop-hdfs-2.7.2.jar
-rw-r--r-- 1 root root 117910 Mar 10 16:42 HdrHistogram-2.1.10.jar
-rw-r--r-- 1 root root 3227 Mar 10 16:42 hk2-2.4.0-b34.jar
-rw-r--r-- 1 root root 173037 Mar 10 16:42 hk2-api-2.4.0-b31.jar
-rw-r--r-- 1 root root 205644 Mar 10 16:42 hk2-config-2.4.0-b34.jar
-rw-r--r-- 1 root root 117502 Mar 10 16:42 hk2-core-2.4.0-b34.jar
-rw-r--r-- 1 root root 179705 Mar 10 16:42 hk2-locator-2.4.0-b31.jar
-rw-r--r-- 1 root root 54379 Mar 10 16:42 hk2-runlevel-2.4.0-b34.jar
-rw-r--r-- 1 root root 118973 Mar 10 16:42 hk2-utils-2.4.0-b34.jar
-rw-r--r-- 1 root root 1475955 Mar 10 16:42 htrace-core-3.1.0-incubating.jar
-rw-r--r-- 1 root root 781831 Mar 10 16:42 httpclient-4.5.4.jar
-rw-r--r-- 1 root root 325123 Mar 10 16:42 httpcore-4.4.7.jar
-rw-r--r-- 1 root root 41752 Mar 10 16:42 httpmime-4.5.4.jar
-rw-r--r-- 1 root root 286596 Mar 10 16:42 hystrix-core-1.4.3.jar
-rw-r--r-- 1 root root 5925194 Mar 10 16:42 icu4j-4.6.jar
-rw-r--r-- 1 root root 8782 Mar 10 16:42 j2objc-annotations-1.1.jar
-rw-r--r-- 1 root root 67897 Mar 10 16:42 jackson-annotations-2.10.0.jar
-rw-r--r-- 1 root root 348635 Mar 10 16:42 jackson-core-2.10.0.jar
-rw-r--r-- 1 root root 228286 Mar 10 16:42 jackson-core-asl-1.9.2.jar
-rw-r--r-- 1 root root 1400944 Mar 10 16:42 jackson-databind-2.10.0.jar
-rw-r--r-- 1 root root 34395 Mar 10 16:42 jackson-datatype-jdk8-2.10.0.jar
-rw-r--r-- 1 root root 105749 Mar 10 16:42 jackson-datatype-jsr310-2.10.0.jar
-rw-r--r-- 1 root root 18323 Mar 10 16:42 jackson-jaxrs-1.9.2.jar
-rw-r--r-- 1 root root 27584 Mar 10 16:42 jackson-jaxrs-base-2.3.2.jar
-rw-r--r-- 1 root root 16511 Mar 10 16:42 jackson-jaxrs-json-provider-2.3.2.jar
-rw-r--r-- 1 root root 765648 Mar 10 16:42 jackson-mapper-asl-1.9.2.jar
-rw-r--r-- 1 root root 31636 Mar 10 16:42 jackson-module-jaxb-annotations-2.3.2.jar
-rw-r--r-- 1 root root 9327 Mar 10 16:42 jackson-module-parameter-names-2.10.0.jar
-rw-r--r-- 1 root root 43740 Mar 10 16:42 jackson-module-paranamer-2.10.0.jar
-rw-r--r-- 1 root root 475328 Mar 10 16:42 jackson-module-scala_2.11-2.10.0.jar
-rw-r--r-- 1 root root 27075 Mar 10 16:42 jackson-xc-1.9.2.jar
-rw-r--r-- 1 root root 13417 Mar 10 16:42 javacsv-2.0.jar
-rw-r--r-- 1 root root 84156 Mar 10 16:42 java-cup-10k.jar
-rw-r--r-- 1 root root 749499 Mar 10 16:42 javassist-3.19.0-GA.jar
-rw-r--r-- 1 root root 26586 Mar 10 16:42 javax.annotation-api-1.3.2.jar
-rw-r--r-- 1 root root 2497 Mar 10 16:42 javax.inject-1.jar
-rw-r--r-- 1 root root 5950 Mar 10 16:42 javax.inject-2.4.0-b34.jar
-rw-r--r-- 1 root root 95806 Mar 10 16:42 javax.servlet-api-3.1.0.jar
-rw-r--r-- 1 root root 36611 Mar 10 16:42 javax.websocket-api-1.0.jar
-rw-r--r-- 1 root root 161351 Mar 10 16:42 javax-websocket-client-impl-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 36872 Mar 10 16:42 javax-websocket-server-impl-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 115534 Mar 10 16:42 javax.ws.rs-api-2.0.1.jar
-rw-r--r-- 1 root root 105134 Mar 10 16:42 jaxb-api-2.2.2.jar
-rw-r--r-- 1 root root 890168 Mar 10 16:42 jaxb-impl-2.2.3-1.jar
-rw-r--r-- 1 root root 1975408 Mar 10 16:42 jaxrs-ri-2.21.jar
-rw-r--r-- 1 root root 18241792 Mar 10 16:42 jdk.tools-1.8.jar
-rw-r--r-- 1 root root 16610 Mar 10 16:42 jersey-apache-client4-1.19.1.jar
-rw-r--r-- 1 root root 134077 Mar 10 16:42 jersey-client-1.19.1.jar
-rw-r--r-- 1 root root 161168 Mar 10 16:42 jersey-client-2.21.jar
-rw-r--r-- 1 root root 694436 Mar 10 16:42 jersey-common-2.21.jar
-rw-r--r-- 1 root root 18102 Mar 10 16:42 jersey-container-servlet-2.23.1.jar
-rw-r--r-- 1 root root 66276 Mar 10 16:42 jersey-container-servlet-core-2.23.1.jar
-rw-r--r-- 1 root root 436795 Mar 10 16:42 jersey-core-1.19.1.jar
-rw-r--r-- 1 root root 68523 Mar 10 16:42 jersey-entity-filtering-2.16.jar
-rw-r--r-- 1 root root 960810 Mar 10 16:42 jersey-guava-2.21.jar
-rw-r--r-- 1 root root 165345 Mar 10 16:42 jersey-json-1.19.jar
-rw-r--r-- 1 root root 72763 Mar 10 16:42 jersey-media-jaxb-2.21.jar
-rw-r--r-- 1 root root 21769 Mar 10 16:42 jersey-media-json-jackson-2.16.jar
-rw-r--r-- 1 root root 64075 Mar 10 16:42 jersey-media-multipart-2.16.jar
-rw-r--r-- 1 root root 705416 Mar 10 16:42 jersey-server-1.19.1.jar
-rw-r--r-- 1 root root 944928 Mar 10 16:42 jersey-server-2.21.jar
-rw-r--r-- 1 root root 129319 Mar 10 16:42 jersey-servlet-1.19.1.jar
-rw-r--r-- 1 root root 21755 Mar 10 16:42 jersey-spring3-2.23.1.jar
-rw-r--r-- 1 root root 67758 Mar 10 16:42 jettison-1.1.jar
-rw-r--r-- 1 root root 539912 Mar 10 16:42 jetty-6.1.26.jar
-rw-r--r-- 1 root root 78505 Mar 10 16:42 jetty-annotations-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 281919 Mar 10 16:42 jetty-client-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 16681 Mar 10 16:42 jetty-continuation-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 195948 Mar 10 16:42 jetty-http-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 135153 Mar 10 16:42 jetty-io-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 54852 Mar 10 16:42 jetty-plus-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 93072 Mar 10 16:42 jetty-security-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 591927 Mar 10 16:42 jetty-server-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 110825 Mar 10 16:42 jetty-servlet-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 93192 Mar 10 16:42 jetty-servlets-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 177131 Mar 10 16:42 jetty-util-6.1.26.jar
-rw-r--r-- 1 root root 499219 Mar 10 16:42 jetty-util-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 128603 Mar 10 16:42 jetty-webapp-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 52044 Mar 10 16:42 jetty-xml-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 581571 Mar 10 16:42 joda-time-2.3.jar
-rw-r--r-- 1 root root 90660 Mar 10 16:42 json4s-ast_2.11-3.5.3.jar
-rw-r--r-- 1 root root 689977 Mar 10 16:42 json4s-core_2.11-3.5.3.jar
-rw-r--r-- 1 root root 48193 Mar 10 16:42 json4s-jackson_2.11-3.5.3.jar
-rw-r--r-- 1 root root 614765 Mar 10 16:42 json4s-scalap_2.11-3.5.3.jar
-rw-r--r-- 1 root root 100636 Mar 10 16:42 jsp-api-2.1.jar
-rw-r--r-- 1 root root 19936 Mar 10 16:42 jsr305-3.0.2.jar
-rw-r--r-- 1 root root 4596 Mar 10 16:42 jul-to-slf4j-1.7.25.jar
-rw-r--r-- 1 root root 29779 Mar 10 16:42 LatencyUtils-2.0.3.jar
-rw-r--r-- 1 root root 1045744 Mar 10 16:42 leveldbjni-all-1.8.jar
-rw-r--r-- 1 root root 280950 Mar 10 16:42 linkis-common-0.9.4.jar
-rw-r--r-- 1 root root 10698 Mar 10 16:42 linkis-hadoop-common-0.9.4.jar
-rw-r--r-- 1 root root 120891 Mar 10 16:42 linkis-httpclient-0.9.4.jar
-rw-r--r-- 1 root root 199384 Mar 10 16:42 linkis-module-0.9.4.jar
-rw-r--r-- 1 root root 248096 Mar 10 16:42 linkis-protocol-0.9.4.jar
-rw-r--r-- 1 root root 415357 Mar 10 16:42 linkis-storage-0.9.4.jar
-rw-r--r-- 1 root root 489884 Mar 10 16:42 log4j-1.2.17.jar
-rw-r--r-- 1 root root 255485 Mar 10 16:42 log4j-api-2.10.0.jar
-rw-r--r-- 1 root root 1597622 Mar 10 16:42 log4j-core-2.10.0.jar
-rw-r--r-- 1 root root 23996 Mar 10 16:42 log4j-jul-2.10.0.jar
-rw-r--r-- 1 root root 24173 Mar 10 16:42 log4j-slf4j-impl-2.10.0.jar
-rw-r--r-- 1 root root 388864 Mar 10 16:42 mail-1.4.jar
-rw-r--r-- 1 root root 352776 Mar 10 16:42 micrometer-core-1.0.5.jar
-rw-r--r-- 1 root root 62135 Mar 10 16:42 mimepull-1.9.3.jar
-rw-r--r-- 1 root root 1006904 Mar 10 16:42 mysql-connector-java-5.1.49.jar
-rw-r--r-- 1 root root 6690 Mar 10 16:42 netflix-commons-util-0.1.1.jar
-rw-r--r-- 1 root root 69073 Mar 10 16:42 netflix-eventbus-0.3.0.jar
-rw-r--r-- 1 root root 101813 Mar 10 16:42 netflix-infix-0.3.0.jar
-rw-r--r-- 1 root root 13772 Mar 10 16:42 netflix-statistics-0.1.1.jar
-rw-r--r-- 1 root root 1199572 Mar 10 16:42 netty-3.6.2.Final.jar
-rw-r--r-- 1 root root 3780056 Mar 10 16:42 netty-all-4.1.17.Final.jar
-rw-r--r-- 1 root root 167898 Mar 10 16:42 netty-buffer-4.0.27.Final.jar
-rw-r--r-- 1 root root 143916 Mar 10 16:42 netty-codec-4.0.27.Final.jar
-rw-r--r-- 1 root root 426622 Mar 10 16:42 netty-codec-http-4.0.27.Final.jar
-rw-r--r-- 1 root root 420854 Mar 10 16:42 netty-common-4.0.27.Final.jar
-rw-r--r-- 1 root root 238957 Mar 10 16:42 netty-handler-4.0.27.Final.jar
-rw-r--r-- 1 root root 316806 Mar 10 16:42 netty-transport-4.0.27.Final.jar
-rw-r--r-- 1 root root 96651 Mar 10 16:42 netty-transport-native-epoll-4.0.27.Final.jar
-rw-r--r-- 1 root root 15555629 Mar 10 16:42 ooxml-schemas-1.3.jar
-rw-r--r-- 1 root root 657585 Mar 10 16:42 org.eclipse.wst.xml.xpath2.processor-2.1.100.jar
-rw-r--r-- 1 root root 20235 Mar 10 16:42 osgi-resource-locator-1.0.1.jar
-rw-r--r-- 1 root root 34654 Mar 10 16:42 paranamer-2.8.jar
-rw-r--r-- 1 root root 2701171 Mar 10 16:42 poi-3.17.jar
-rw-r--r-- 1 root root 1479023 Mar 10 16:42 poi-ooxml-3.17.jar
-rw-r--r-- 1 root root 5924600 Mar 10 16:42 poi-ooxml-schemas-3.17.jar
-rw-r--r-- 1 root root 533455 Mar 10 16:42 protobuf-java-2.5.0.jar
-rw-r--r-- 1 root root 129763 Mar 10 16:42 reflections-0.9.10.jar
-rw-r--r-- 1 root root 86725 Mar 10 16:42 ribbon-2.2.5.jar
-rw-r--r-- 1 root root 83270 Mar 10 16:42 ribbon-core-2.2.5.jar
-rw-r--r-- 1 root root 17688 Mar 10 16:42 ribbon-eureka-2.2.5.jar
-rw-r--r-- 1 root root 54587 Mar 10 16:42 ribbon-httpclient-2.2.5.jar
-rw-r--r-- 1 root root 161793 Mar 10 16:42 ribbon-loadbalancer-2.2.5.jar
-rw-r--r-- 1 root root 40769 Mar 10 16:42 ribbon-transport-2.2.5.jar
-rw-r--r-- 1 root root 1127547 Mar 10 16:42 rxjava-1.2.0.jar
-rw-r--r-- 1 root root 402675 Mar 10 16:42 rxnetty-0.4.9.jar
-rw-r--r-- 1 root root 53987 Mar 10 16:42 rxnetty-contexts-0.4.9.jar
-rw-r--r-- 1 root root 29155 Mar 10 16:42 rxnetty-servo-0.4.9.jar
-rw-r--r-- 1 root root 15487351 Mar 10 16:42 scala-compiler-2.11.8.jar
-rw-r--r-- 1 root root 5744974 Mar 10 16:43 scala-library-2.11.8.jar
-rw-r--r-- 1 root root 802818 Mar 10 16:43 scalap-2.11.8.jar
-rw-r--r-- 1 root root 423753 Mar 10 16:43 scala-parser-combinators_2.11-1.0.4.jar
-rw-r--r-- 1 root root 4573750 Mar 10 16:43 scala-reflect-2.11.8.jar
-rw-r--r-- 1 root root 647891 Mar 10 16:43 scala-xml_2.11-1.0.4.jar
-rw-r--r-- 1 root root 224619 Mar 10 16:43 servo-core-0.10.1.jar
-rw-r--r-- 1 root root 174154 Mar 10 16:43 servo-internal-0.10.1.jar
-rw-r--r-- 1 root root 32127 Mar 10 16:43 slf4j-api-1.7.12.jar
-rw-r--r-- 1 root root 297518 Mar 10 16:43 snakeyaml-1.19.jar
-rw-r--r-- 1 root root 366340 Mar 10 16:43 spring-aop-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 660545 Mar 10 16:43 spring-beans-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 930680 Mar 10 16:43 spring-boot-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 461472 Mar 10 16:43 spring-boot-actuator-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 379277 Mar 10 16:43 spring-boot-actuator-autoconfigure-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 1162436 Mar 10 16:43 spring-boot-autoconfigure-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 592 Mar 10 16:43 spring-boot-starter-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 612 Mar 10 16:43 spring-boot-starter-actuator-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 599 Mar 10 16:43 spring-boot-starter-aop-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 590 Mar 10 16:43 spring-boot-starter-jetty-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 645 Mar 10 16:43 spring-boot-starter-json-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 596 Mar 10 16:43 spring-boot-starter-log4j2-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 588 Mar 10 16:43 spring-boot-starter-web-2.0.3.RELEASE.jar
-rw-r--r-- 1 root root 13138 Mar 10 16:43 spring-bridge-2.4.0-b34.jar
-rw-r--r-- 1 root root 128382 Mar 10 16:43 spring-cloud-commons-2.0.0.RELEASE.jar
-rw-r--r-- 1 root root 35195 Mar 10 16:43 spring-cloud-config-client-2.0.0.RELEASE.jar
-rw-r--r-- 1 root root 104246 Mar 10 16:43 spring-cloud-context-2.0.0.RELEASE.jar
-rw-r--r-- 1 root root 429798 Mar 10 16:43 spring-cloud-netflix-core-1.4.4.RELEASE.jar
-rw-r--r-- 1 root root 93697 Mar 10 16:43 spring-cloud-netflix-eureka-client-1.4.4.RELEASE.jar
-rw-r--r-- 1 root root 2227 Mar 10 16:43 spring-cloud-starter-2.0.0.RELEASE.jar
-rw-r--r-- 1 root root 2315 Mar 10 16:43 spring-cloud-starter-config-2.0.0.RELEASE.jar
-rw-r--r-- 1 root root 3870 Mar 10 16:43 spring-cloud-starter-eureka-1.4.4.RELEASE.jar
-rw-r--r-- 1 root root 2429 Mar 10 16:43 spring-cloud-starter-netflix-archaius-1.4.4.RELEASE.jar
-rw-r--r-- 1 root root 2491 Mar 10 16:43 spring-cloud-starter-netflix-eureka-client-1.4.4.RELEASE.jar
-rw-r--r-- 1 root root 2359 Mar 10 16:43 spring-cloud-starter-netflix-ribbon-1.4.4.RELEASE.jar
-rw-r--r-- 1 root root 1090739 Mar 10 16:43 spring-context-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 178021 Mar 10 16:43 spring-context-support-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 1226075 Mar 10 16:43 spring-core-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 280032 Mar 10 16:43 spring-expression-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 21703 Mar 10 16:43 spring-jcl-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 72378 Mar 10 16:43 spring-security-crypto-5.0.6.RELEASE.jar
-rw-r--r-- 1 root root 19745 Mar 10 16:43 spring-security-rsa-1.0.5.RELEASE.jar
-rw-r--r-- 1 root root 1254656 Mar 10 16:43 spring-web-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 789866 Mar 10 16:43 spring-webmvc-5.0.7.RELEASE.jar
-rw-r--r-- 1 root root 161867 Mar 10 16:43 stax2-api-3.1.4.jar
-rw-r--r-- 1 root root 26514 Mar 10 16:43 stax-api-1.0.1.jar
-rw-r--r-- 1 root root 23346 Mar 10 16:43 stax-api-1.0-2.jar
-rw-r--r-- 1 root root 148627 Mar 10 16:43 stringtemplate-3.2.1.jar
-rw-r--r-- 1 root root 17506 Mar 10 16:43 tiger-types-1.4.jar
-rw-r--r-- 1 root root 63777 Mar 10 16:43 validation-api-1.1.0.Final.jar
-rw-r--r-- 1 root root 42354 Mar 10 16:43 websocket-api-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 35643 Mar 10 16:43 websocket-client-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 196553 Mar 10 16:43 websocket-common-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 35307 Mar 10 16:43 websocket-server-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 21327 Mar 10 16:43 websocket-servlet-9.4.11.v20180605.jar
-rw-r--r-- 1 root root 486013 Mar 10 16:43 woodstox-core-asl-4.4.1.jar
-rw-r--r-- 1 root root 2009630 Mar 10 16:43 xerces2-xsd11-2.11.1.jar
-rw-r--r-- 1 root root 1229125 Mar 10 16:43 xercesImpl-2.9.1.jar
-rw-r--r-- 1 root root 51073 Mar 10 16:43 xlsx-streamer-1.2.1.jar
-rw-r--r-- 1 root root 220536 Mar 10 16:43 xml-apis-1.4.01.jar
-rw-r--r-- 1 root root 2666695 Mar 10 16:43 xmlbeans-2.3.0.jar
-rw-r--r-- 1 root root 15010 Mar 10 16:43 xmlenc-0.52.jar
-rw-r--r-- 1 root root 7188 Mar 10 16:43 xmlpull-1.1.3.1.jar
-rw-r--r-- 1 root root 84091 Mar 10 16:43 xml-resolver-1.2.jar
-rw-r--r-- 1 root root 24956 Mar 10 16:43 xpp3_min-1.1.4c.jar
-rw-r--r-- 1 root root 589803 Mar 10 16:43 xstream-1.4.10.jar
-rw-r--r-- 1 root root 94672 Mar 10 16:43 xz-1.0.jar
-rw-r--r-- 1 root root 792964 Mar 10 16:43 zookeeper-3.4.6.jar

重启DSS服务

修改网关端口为9002后需确认

网关修改后需要确认dss_application表中各组件配置的端口号,要与配置信息一致。

微众平台文档连接

DSS github 地址

https://github.com/WeBankFinTech/DataSphereStudio

Linkis github 地址

https://github.com/WeBankFinTech/Linkis

Scriptis github 地址

https://github.com/WeBankFinTech/Scriptis

Visualis github 地址

https://github.com/WeBankFinTech/Visualis

Qualitis github 地址

https://github.com/WeBankFinTech/Qualitis

Azkaban github 地址

https://azkaban.github.io/

部署文档(仅供参考)

https://github.com/WeBankFinTech/DataSphereStudio/blob/master/docs/zh_CN/ch2/DSS_LINKIS_Quick_Install.md

DSS 用户使用手册

https://github.com/WeBankFinTech/DataSphereStudio/blob/b7ed704228a86b9235936f84f119cb33907e5f5f/docs/zh_CN/ch3/DSS_User_Manual.md

DSS 快速使用手册

https://github.com/WeBankFinTech/DataSphereStudio/blob/b7ed704228a86b9235936f84f119cb33907e5f5f/docs/zh_CN/ch3/DataSphere_Studio_QuickStart.md

Scriptis SparkSQL demo

https://github.com/WeBankFinTech/DataSphereStudio/blob/b7ed704228a86b9235936f84f119cb33907e5f5f/docs/zh_CN/ch3/DSS_User_Tests3_SparkSQL.md

DSS 接入第三方系统指南

https://github.com/WeBankFinTech/DataSphereStudio/blob/master/docs/zh_CN/ch4/%E7%AC%AC%E4%B8%89%E6%96%B9%E7%B3%BB%E7%BB%9F%E6%8E%A5%E5%85%A5DSS%E6%8C%87%E5%8D%97.md

DSS 手动接入Azkaban 说明文档

https://github.com/WeBankFinTech/DataSphereStudio/blob/b7ed704228a86b9235936f84f119cb33907e5f5f/docs/zh_CN/ch4/%E5%A6%82%E4%BD%95%E6%8E%A5%E5%85%A5%E8%B0%83%E5%BA%A6%E7%B3%BB%E7%BB%9FAzkaban.md

JobType Azkaban 说明文档

https://github.com/WeBankFinTech/DataSphereStudio/blob/master/docs/zh_CN/ch2/Azkaban_LinkisJobType_Deployment_Manual.md

DSS 安装常见问题(官方提供)

https://github.com/WeBankFinTech/DataSphereStudio/blob/master/docs/zh_CN/ch1/DSS%E5%AE%89%E8%A3%85%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98%E5%88%97%E8%A1%A8.md

Linkis 安装常见问题(官方提供)

https://github.com/WeBankFinTech/Linkis/wiki/Q&A-%E9%97%AE%E9%A2%98%E6%80%BB%E7%BB%93

官方Q&A 总结

https://github.com/WeBankFinTech/Linkis/wiki/Q&A-%E9%97%AE%E9%A2%98%E6%80%BB%E7%BB%93