基于Kettle开发的web版数据集成开源工具(data-integration)-部署篇

*️⃣主目录:ETL&ELT专栏

🔼下一集:基于Kettle开发的web版数据集成开源工具(data-integration)-介绍篇

📚第一章 前言

📗背景

在前面的ETL专栏中有提到,最近一直在寻找基于Kettle开发的web版开源工具,后面都放弃准备研究kettle源码了,结果遇到了一款,今天先来研究下Linux环境手动部署(官方给的都是docker方式部署,需要搭建docker、mvn、node、npm等环境,太复杂了!)

📗目的

本身公司有数据中台产品,只是需要Kettle任务绘制这一块内容

📗总体方向

参考该开源产品核心代码

📚第二章 下载编译

📗下载

下载地址:https://github.***/young-datafan-ooooo1/data-integration

📗编译

导入开发工具进行编译:

mvn clean install -Prelease  -Dcheckstyle.skip=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true


📚第三章 部署

📗准备工作

📕 安装数据库&redis&consul

  • 数据源:使用MySQL数据库,没有的自行安装(docker安装mysql-建议本地手动安装mysql,这是之前安装docker练手用的,需要可以参考),博主使用的是8.0版本,建库语句如下供参考
    CREATE DATABASE `dataintegration_db`;
    create user 'stelladp'@'%' identified by 'Renxiaozhao@2023';
    grant create,alter,drop,select,insert,update,delete,INDEX,REFERENCES on dataintegration_db.* to stelladp@'%';
    flush privileges; 
    
  • 安装redis,可参照redis安装使用及告警处理
  • 安装consul,参照Linux安装consul的两种方式(在线和离线)

📕 修改配置文件的数据库、redis、consul信息

修改所有模块application-local.yamlbootstrap.yaml文件,涉及数据库、redis、consul的地方全都改掉



📘 /dataintegration-gateway/src/main/resources/application-local.yaml修改用户认证服务SSO
id: dataintegration-***mon-sso-provider
#uri: lb://dataintegration-***mon-sso-provider
uri: http://localhost:10217/oauth/

📗服务器-应用目录结构

├─data-integration
│  └─bin     --存放启停脚本
│  └─conf    --存放配置文件(暂未用到,直接使用的jar包中的脚本)
│  └─lib     --jar包文件
│  └─logs    --日志目录
│  └─ui      --前端部署文件

📗重新编译并上传jar包

修改完配置,重新编译项目,找到所有模块的target目录下的jar包(有兴趣的可以尝试修改pom.xml文件,统一打包到某一路径下),上传到服务器


订正:只需要provider目录下面的jar包和gateway的jar包

-rw-rw-r--. 1 opensource opensource  93704942 12 18:06 dataintegration-file-management-provider-1.0.0-SNAPSHOT.jar
-rw-rw-r--. 1 opensource opensource  66606827 12 18:04 dataintegration-gateway-1.0.0-SNAPSHOT.jar
-rw-rw-r--. 1 opensource opensource  55597256 12 18:04 dataintegration-group-provider-1.0.0-SNAPSHOT.jar
-rw-rw-r--. 1 opensource opensource 161231342 12 18:05 dataintegration-model-management-provider-1.0.0-SNAPSHOT.jar
-rw-rw-r--. 1 opensource opensource  55634210 12 18:03 dataintegration-project-provider-1.0.0-SNAPSHOT.jar
-rw-rw-r--. 1 opensource opensource 179893827 12 18:04 dataintegration-run-management-provider-1.0.0-SNAPSHOT-ark-biz.jar
-rw-rw-r--. 1 opensource opensource  71467663 12 18:05 dataintegration-sso-provider-1.0.0-SNAPSHOT.jar
-rw-rw-r--. 1 opensource opensource  63207137 12 18:05 dataintegration-sys-management-provider-1.0.0-SNAPSHOT.jar

📗启动后台服务

启动脚本如下:

#! /bin/bash
APP_LIB=/home/opensource/app/data-integration/lib
APP_LOGS=/home/opensource/app/data-integration/logs

echo "启动系统管理模块..................."
exec nohup java -jar ${APP_LIB}/dataintegration-sys-management-provider-1.0.0-SNAPSHOT.jar > ${APP_LOGS}/sys.log 2>&1 &

echo "启动分组管理模块..................."
exec nohup java -jar ${APP_LIB}/dataintegration-group-provider-1.0.0-SNAPSHOT.jar > ${APP_LOGS}/group.log 2>&1 &

echo "启动服务网关模块..................."
exec nohup java -jar ${APP_LIB}/dataintegration-gateway-1.0.0-SNAPSHOT.jar > ${APP_LOGS}/gateway.log 2>&1 &

echo "启动脚本管理模块..................."
exec nohup java -jar ${APP_LIB}/dataintegration-project-provider-1.0.0-SNAPSHOT.jar > ${APP_LOGS}/project.log 2>&1 &

echo "启动单点登录模块..................."
exec nohup java -jar ${APP_LIB}/dataintegration-sso-provider-1.0.0-SNAPSHOT.jar > ${APP_LOGS}/sso.log 2>&1 &

echo "启动模型管理模块..................."
exec nohup java -jar ${APP_LIB}/dataintegration-model-management-provider-1.0.0-SNAPSHOT.jar > ${APP_LOGS}/model.log 2>&1 &

echo "启动文件管理模块..................."
exec nohup java -jar ${APP_LIB}/dataintegration-file-management-provider-1.0.0-SNAPSHOT.jar > ${APP_LOGS}/file.log 2>&1 &

echo "启动数据集成运行模块..................."
#exec nohup java -jar ${APP_LIB}/dataintegration-run-management-provider-1.0.0-SNAPSHOT.jar  > ${APP_LOGS}/run.log 2>&1 &
#exec nohup java -jar ${APP_LIB}/dataintegration-run-management-provider-1.0.0-SNAPSHOT-ark-biz.jar  > ${APP_LOGS}/run.log 2>&1 &
exec nohup java -jar ${APP_LIB}/dataintegration-run-management-provider-1.0.0-SNAPSHOT-ark-executable.jar  > ${APP_LOGS}/run.log 2>&1 &

其中文件模块和运行模块启动失败,不知道有没有影响,先跳过不管,最后会统一放到问题记录章节处理

订正file服务需要hdfs等存储环境,启动失败只会影响涉及文件的任务无法使用,run服务启动失败,会导致页面报错,很多查询接口都会报错,按照上面的脚本启动应该没有问题,一开始启动失败是因为选错了jar

📕consul监控页面可以看到启动成功的服务


file服务忽略,正常应该有7个服务

📗前端部署

可参照smartKettle离线部署及问题记录中的前端部署章节,需要一些基础环境,这里不在赘述

📕编译

  • 博主使用的VSCode工具,导入前端项目,首先npm install
  • 编译打包npm run build

  • dist目录对应就是部署文件
  • 上传到服务器

📕nginx配置

同样参照参照smartKettle离线部署及问题记录中的nginx配置章节,需要安装nginx,这里不在赘述

server {
    listen       8785;
    server_name  localhost;

    location / {
        root /home/opensource/app/data-integration/ui/dist;
        try_files $uri $uri/ /index.html;
        index  index.html index.htm;
    }
    location /cloud {
        alias /home/opensource/app/data-integration/ui/dist;
        index  index.html index.htm;
        try_files $uri $uri/ /cloud/index.html;     #4.重定向,内部文件的指向
    }

    location /api {
        proxy_pass http://localhost:10200/api/;
        proxy_set_header Host $host:$server_port;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header REMOTE-HOST $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
}

📕登录验证

用户密码admin/Prime@2020,登录失败😭

错误原因参考下面的问题二,修复后可以正常登录

run服务启动成功后,也面正常展示

⁉️问题记录

❓问题一:-cp方式启动报错:找不到类

❗解决方式:直接jar包启动

❓问题二:Failed to handle request…10200/api/dataintegration-***mon-sso-provider/oauth/token

2024-01-02 16:37:16.466 [traceCode:] [reactor-http-epoll-8] ERROR c.y.g.c.g.GatewayJsonExceptionHandler - Failed to handle request ....10200/api/dataintegration-***mon-sso-provider/oauth/token]: Search domain query failed. Original hostname: 'localhost' failed to resolve 'localhost.localdomain' after 2 queries 
io.***ty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'localhost' failed to resolve 'localhost.localdomain' after 2 queries 
        at io.***ty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1013)
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
        |_ checkpoint ⇢ springfox.boot.starter.autoconfigure.SwaggerUiWebFluxConfiguration$CustomWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$OAuth2ResourceServerSpec$BearerTokenAuthenticationWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ HTTP POST "/api/dataintegration-***mon-sso-provider/oauth/token" [ExceptionHandlingWebHandler]
Stack trace:
                at io.***ty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1013)
                at io.***ty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:966)
                at io.***ty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:414)
                at io.***ty.resolver.dns.DnsResolveContext.a***ess$600(DnsResolveContext.java:63)
                at io.***ty.resolver.dns.DnsResolveContext$2.operation***plete(DnsResolveContext.java:463)
                at io.***ty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
                at io.***ty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
                at io.***ty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
                at io.***ty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
                at io.***ty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
                at io.***ty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
                at io.***ty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
                at io.***ty.resolver.dns.DnsQueryContext.tryFailure(DnsQueryContext.java:225)
                at io.***ty.resolver.dns.DnsQueryContext$4.run(DnsQueryContext.java:177)
                at io.***ty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
                at io.***ty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
                at io.***ty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
                at io.***ty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
                at io.***ty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
                at io.***ty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
                at io.***ty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                at io.***ty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
                at java.lang.Thread.run(Thread.java:750)
Caused by: io.***ty.resolver.dns.DnsNameResolverTimeoutException: [/114.114.114.114:53] query via UDP timed out after 5000 milliseconds (no stack trace available)
2024-01-02 16:38:14.771 [traceCode:] [scheduling-1] INFO  c.y.g.r.GatewayRateLimitSyncTask - start syncRateLimit ,规则记录数:0 ,耗时:33
2024-01-02 16:38:15.635 [traceCode:] [scheduling-1] INFO  c.y.g.route.GatewayRouteSyncTask - sta

❕原因:hosts文件被注释掉了

❗解决方式:取消注释,重启gateway服务

放开注释:

不要忘了重启gateway服务:

exec nohup java -jar /home/opensource/app/data-integration/lib/dataintegration-gateway-1.0.0-SNAPSHOT.jar > /home/opensource/app/data-integration/logs/gateway.log 2>&1 &

❓问题三:新建集成操作报错


❕原因:运行模块没启动成功导致的

查看gateway日志, dataintegration-di-run-management-provider没启动成功导致

  • dataintegration-run-management-provider-1.0.0-SNAPSHOT.jar 中没有主清单属性
    [opensource@bigdata02 logs]$ less run.log 
    nohup: 忽略输入
    /home/opensource/app/data-integration/lib/dataintegration-run-management-provider-1.0.0-SNAPSHOT.jar中没有主清单属性
    
    • 错误解释:缺少Main-Class属性
  • 解决方式:启动dataintegration-run-management-provider-1.0.0-SNAPSHOT-ark-executable.jar

    dataintegration-run-management-provider下面有三个jar包,真正配置DiRunManagementApplication的是dataintegration-run-management-provider-1.0.0-SNAPSHOT-ark-biz.jar结果走了弯路,实际应该启动的是dataintegration-run-management-provider-1.0.0-SNAPSHOT-ark-executable.jar

    弯路从这开始…(又没选对启动jar包

❗解决方式:正确应该启动的是dataintegration-run-management-provider-1.0.0-SNAPSHOT-ark-executable.jar

正确应该是启动dataintegration-run-management-provider-1.0.0-SNAPSHOT-ark-executable.jar

启动成功后,页面正常

❓问题五:DiRunManagementApplication启动报错·Exception in thread “main” java.lang.NoClassDefFoundError: org/springframework/boot/builder/SpringApplicationBuilder

Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/boot/builder/SpringApplicationBuilder
        at ***.youngdatafan.di.run.management.DiRunManagementApplication.main(DiRunManagementApplication.java:29)
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.builder.SpringApplicationBuilder
        at java.***.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
转载请说明出处内容投诉
CSS教程_站长资源网 » 基于Kettle开发的web版数据集成开源工具(data-integration)-部署篇

发表评论

欢迎 访客 发表评论

一个令你着迷的主题!

查看演示 官网购买