Flink CDC 3.3.0 实战：手把手搭建 MySQL 到 StarRocks 的秒级实时数仓

1. 核心版本与兼容性要求Flink CDC 版本3.3.0Flink 引擎版本1.19.3MySQL 版本8.0.39️ 注意不支持 8.4 及以上版本因废弃了SHOW MASTER STATUS语法StarRocks 版本3.2.6(Docker allin1 镜像)1.1 Flink 1.19.3 下载与解压# 下载 Flink 1.19.3 官方发行包 curl -O https://dlcdn.apache.org/flink/flink-1.19.3/flink-1.19.3-bin-scala_2.12.tgz # 解压并重命名目录 tar -xzf flink-1.19.3-bin-scala_2.12.tgz mv flink-1.19.3 flink-1.19.3-bin1.2 Flink CDC 3.3.0 下载与解压# 下载 Flink CDC 3.3.0 官方发行包 curl -L -O https://github.com/apache/flink-cdc/releases/download/release-3.3.0/flink-cdc-3.3.0-bin.tar.gz # 解压并重命名目录 tar -xzf flink-cdc-3.3.0-bin.tar.gz mv flink-cdc-3.3.0 flink-cdc-3.3.0-bin2. Flink CDC 依赖包下载与放置规范2.1 依赖包下载命令 (curl)# 1. 下载 Flink CDC 3.3.0 核心包 curl -O https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-dist/3.3.0/flink-cdc-dist-3.3.0.jar # 2. 下载 MySQL Pipeline 连接器 curl -O https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-mysql/3.3.0/flink-cdc-pipeline-connector-mysql-3.3.0.jar # 3. 下载 StarRocks Pipeline 连接器 curl -O https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-starrocks/3.3.0/flink-cdc-pipeline-connector-starrocks-3.3.0.jar # 4. 下载 MySQL JDBC 驱动 curl -O https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar2.2 Jar 包放置规范 (极其重要)Flink CDC 专属 lib 目录(flink-cdc-3.3.0/lib/)flink-cdc-dist-3.3.0.jarflink-cdc-pipeline-connector-mysql-3.3.0.jarflink-cdc-pipeline-connector-starrocks-3.3.0.jarFlink 引擎基础 lib 目录(flink-1.19.3/lib/)mysql-connector-java-8.0.27.jar(注该目录下绝对不能包含任何flink-cdc开头的包否则会导致类加载冲突)3. MySQL 8.0.39 部署与配置3.1 Docker 启动命令docker run -d --name mysql8 \ -e MYSQL_ROOT_PASSWORDQwer1234 \ -p 3306:3306 \ -v mysql_data:/var/lib/mysql \ mysql:8.0.39 \ --log-binmysql-bin \ --binlog-formatROW \ --binlog-row-imageFULL \ --server-id13.2 权限配置GRANT SELECT, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO root%; FLUSH PRIVILEGES;4. StarRocks 3.2.6 部署与配置4.1 Docker 启动命令docker run -d \ --name starrocks \ --privilegedtrue \ -p 8030:8030 \ -p 9030:9030 \ -p 8040:8040 \ -e HOST_TYPE1 \ starrocks/allin1-ubuntu:3.2.64.2 密码设置与验证# 进入容器 docker exec -it starrocks bash # 无密码登录并修改密码 mysql -h 127.0.0.1 -P 9030 -uroot ALTER USER root% IDENTIFIED BY Qwer1234; FLUSH PRIVILEGES; exit # 验证 BE 状态 mysql -h 127.0.0.1 -P 9030 -uroot -pQwer1234 -e SHOW BACKENDS\G # 确认 Alive: true 且 HttpPort: 80405. Flink CDC 任务配置 (mysql-to-starrocks.yaml)source: type: mysql hostname: localhost port: 3306 username: root password: Qwer1234 # 根据mysql时区配置 server-time-zone: UTC # 指定同步数据库及数据表 tables: test_db.t_user_info sink: type: starrocks load-url: 127.0.0.1:8030 jdbc-url: jdbc:mysql://127.0.0.1:9030 username: root password: Qwer1234 sink.buffer-flush.max-bytes: 94371840 sink.buffer-flush.interval-ms: 15000 # 路由规则将源端带横杠的库名映射到目标端合规的库名(mysql支持-连接符 starrocks不支持) route: - source-table: test_db.t_user_info sink-table: test_db.t_user_info pipeline: name: MySQL to StarRocks Sync parallelism: 16. 任务提交与运维6.1 启动 Flink 集群并提交任务# 启动集群 ./flink-1.19.3-bin/bin/start-cluster.sh # 提交任务 cd flink-cdc-3.3.0-bin ./bin/flink-cdc.sh ./mysql-to-starrocks.yaml6.2 停止任务# 查看 Job ID ./flink-1.19.3-bin/bin/flink list # 取消任务 ./flink-1.19.3-bin/bin/flink cancel JobID7. 常见问题排查速查表错误现象根本原因解决方案ClassCastException/NoClassDefFoundErrorJar 包放错目录CDC 专属包放flink-cdc/libJDBC 驱动放flink/libCannot read the binlog filename... SHOW MASTER STATUSMySQL 版本过高降级至 MySQL 8.0.xUnexpected input -/Failed to create database库名包含特殊字符提前在 StarRocks 建库并在 YAML 中配置route规则Connection refused: 127.0.0.1:8040BE HTTP 端口未映射重建 StarRocks 容器加上-p 8040:8040Cant connect to local MySQL server through socket容器内服务未就绪等待 1-2 分钟并使用-h 127.0.0.1强制 TCP 连接8. 环境验证清单Flink Web UI (http://localhost:8081) 能够正常访问。StarRocks 容器内SHOW BACKENDS显示Alive: true。宿主机可以通过mysql -h 127.0.0.1 -P 9030成功连接 StarRocks。Flink CDC 任务提交后状态为RUNNING。在 MySQL 插入测试数据后StarRocks 能够在几秒内查询到该数据。