Deployment of Database Synchronization Tool Debezium

Debezium can monitor the incremental changes of the following databases:
MongoDB
MySQL
PostgreSQL
SQL Server
Oracle
Db2
Cassandra

Then it synchronizes the incremental data to Kafka.
When the data is synchronized to Kafka, we can choose to develop our own tool to read data from Kafka, and then perform data cleansing from Kafka.
Alternatively, we can use the official Kafka Connect tool, which supports synchronization to places like ElasticSearch.

Next, let's demonstrate how to deploy Debezium.

Deploying Kafka

Since Debezium needs to synchronize incremental data to Kafka, we must first create a Kafka instance.
You can quickly deploy using docker-compose, refer to https://kafka.whuanle.cn

Or quickly deploy for practice using Docker.

docker run -itd --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 quay.io/debezium/zookeeper

docker run -itd --name kafka -p 9092:9092 --link zookeeper:zookeeper quay.io/debezium/kafka

Then you can deploy a kafdrop panel to view Kafka.

docker run -dit -p 9000:9000 \
-e JVM_OPTS=" -Xms32M -Xmx64M " \
-e KAFKA_BROKERCONNECT=192.168.0.0:9092 \
-e SERVER_SERVLET_CONTEXTPATH="/" \
obsidiandynamics/kafdrop

Replace 192.168.0.0:9092 with the specific IP.

Deploying MySQL

If you already have a MySQL database, you can skip this step.

docker run -d -itd --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=debezium -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw debezium/example-mysql

Deploying Debezium

docker run -itd --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link kafka:kafka quay.io/debezium/connect:2.3

Since the source on Docker hub is relatively old, it is recommended to use quay.io/debezium/connect:{version}.
Repository address https://quay.io/repository/debezium/connect?tab=tags

Then deploy a management panel.

docker run -itd --name debezium-ui -p 8080:8080 -e KAFKA_CONNECT_URIS=http://192.168.0.0:8083 debezium/debezium-ui

Remember to replace http://192.168.0.0:8083

Configuring Synchronization

Open port 8080.

Then click Validate. If the validation is successful, click Next to proceed.


You need to fill in the database you want to monitor.

Additionally, you need to enable the configuration for Kafka automatic topic creation.