Debezium is providing out of the box CDC solution from various databases. In my last blog post, I have published how to configure the Debezium MySQL connector. This is the next part of that post. Once we deployed the debezium, to we need some kind of monitoring to keep track of whats happening in the debezium connector. Luckily Debezium has its own metrics that are already integrated with the connectors. We just need to capture them using the JMX exporter agent. Here I have written how to monitor Debezium MySQL connector with Prometheus and Grafana. But the dashboard is having the basic metrics only. You can build your own dashboard for more detailed monitoring.
Install JMX exporter in Kafka Distributed connector:
All the connectors are managed by the Kafka connect(Distributed or standalone). In our previous blog, we used Distributed Kafka connect service. So we are going to modify the distributed service binary file.
Download the JMX exporter.
Create config file.
Add the JMX export to the Kafka connect binary File.
Restart the Distributed Connect Service.
Verify the JMX Agent installation.
Get the debezium metrics.
You can these metrics in your browser as well.
Im using a separate server for Prometheus and Grafana.
Create a user for Prometheus:
Create Directories for Prometheus:
Download the Prometheus binary files:
Copy the binary files to respective locations:
Create a Prometheus config file:
Set permission for config file:
Create a Prometheus systemctl file:
Start the Prometheus Service:
Add Debezium MySQL connector metrics to Prometheus:
Restart the Prometheus service:
Check the status:
In your browser Open the below URL.
It’ll start listening to the port 3000. The default username and password admin/admin. You can change once you logged in.
Add the Debezium MySQL Dashboard:
This dashboard is taken from the official Debezium’s example repo. But they gave this for MSSQL Server. With some changes and fixes, we can use the same for MySQL and other databases. I made it as a template.
In grafana add the Prometheus datasource.
Click on Add Data source, select Prometheus.
URL: localhost:9090 (I have installed grafana and Prometheus on the same server, If you have different server for Prometheus, use that IP instead of localhost).
Click on Save & Test.
You’ll get a pop-up message that its is connected.
Now go to the dashboards page and import the Template JSON.
Click on Import button.
Copy the Template JSON file from here. Paste it or download the JSON file and choose the upload button. Now the dashboard is ready. You can see a few basic metrics.
Debezium is a great platform for who wants to do real-time analytics. But in terms of monitoring, still, I feel it should get more contribution. This template is just a kickstart. We can build a more detailed monitoring dashboard for the debezium connectors. Please feel free to contribute to repo. Pull requests are welcome. Lets make the debezium more powerful.