The Data Guy

Monitor Debezium MySQL Connector With Prometheus And Grafana


Debezium is providing out of the box CDC solution from various databases. In my last blog post, I have published how to configure the Debezium MySQL connector. This is the next part of that post. Once we deployed the debezium, to we need some kind of monitoring to keep track of whats happening in the debezium connector. Luckily Debezium has its own metrics that are already integrated with the connectors. We just need to capture them using the JMX exporter agent. Here I have written how to monitor Debezium MySQL connector with Prometheus and Grafana. But the dashboard is having the basic metrics only. You can build your own dashboard for more detailed monitoring.

Reference: List of Debezium monitoring metrics

Install JMX exporter in Kafka Distributed connector:

All the connectors are managed by the Kafka connect(Distributed or standalone). In our previous blog, we used Distributed Kafka connect service. So we are going to modify the distributed service binary file.
Download the JMX exporter.

    cd /opt/jmx/
    mv jmx_prometheus_javaagent-0.12.0.jar jmx-exporter.jar

Create config file.

vi /opt/jmx/config.yml
startDelaySeconds: 0
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false
- pattern : "kafka.connect<type=connect-worker-metrics>([^:]+):"
  name: "kafka_connect_connect_worker_metrics_$1"
- pattern : "kafka.connect<type=connect-metrics, client-id=([^:]+)><>([^:]+)"
  name: "kafka_connect_connect_metrics_$2"
    client: "$1"
- pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^,]+), key=([^>]+)><>RowsScanned"
  name: "debezium_metrics_RowsScanned"
    plugin: "$1"
    name: "$3"
    context: "$2"
    table: "$4"
- pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^>]+)>([^:]+)"
  name: "debezium_metrics_$4"
    plugin: "$1"
    name: "$3"
    context: "$2"

Add the JMX export to the Kafka connect binary File.

    vi /usr/bin/connect-distributed
    -- Find this line below export CLASSPATH
    exec $(dirname $0)/kafka-run-class $EXTRA_ARGS org.apache.kafka.connect.cli.ConnectDistributed "[email protected]"
    --Replace with
    exec $(dirname $0)/kafka-run-class $EXTRA_ARGS -javaagent:/opt/jmx/jmx-exporter.jar=7071:/opt/jmx/config.yml org.apache.kafka.connect.cli.ConnectDistributed "[email protected]"

Restart the Distributed Connect Service.

    systemctl restart confluent-connect-distributed

Verify the JMX Agent installation.

    netstat -tulpn | grep 7071
    tcp6       0      0 :::7071                 :::*                    LISTEN      2885/java

Get the debezium metrics.

    curl localhost:7071 | grep debezium
    :-debezium_metrics_NumberOfDisconnects{context="binlog",name="mysql-db01",plugin="mysql",} 0.

You can these metrics in your browser as well.


Install Prometheus

Im using a separate server for Prometheus and Grafana.

Create a user for Prometheus:

    sudo useradd --no-create-home --shell /bin/false prometheus

Create Directories for Prometheus:

    sudo mkdir /etc/prometheus
    sudo mkdir /var/lib/prometheus
    sudo chown prometheus:prometheus /etc/prometheus
    sudo chown prometheus:prometheus /var/lib/prometheus

Download the Prometheus binary files:

    cd /tmp
    tar -zxvf prometheus-2.15.0.linux-amd64.tar.gz

Copy the binary files to respective locations:

    cd prometheus-2.15.0.linux-amd64
    cp prometheus /usr/local/bin/
    cp promtool /usr/local/bin/
    sudo chown prometheus:prometheus /usr/local/bin/prometheus
    sudo chown prometheus:prometheus /usr/local/bin/promtool
    cp -r consoles /etc/prometheus
    cp -r console_libraries /etc/prometheus
    sudo chown -R prometheus:prometheus /etc/prometheus/consoles
    sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Create a Prometheus config file:

vi  /etc/prometheus/prometheus.yml
  scrape_interval: 15s

  - job_name: 'prometheus'
    scrape_interval: 5s
      - targets: ['localhost:9090']

Set permission for config file:

    sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Create a Prometheus systemctl file:

    vi /etc/systemd/system/prometheus.service
    ExecStart=/usr/local/bin/prometheus \
        --config.file /etc/prometheus/prometheus.yml \
        --storage.tsdb.path /var/lib/prometheus/ \
        --web.console.templates=/etc/prometheus/consoles \

Start the Prometheus Service:

    sudo systemctl daemon-reload
    sudo systemctl start prometheus
    sudo systemctl enable prometheus

Add Debezium MySQL connector metrics to Prometheus:

vi  /etc/prometheus/prometheus.yml

  - job_name: debezium
    scrape_interval: 5s
      - targets:
          - debezium-node-ip:7071

Restart the Prometheus service:

    sudo systemctl restart prometheus

Check the status:

In your browser Open the below URL.


Install Grafana:

    sudo dpkg -i grafana_6.5.2_amd64.deb
    sudo systemctl daemon-reload
    sudo systemctl start grafana-server

It’ll start listening to the port 3000. The default username and password admin/admin. You can change once you logged in.


Add the Debezium MySQL Dashboard:

This dashboard is taken from the official Debezium’s example repo. But they gave this for MSSQL Server. With some changes and fixes, we can use the same for MySQL and other databases. I made it as a template.
In grafana add the Prometheus datasource.


Click on Add Data source, select Prometheus.

Click on Save & Test.

You’ll get a pop-up message that its is connected.

Now go to the dashboards page and import the Template JSON.


Click on Import button.

Copy the Template JSON file from here. Paste it or download the JSON file and choose the upload button. Now the dashboard is ready. You can see a few basic metrics.


Debezium is a great platform for who wants to do real-time analytics. But in terms of monitoring, still, I feel it should get more contribution. This template is just a kickstart. We can build a more detailed monitoring dashboard for the debezium connectors. Please feel free to contribute to repo. Pull requests are welcome. Lets make the debezium more powerful.

Debezium Series blogs:

  1. Build Production Grade Debezium Cluster With Confluent Kafka
  2. Monitor Debezium MySQL Connector With Prometheus And Grafana
  3. Debezium MySQL Snapshot From Read Replica With GTID
  4. Debezium MySQL Snapshot From Read Replica And Resume From Master
  5. Debezium MySQL Snapshot For AWS RDS Aurora From Backup Snaphot
  6. RealTime CDC From MySQL Using AWS MSK With Debezium
· Kafka, grafana, prometheus, debezium, mysql, jmx, monitoring


Loading Comments