In this article, we will take a step-by-step look at how to configure the Debezium Kafka connector so that it starts sending metrics in Prometheus format over the HTTP protocol.

We will make the configuration by customizing the Docker image. At the end of the article we will get a Docker image ready for further manipulation (including deployment to production).

Before you start

In order to run the code examples given in the article, make sure that you have installed:

• cURL
• Docker
• Docker Compose

Step 1. Select the basic Docker image

We use debezium/connect:1.6 as the base Docker image.

Why this one? It’s simple: version 1.6 is the most recent stable version at the time of writing.

FROM debezium/connect:1.6

Step 2. Export JMX Mbeans in Prometheus format

The official Debezium documentation states that JMX can be used to monitor the connector. In fact, JMX Mbeans are the only available way to monitor not only Kafka connectors, but also Apache Zookeeper, as well as Apache Kafka itself.

The Debezium documentation suggests configuring JMX via environment variables. This is not a mandatory step for exporting metrics to Prometheus. In our case, it is necessary to correctly install and configure JMS Exporter Java agent.

In the README of this project, it is strongly recommended to use it specifically as a Java agent, but it is also allowed to install it as an independent http server (this option will not be considered in this article).

Step 2.1. Download JMX Exporter

Download JMX Exporter to the directory $KAFKA_HOME/etc:

RUN mkdir -p $KAFKA_HOME/etc && cd $KAFKA_HOME/etc && \
    curl -so jmx_prometheus_javaagent.jar \
    https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.jar

Step 2.2. Describe the configuration for JMX Exporter

The configuration for JMX Exporter must be described in YAML format. As a basic configuration, let’s take this example from the debezium-examples repository:

RUN cd $KAFKA_HOME/etc && \
    curl -so metrics-config.yml \
    https://raw.githubusercontent.com/debezium/debezium-examples/cf8eaf05f3259dc51c2f9316f6c14a3b39b185fb/monitoring/debezium-jmx-exporter/config.yml

A more detailed description of all configuration options can be found in the relevant README section of prometheus/jmx_exporter project.

Step 2.3. Set the javaagent option for the connector

All that remains is to put everything together by specifying the javaagent option for the Kafka connector. The environment variable KAFKA_OPTS, which is used to set all the JVM parameters of the java process of the Kafka connector, will help us in this:

ARG METRICS_PORT=8080
ENV KAFKA_OPTS="-javaagent:$KAFKA_HOME/etc/jmx_prometheus_javaagent.jar=$METRICS_PORT:$KAFKA_HOME/etc/metrics-config.yml"

If the javaagent option is set correctly, then the metrics should be available inside the Docker container at http://localhost:8080/metrics .

Below we will see this in practice.

The final Dockerfile

Throughout the article, we wrote the Dockerfile for the Kafka connector line by line. If you put them together, the resulting Dockerfile will look like this:

FROM debezium/connect:1.6

ARG JMX_AGENT_VERSION=0.16.1
ARG METRICS_PORT=8080

# Downloading jmx_prometheus_javaagent
RUN mkdir -p $KAFKA_HOME/etc && cd $KAFKA_HOME/etc && \
    curl -so jmx_prometheus_javaagent.jar \
    https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/$JMX_AGENT_VERSION/jmx_prometheus_javaagent-$JMX_AGENT_VERSION.jar

# Downloading example config for jmx_prometheus_javaagent
RUN cd $KAFKA_HOME/etc && \
    curl -so metrics-config.yml \
    https://raw.githubusercontent.com/debezium/debezium-examples/cf8eaf05f3259dc51c2f9316f6c14a3b39b185fb/monitoring/debezium-jmx-exporter/config.yml

# Adding `-javaagent` argument to KAFKA_OPTS (JAVA_OPTS of Kafka process)
ENV KAFKA_OPTS="-javaagent:$KAFKA_HOME/etc/jmx_prometheus_javaagent.jar=$METRICS_PORT:$KAFKA_HOME/etc/metrics-config.yml"

Check the correctness of the result

If we try to run the resulting Docker image, we will get the following error:

Connection to node -1 (/0.0.0.0:9092) could not be established.
Broker may not be available.

In order to fix this error, we need to run a container with Apache Kafka. In turn, Apache Kafka requires Apache Zookeeper for its work, which we will similarly run in a container.

Here, docker-compose will come to our aid, which will simplify our work with this set of Docker containers. I have prepared the docker-compose.yml file, in which I described the minimum configuration required to start the local Kafka connector:

services:
  zookeeper:
    image: debezium/zookeeper:1.6

  kafka:
    image: debezium/kafka:1.6
    links:
      - zookeeper
    environment:
      - ZOOKEEPER_CONNECT=zookeeper:2181

  debezium-connector:
    build:
      context: .
      args:
        METRICS_PORT: 8080
    ports:
      - '8080:8080'
    links:
      - kafka
    environment:
      - BOOTSTRAP_SERVERS=kafka:9092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_connect_statuses

This docker-compose.yml file should be placed next to the Dockerfile file described above.

It would be very convenient to take and run this bunch with a single command. Said – done!

Description of what the combo command below does:

• creates a temporary folder and goes to it,
• downloads the ‘docker-compose’ files.ymlandDockerfile`,
• launches all services declared in docker-compose.yml (including building a Docker image from a Dockerfile),
• waits 60 seconds for the debezium connector to become available on localhost:8080,
• and finally executes a command that outputs metrics in Prometheus format to the console,
• after that, it stops all Docker containers, regardless of the success of the previous instructions.

cd $(mktemp -d) && \
curl -sO https://iakunin.dev/posts/debezium-kafka-connector-metrics/code/Dockerfile && \
curl -sO https://iakunin.dev/posts/debezium-kafka-connector-metrics/code/docker-compose.yml && \
docker-compose up --build --detach && \
timeout 60 bash -c "while true; do if curl -I 'localhost:8080' 2>&1 | grep -wq 'HTTP/1.1 200 OK'; then exit 0; fi done" && \
echo "\n\n\n\n\n" && \
curl 'http://localhost:8080/metrics' && \
echo "\n\n\n\n\n" ; \
docker-compose down

The resulting output of this command should be something like this.

If the stdout in your console is similar to it, then I did not deceive you and set up the metrics for the Kafka Debezium connector absolutely correctly. If something went wrong, then let me know about it (you can find my contacts in the site header).