One article teaches you how to monitor Kafka Topic's producer client

introduction

Apache Kafka is a widely used message queueing and stream processing platform in modern distributed systems. In a real production environment, understanding which clients are producing messages to a specific Topic is an important task for operation and troubleshooting. This article will introduce in detail how to fully grasp the producer information of Kafka Topic through command line tools, JMX monitoring, log analysis and other methods, and will also be equipped with a guide to installing and using Kafka command line tools.

1. Why monitor Kafka producers

In a production environment, Kafka Topic's source may involve multiple microservices or clients. If a Topic has message stacking, delaying, or exception data, we need to quickly locate:

• Which applications are writing to the Topic?
• What are the producer's IP address and client ID?
• Is the producer's write rate normal?

Mastering this information can help:

• Troubleshooting the backlog of messages
• Source of audit data
• Optimize Kafka cluster performance

2. Method 1: Use the Kafka command line tool

(1) Install Kafka command line tool

The Kafka command line tool is included in the Kafka distribution, and the installation steps are as follows:

# Download Kafka (taking 3.7.0 as an example)wget /kafka/3.7.0/kafka_2.13-3.7.
tar -xzf kafka_2.13-3.7.
cd kafka_2.13-3.7.0

# Make sure Java is installed (Kafka depends on Java to run)sudo apt install openjdk-17-jdk  # Ubuntu/Debian
sudo yum install java-17-openjdk-devel  # CentOS/RHEL

# Verify installationbin/ --version

(2) View active producers

# List all consumer groups (some producer information may be displayed here)bin/ --bootstrap-server localhost:9092 --list

# Get the write offset of Topic (indirectly judge the producer activity)bin/  \
  --broker-list localhost:9092 \
  --topic my-topic \
  --time -1

Output example:

my-topic:0:12345
my-topic:1:67890

Represents the latest offsets for partitions 0 and 1 of my-topic.

3. Method 2: Monitor Kafka Producers via JMX

Kafka exposes a wealth of JMX metrics that can monitor the writes of producers.

(1) Enable JMX

# Enable JMX when starting Kafkaexport JMX_PORT=9999
bin/ config/ &

(2) Connect using JConsole

Run jconsole (Java comes with its own tools).

Connect localhost:9999.

View :type=BrokerTopicMetrics,name=MessagesInPerSec to get the write rate of Topic.

(3) Query JMX using the command line

# Use jcmd to view metrics (Java 11+ required)jcmd <Kafka_PID>  | grep Producer

4. Method 3: Analyze Kafka Broker Logs

The Kafka Broker log is located in logs/ by default, from which producer information can be extracted:

# View the recent producer connectionsgrep "ProducerId" logs/

# Filter by client IPgrep "Accepted connection from" logs/ | awk '{print $NF}'

5. Method 4: Use the Kafka AdminClient API

If you need to programmatically obtain producer information, you can use AdminClient:

import .*;

public class KafkaProducerMonitor {
    public static void main(String[] args) {
        Properties props = new Properties();
        (AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        
        try (AdminClient admin = (props)) {
            ListConsumerGroupsResult groups = ();
            ().get().forEach(::println);
        }
    }
}

6. Method 5: Network traffic monitoring

If Kafka does not enable authentication, you can analyze the producer IP by catching packets:

# Use tcpdump to crawl Kafka traffic (port 9092)sudo tcpdump -i eth0 port 9092 -A | grep "PRODUCE"

7. Summary and Best Practices

Best Practice Recommendations:

• The production environment enables ACL to restrict unauthorized client access.
• Combined with Prometheus + Grafana to monitor producer metrics for a long time.
• Regularly audit Topic write sources to avoid abuse of unknown clients.

Conclusion

This article details 5 ways to monitor Kafka producers, covering command line, JMX, logs, APIs, and network analytics. Choosing the right method depends on your specific needs. It is recommended to combine multiple methods to achieve comprehensive monitoring.

This is the article about this article teaching you how to monitor Kafka Topic's producer client. For more information on monitoring Kafka Topic's producer client, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!