Kafka, Python, and the _wait_on_metadata Problem

When Python’s KafkaProducer is simply stuck.

Google couldn’t help me fixing the problem, so I decided to quickly write up how I solved it to be of use for future generations.

Essence of my setup

I use kafka-python, json encoders and some loop to continuously emit messages.

I want to emit tuples for different topics, thus I assign a variable as topic (op_id).

from kafka import KafkaProducer #kafka-python package
from json import dumps
my_producer = KafkaProducer(bootstrap_servers='{}:9092'.format('my.server.ip'),
                                 value_serializer=lambda x:
                                 dumps(x).encode('utf-8'))
op_id = 1
while True:
     my_producer.send(op_id, "hello world")

It did not work.

I tested a command line producer, consumer, etc.

bin/kafka-console-consumer.sh — bootstrap-server my.server.ip:9092 — topic 1

Everything is fine, i.e., my Kafka and Zookeeper work nicely. I overwrite the advertised listeners from the Kafka config (I call it with a script, you can also directly change that value in a config. I need to switch server-IPs thus it’s more convenient for me this way)

bin/kafka-server-start.sh config/server.properties --override advertised.listeners=PLAINTEXT://my.server.ip:9092

Everything worked. Only my producer did not send.

The answer was the Integer

my_producer.send(str(op_id), "hello world")

The hidden logic behind it was that Kafka could not find the topic (usually Kafka generates unknown topics automatically and even though this might not be best practice, for my research purposes it’s sufficient). Obviously, Integer values aren’t good topics. Or are secretly turned into String-Topics.

So when Pythons KafkaProducer refuses to send (in my case I did not even get an error, only the _wait_on_metadata information after killing the send-process) — CHECK YOUR TOPICS.

Cheers