Connecting to Kafka running in Docker

The most common Kafka-in-Docker failure: Streambench connects to localhost:9092 and fetches metadata fine, but consuming or producing hangs and times out. Streambench detects this and points you here.

Why it happens

When a client connects, the broker replies with the addresses of the brokers to really talk to — its advertised.listeners. Inside Docker that's often an internal hostname like kafka:9092, which doesn't resolve on your Mac. The initial connection works; everything after it silently can't connect.

The fix: two listeners

Give the broker one listener for containers and one advertised as localhost for your Mac:

services:
  kafka:
    image: apache/kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: >-
        PLAINTEXT://:29092,CONTROLLER://:29093,PLAINTEXT_HOST://:9092
      KAFKA_ADVERTISED_LISTENERS: >-
        PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >-
        PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Containers connect to kafka:29092; Streambench (and anything else on your Mac) connects to localhost:9092.

Redpanda equivalents: --kafka-addr and --advertise-kafka-addr with internal/external pairs.