Kafka Consumer

Kafka consumer for event streaming in Ecotone

To quickly get up and running with consuming existing Topics, we can use Kafka Consumer feature. Simply by marking given method with Kafka Consumer attribute, we are getting access to asynchronous process that will now run and consume Message from defined Topic.

Kafka Consumer

To set up Consumer, consuming from given topics, all we need to do, is to mark given method with KafkaConsumer attribute:

#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders']
)]
public function handle(string $payload, array $metadata): void
{
    // do something
}

Then we run it as any other asynchronous consumer, using orderConsumer name.

Dynamic Topics

Besides literal topic names, topics also accepts an expression string, resolved once the first time the consumer subscribes. This lets the topic list come from a referenced service or a configuration variable instead of being fixed at deploy time, which is useful for multi-tenant setups where topics get added without a redeploy.

#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: "reference('tenantTopics')->getOrderTopics()"
)]
public function handle(string $payload, array $metadata): void
{
    // do something
}

Topics can also be resolved from a configuration variable using parameter:

#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: "parameter('app.multi_tenancy.kafka.orders.topics')"
)]
public function handle(string $payload, array $metadata): void
{
    // do something
}

A topics entry is treated as an expression whenever it contains (; anything else is used as a literal topic name, exactly as before. Expressions can be mixed with literal topic names in the same array, and results still go through topic reference-name mapping. There is no Message available yet when topics are resolved, so payload and headers cannot be used here — only reference and parameter.

Read more about reference and parameter expressions in related section.

Providing group id

By default Consumer Group id will be same as endpoint id, however we can provide customized name if needed:

#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders'],
    groupId: 'order_subscriber'
)]
public function handle(string $payload, array $metadata): void
{
    // do something
}

Providing custom rdkafka configuration

You can modify your Message Consumer with specific rdkafka configuration:

#[ServiceContext] 
public function distributedPublisher()
{
    return KafkaConsumerConfiguration::createWithDefaults(
        endpointId: 'orderConsumers'
    )
        ->setConfiguration('auto.offset.reset', 'earliest');
}

Kafka Headers

We can accesss specific Kafka Headers using standard Ecotone's metadata mechanism

• kafka_topic - Topic name for incoming message
• kafka_partition - Partition of incoming message
• kafka_offset - Offset of Message Consumer

#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders']
)]
public function handle(
    string $payload, 
    #[Header("kafka_topic")] string $topicName,
): void
{
    // do something
}

Automatic Conversion

If we have our Custom Conversion or JMS Module installed, then we can leverage automatic conversion:

#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders']
)]
public function handle(Order $payload): void
{
    // do something
}

Read more about Conversion in related section.

Default Content Type

Messages published by external systems may come without contentType header, and then Ecotone does not know from which format the payload should be deserialized. If our Topic follows known format, we can state it explicitly using ContentType attribute:

#[ContentType('application/json')]
#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders']
)]
public function handle(Order $payload): void
{
    // do something
}

If Message already carries contentType header, it stays untouched — the attribute works as a default. To always enforce given Media Type, we can use replaceIfExists flag:

#[ContentType('application/json', replaceIfExists: true)]

Read more about Default Content Type in related section.

Instant Retry

In case of failure we may try to recover instantly. To do so we can provide InstantRetry attribute:

#[InstantRetry(retryTimes: 3)]
#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders']
)]
public function handle(Order $payload): void
{
    // handle
}

Read more about Instant Retry in related section.

Error Channel

To handle Message when failure happen, we may decide to send it to Error Channel. This can be used to for example store the Message for later review.

#[ErrorChannel('customErrorChannel')]
#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders']
)]
public function handle(Order $payload): void
{
    // handle
}

Read more about Error Channel in related section.

Final Failure Strategy

We can also define in case of unrecoverable failure, what should happen:

#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders'],
    finalFailureStrategy: FinalFailureStrategy::RESEND
)]
public function handle(Order $payload): void
{
    // handle
}

Read more about final failure strategy in related section.

Deduplication

We can define custom deduplication key to ensure no same Message will be handled twice.

#[Deduplicated('orderId')]
#[KafkaConsumer(
    endpointId: 'orderConsumers', 
    topics: ['orders']
)]
public function handle(Order $payload): void
{
    // handle
}

Read more about deduplication in related section.