Real-World Scenarios

Complete end-to-end application examples showing Kafka APIs in production contexts.

Scenario 1: Event-Driven Microservice

Complete microservice integrating Producer, Consumer, and Admin APIs with error handling, monitoring, and graceful shutdown.

See the complete implementation in the original index.md lines 1678-1960, which includes:

• Multi-API integration (Producer + Consumer + Admin)
• Automatic topic management
• Error handling with DLQ
• Health metrics collection
• Graceful shutdown patterns
• Rebalance handling

Key Features:

• Consumes from input-events topic
• Transforms and produces to output-events topic
• Sends errors to error-events topic
• Automatic topic creation
• Manual offset management
• Health metrics endpoint

Code Location: See reference/complete-applications.md

Scenario 2: Multi-Topic Aggregator

Stateful aggregation across multiple topics with external state checkpointing.

See the complete implementation in the original index.md lines 1963-2123, which includes:

• Multiple topic consumption (orders, payments, shipments)
• Stateful aggregation with in-memory store
• External state checkpointing on rebalance
• State restoration on partition assignment

Key Features:

• Aggregates data from 3 topics
• Maintains state per key
• Checkpoints state before rebalance
• Produces aggregated results

Code Location: See reference/complete-applications.md

Scenario 3: Configuration Validation

Utility for validating Kafka configurations before deployment.

See the complete implementation in the original index.md lines 2126-2343, which includes:

• Producer configuration validation
• Consumer configuration validation
• Interdependency checking
• Detailed error messages

Key Features:

• Validates required properties
• Checks configuration interdependencies
• Validates idempotence requirements
• Validates transaction requirements
• Provides configuration summary

Code Location: See reference/complete-applications.md

Scenario 4: Exactly-Once Processing Pipeline

Consume-transform-produce with exactly-once semantics using transactions.

See the complete implementation in the original index.md lines 2347-2418, which includes:

• Transactional producer configuration
• Read-committed consumer configuration
• Transaction lifecycle management
• Offset management within transactions

Key Features:

• Exactly-once semantics
• Transaction error handling
• Offset commits within transaction
• Automatic retry on abort

Code Location: See reference/complete-applications.md

Scenario 5: Dead Letter Queue with Retry

DLQ pattern with configurable retry attempts and comprehensive error metadata.

See the complete implementation in the original index.md lines 2422-2561, which includes:

• Configurable retry attempts
• Retry count tracking via headers
• Comprehensive error metadata
• DLQ routing after max retries

Key Features:

• Header-based retry counting
• Exponential backoff (implicit)
• Error metadata preservation
• Critical error alerting

Code Location: See reference/complete-applications.md

Additional Patterns

For more advanced patterns, see:

• Advanced Patterns