Apache Kafka 4.1.1

Apache Kafka is a distributed event streaming platform that provides three key capabilities:

1. Message Broker: Publish and subscribe to streams of events with high throughput and low latency
2. Durable Storage: Store streams of events reliably for as long as needed
3. Stream Processing: Process streams of events in real-time or retrospectively

Package Information

Maven Coordinates:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.13</artifactId>
    <version>4.1.1</version>
</dependency>

For client-only applications (Producer/Consumer):

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>4.1.1</version>
</dependency>

Quick Start

Get started quickly with Kafka:

• Quick Start Guide - Get up and running in minutes
• Real-World Scenarios - Complete application examples

API Selection Guide

Decision Guide

Use Producer API when:

• Publishing events to Kafka topics
• Need high throughput message ingestion
• Require exactly-once semantics with transactions
• Building data pipelines that write to Kafka

Use Consumer API when:

• Reading events from Kafka topics
• Need consumer group coordination for load balancing
• Require at-least-once or exactly-once consumption
• Building microservices that react to events

Use ShareConsumer API (New in 4.1.1) when:

• Multiple consumers need to process from same partitions cooperatively
• Need explicit record acknowledgement (ACCEPT/RELEASE/REJECT)
• Implementing work queue patterns with automatic retry
• Require delivery count tracking for dead-letter queues

Use Admin API when:

• Managing Kafka cluster resources programmatically
• Automating topic/configuration operations
• Monitoring consumer groups and lag
• Building operational tools and dashboards

Use Streams API when:

• Processing data in real-time as it flows through Kafka
• Need stateful transformations and aggregations
• Building stream processing applications
• Require exactly-once processing guarantees end-to-end

Use Connect API when:

• Integrating Kafka with external systems (databases, filesystems, etc.)
• Need pre-built connectors for common systems
• Building reusable data integration components
• Require schema management and data transformation

Core APIs

Client APIs

• Producer API - Publish streams of records to Kafka topics

  • Thread-safe, asynchronous sending
  • Exactly-once semantics with transactions
  • Custom partitioning strategies

• Consumer API - Subscribe to topics and process streams of records

  • Consumer groups for load balancing
  • Manual and automatic offset management
  • ShareConsumer for cooperative consumption (New in 4.1.1)

• Admin API - Manage Kafka resources programmatically

  • Topic management (create, delete, describe, configure)
  • Consumer group management and monitoring
  • ACL and quota management

Streams API

• Streams Overview - Build and run stream processing applications
• Streams DSL - High-level functional DSL for transformations
• State Stores - Persistent and in-memory state management
• Processor API - Low-level API for custom processing
• Streams Testing - Testing utilities for Kafka Streams

Connect API

• Connector Framework - Build data integration connectors
• Data Schemas - Schema system for structured data
• Transformations - Single message transformations

Common Components

• Error Handling - 189 exceptions with handling strategies
• Security - SSL/TLS, SASL authentication, ACL authorization
• Serialization - Built-in and custom serializers

Core Imports Reference

Producer

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.clients.producer.Callback;
import org.apache.kafka.clients.producer.ProducerConfig;

Consumer

import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.ConsumerConfig;

Admin

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.clients.admin.CreateTopicsResult;

Streams

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;

Connect

import org.apache.kafka.connect.source.SourceConnector;
import org.apache.kafka.connect.source.SourceTask;
import org.apache.kafka.connect.sink.SinkConnector;
import org.apache.kafka.connect.sink.SinkTask;
import org.apache.kafka.connect.data.Schema;
import org.apache.kafka.connect.data.Struct;

Quick Reference Tables

Producer Configuration Presets

Consumer Configuration Presets

Streams Configuration Presets

Common Patterns

Exactly-Once Semantics

// Producer with transactions
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "my-transactional-id");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);

Producer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

producer.beginTransaction();
producer.send(record);
producer.commitTransaction();

Consumer Groups

// Multiple consumers coordinate through consumer groups
props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");

Consumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("my-topic"));

Stateful Stream Processing

// Kafka Streams with state stores
StreamsBuilder builder = new StreamsBuilder();
KTable<String, Long> counts = builder.stream("input")
    .groupByKey()
    .count(Materialized.as("counts-store"));

Documentation Structure

Guides

Step-by-step instructions for common workflows:

• Quick Start - Get started in minutes

Examples

Real-world scenarios and use cases:

• Real-World Scenarios - Complete application examples
• Edge Cases - Advanced scenarios and edge case handling

Reference

Detailed API specifications and configuration:

• Configuration Reference - Complete configuration guide
• Error Handling Reference - Exception handling strategies
• Complete Applications - Full end-to-end examples
• Advanced Patterns - Production-ready patterns

API Documentation

Comprehensive API references:

• Producer API - Complete Producer documentation
• Consumer API - Complete Consumer documentation
• Admin API - Complete Admin documentation
• Streams API - Complete Streams documentation
• Connect API - Complete Connect documentation

Common Components

Shared functionality across APIs:

• Error Handling - 189 exceptions documented
• Security - SSL/TLS, SASL, ACLs
• Serialization - Built-in and custom serializers

Key Features

New in 4.1.1

• ShareConsumer API - Cooperative partition consumption
• Share Groups - Support in Admin API
• Enhanced Transactions - Improved exactly-once semantics

Core Capabilities

• High Throughput - Millions of messages per second
• Low Latency - Sub-millisecond latencies
• Durability - Configurable replication and persistence
• Scalability - Horizontal scaling with partitions
• Exactly-Once - End-to-end exactly-once semantics
• Stream Processing - Built-in stream processing library

Troubleshooting

Common issues and solutions:

• Producer: Buffer exhaustion, timeouts, record size limits → See clients/producer.md#troubleshooting
• Consumer: Rebalancing, lag, offset issues → See clients/consumer.md#troubleshooting
• Streams: State store errors, processing lag → See streams/overview.md#troubleshooting
• All APIs: See common/errors.md for complete exception reference

Performance Optimization

Quick optimization guides:

• Producer: See clients/producer.md#performance-tuning-guide
• Consumer: See clients/consumer.md#performance-tuning
• Streams: See streams/overview.md#performance-optimization

Security

Kafka provides comprehensive security features:

• SSL/TLS - Encryption and certificate-based authentication
• SASL - PLAIN, SCRAM, GSSAPI (Kerberos), OAUTHBEARER
• ACLs - Fine-grained authorization

See common/security.md for complete security documentation.

Version Information

• Version: 4.1.1
• Scala Version: 2.13
• Java Requirement: 8+
• License: Apache-2.0

Migration Notes:

• ShareConsumer requires broker version 4.0+
• Exactly-once-v2 recommended over exactly-once (deprecated)
• New consumer group protocol available but not default

Additional Resources

• Complete Documentation: See the reference documentation for comprehensive API details