Messaging System (Kafkax)
Table of Contents
- Introduction
- Project Structure
- Core Components
- Architecture Overview
- Detailed Component Analysis
- Dependency Analysis
- Performance Considerations
- Troubleshooting Guide
- Conclusion
- Appendices
Introduction
This document describes the Kafkax messaging system, a Go-based abstraction over the kafka-go client tailored for production-grade messaging. It covers client configuration, producer and consumer patterns, serialization strategies, topic and partitioning, consumer group coordination, synchronous/asynchronous messaging, batching, error handling with retries, compression, ordering guarantees, performance tuning, monitoring, dead letter queues, message replay, and integration patterns with event sourcing and CQRS.
Project Structure
Kafkax resides under bi-common/mq/kafkax and exposes a clean API surface for building producers and consumers, configuring security and batching, and integrating with OpenTelemetry for distributed tracing. Key modules include configuration, options, environment loading, dialer/transport builders, compression and partitioning balancers, producer/consumer wrappers, consumption loops with retries and DLQ, multi-topic consumers, retry policies, schema registry integration, and tracing.
Diagram sources
- [config.go]
- [option.go]
- [env.go]
- [setup.go]
- [dialer.go]
- [compression.go]
- [balancer.go]
- [producer.go]
- [consumer.go]
- [consume.go]
- [multi_consumer.go]
- [retry.go]
- [schema_registry.go]
- [tracing.go]
Section sources
Core Components
- Configuration and defaults: Centralized configuration structs for producers and consumers, with embedded YAML defaults and environment variable overrides.
- Options and builders: Functional options to configure producers/consumers programmatically and via protobuf-derived options.
- Security and transport: SASL and TLS configuration builders for both producers and consumers.
- Compression and partitioning: Named compression codecs and partitioning balancers mapped to kafka-go equivalents.
- Producers: High-level APIs for sending single or batched messages, JSON serialization helpers, and optional async mode.
- Consumers: Single and multi-topic consumers, manual/auto commit modes, batch consumption, and graceful shutdown.
- Retries and DLQ: Configurable exponential backoff retries and automatic forwarding to a dead-letter topic.
- Schema Registry: Client for Confluent-compatible Schema Registry and serializer adhering to the Confluent wire format.
- Tracing: OpenTelemetry integration for producer and consumer spans, propagating context via message headers.
Section sources
- [config.go]
- [option.go]
- [dialer.go]
- [compression.go]
- [balancer.go]
- [producer.go]
- [consumer.go]
- [consume.go]
- [multi_consumer.go]
- [retry.go]
- [schema_registry.go]
- [tracing.go]
Architecture Overview
Kafkax composes kafka-go’s Writer and Reader with additional layers for configuration, security, batching, retries, DLQ, tracing, and schema registry integration. Producers and consumers are thin wrappers around kafka-go primitives, enabling safe defaults, environment-driven configuration, and robust operational features.
Diagram sources
Detailed Component Analysis
Configuration and Environment
- Root configuration embeds default YAML and exposes nested ProducerConfig and ConsumerConfig.
- Defaults include sensible batching, timeouts, compression, and balancing strategies.
- Environment variables override defaults for brokers, credentials, and runtime parameters.
- Protobuf-to-options conversion supports dynamic configuration from external systems.
Diagram sources
Section sources
Producer Patterns
- Synchronous vs asynchronous: Async mode enables fire-and-forget delivery with lower latency but less durability guarantees.
- Batching: Tune BatchSize, BatchBytes, and BatchTimeout to balance throughput and latency.
- Compression: Choose among gzip, snappy, lz4, zstd, or none.
- Partitioning: Select balancers (round-robin, least-bytes, hash, CRC32, Murmur2).
- Security: Configure SASL and TLS via options or environment.
- Serialization: Helpers for JSON and raw bytes; headers supported for correlation and tracing.
Diagram sources
Section sources
Consumer Patterns and Group Coordination
- Consumer groups: Automatic partition assignment via GroupID; isolation levels support read_committed semantics.
- Single-partition mode: Explicit Partition selection for deterministic processing or testing.
- Manual vs auto commit: Manual commit gives stronger exactly-once semantics when combined with idempotent processing.
- Graceful shutdown: Captures OS signals and drains current work before closing.
- Multi-topic consumers: Manage multiple topics with shared or per-topic group IDs.
Diagram sources
Section sources
Batch Processing and Replay
- Batch consumption: Accumulates messages until BatchSize or BatchTimeout triggers handler invocation.
- Replay: Use SetOffset or SetOffsetAt to re-consume from a specific position or timestamp.
- Lag monitoring: Access Reader lag for operational visibility.
Diagram sources
Section sources
Error Handling, Retries, and Dead Letter Queues
- Retry policy: Exponential backoff with jitter and configurable max retries.
- DLQ: On max retries exceeded, messages are forwarded to a configured topic with original metadata in headers.
- Error callbacks: Optional OnError hook for logging or metrics.
Diagram sources
Section sources
Serialization Strategies
- JSON: Built-in helpers for structured payloads.
- Schema Registry: Confluent-compatible integration with caching and wire-format serialization.
- Raw bytes: Direct payload writing for performance-sensitive scenarios.
Diagram sources
Section sources
Monitoring, Tracing, and Observability
- Stats: Access WriterStats and ReaderStats for operational insights.
- Tracing: Producer and Consumer wrappers emit spans and propagate context via message headers.
- Health checks: Ping methods to verify connectivity to brokers.
Diagram sources
Section sources
Integration Patterns: Event Sourcing and CQRS
- Event Sourcing: Use Schema Registry serialization to evolve event schemas safely; producers publish domain events; consumers replay from offsets for projections.
- CQRS: Separate topics for commands and events; consumers subscribe to events to update read models; producers emit events upon command handling.
[No sources needed since this section provides conceptual guidance]
Dependency Analysis
Kafkax depends on kafka-go for transport and protocol handling, and optionally on OpenTelemetry for tracing and on a Schema Registry for schema evolution. Options and environment loaders decouple configuration from runtime behavior.
Diagram sources
Section sources
Performance Considerations
- Batching: Increase BatchSize and BatchBytes to improve throughput; tune BatchTimeout to balance latency.
- Compression: Enable compression for bandwidth-bound deployments; choose codec based on CPU budget.
- Partitioning: Use hash balancers for keyed workloads to ensure even distribution.
- Async mode: Enable for fire-and-forget scenarios; monitor ack failures and DLQ growth.
- TLS/SASL: Keep overhead in mind; reuse connections via shared transport.
- Consumer isolation: Use read_committed for exactly-once semantics with idempotent processing.
[No sources needed since this section provides general guidance]
Troubleshooting Guide
- No brokers configured: Creation fails early with a dedicated error.
- Authentication failures: Verify SASL mechanism and credentials; check TLS certificate paths and server name.
- Consumer not consuming: Ensure GroupID is set for consumer groups; enable WatchPartitionChanges for auto-created topics.
- DLQ not receiving messages: Confirm DLQTopic is set and DLQ writer initialization succeeds; inspect error logs.
- High retry rates: Review handler idempotency and backoff configuration; consider increasing max retries or adjusting jitter.
Section sources
Conclusion
Kafkax provides a pragmatic, production-ready abstraction over kafka-go with strong defaults, robust error handling, flexible configuration, and operational features such as retries, DLQ, tracing, and schema registry integration. It supports both simple and advanced use cases, from basic pub/sub to complex event-driven architectures.
[No sources needed since this section summarizes without analyzing specific files]
Appendices
Configuration Reference
- Producer options: Brokers, Topic, ClientID, BatchSize, BatchBytes, BatchTimeout, RequiredAcks, MaxAttempts, Async, Compression, Balancer, WriteTimeout, ReadTimeout, SASL, TLS, Logger, ErrorLogger.
- Consumer options: Brokers, Topic, GroupID, ClientID, Partition, MinBytes, MaxBytes, StartOffset, MaxWait, CommitInterval, HeartbeatInterval, SessionTimeout, RebalanceTimeout, RetentionTime, ReadLagInterval, WatchPartitionChanges, PartitionWatchInterval, IsolationLevel, SASL, TLS, Logger, ErrorLogger.
Section sources
Environment Variables
Commonly used environment variables include broker lists, client ID, SASL/TLS toggles and paths, producer/consumer topic and offsets, batching parameters, and timeouts.
Section sources