Apache Kafka Implementation in .NET – A Practical Microservices Guide

Modern enterprise applications demand high throughput, fault tolerance, and real-time data processing. Traditional synchronous APIs struggle at scale. This is where Apache Kafka becomes a backbone for event-driven microservices.

In this article, we will explore Kafka implementation using .NET with real-world architecture, producer & consumer examples, and best practices.

Why Kafka for Microservices?

  • High-throughput event streaming
  • Loose coupling between services
  • Fault tolerance and durability
  • Replayable event history
  • Horizontal scalability

Kafka Architecture – Core Concepts

Kafka Components

  • Producer – Publishes events
  • Topic – Logical event stream
  • Partition – Parallelism unit
  • Broker – Kafka server
  • Consumer Group – Scalable consumers
  • Offset – Message position

Kafka Microservices Architecture

+-------------------+
|  Web / API Layer  |
+-------------------+
          |
          v
+-------------------+
| Kafka Producer    |
| (.NET Service)    |
+-------------------+
          |
          v
=================================
|        Kafka Cluster           |
|  Topic: meter-reading-events  |
|  Partitions: 0 | 1 | 2        |
=================================
          |
          v
+-------------------+
| Kafka Consumers   |
| (.NET Workers)   |
+-------------------+
          |
          v
+-------------------+
| DB / Cache / DWH  |
+-------------------+

This architecture enables asynchronous processing, better fault tolerance, and independent service scaling.

Kafka Client for .NET

The official Kafka client for .NET:

Confluent.Kafka

Install Package

dotnet add package Confluent.Kafka

Kafka Producer Implementation (.NET)

Producer Configuration

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    Acks = Acks.All,
    EnableIdempotence = true
};

Producer Code

public class KafkaProducer
{
    private readonly IProducer<string, string> _producer;

    public KafkaProducer()
    {
        var config = new ProducerConfig
        {
            BootstrapServers = "localhost:9092"
        };

        _producer = new ProducerBuilder<string, string>(config).Build();
    }

    public async Task PublishAsync<T>(string topic, string key, T message)
    {
        var json = JsonSerializer.Serialize(message);

        await _producer.ProduceAsync(topic, new Message<string, string>
        {
            Key = key,
            Value = json
        });
    }
}

Kafka Consumer Implementation (.NET)

Consumer Configuration

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "meter-reading-group",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = false
};

Consumer Code

public class KafkaConsumer
{
    public void Start()
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "meter-reading-group",
            AutoOffsetReset = AutoOffsetReset.Earliest,
            EnableAutoCommit = false
        };

        using var consumer =
            new ConsumerBuilder<string, string>(config).Build();

        consumer.Subscribe("meter-reading-events");

        while (true)
        {
            var result = consumer.Consume();

            try
            {
                var data = JsonSerializer.Deserialize<MeterReading>(
                    result.Message.Value);

                Process(data);

                consumer.Commit(result);
            }
            catch (Exception)
            {
                // retry / dead-letter handling
            }
        }
    }
}

Offset Management Strategy

Auto-commit is NOT recommended for production systems. Manual offset commits ensure safe processing. 

consumer.Commit(result);

Error Handling & Dead Letter Topic (DLT)

Main Topic: meter-reading-events
        |
        v
Consumer Processing
        |
        |-- Success --> Commit Offset
        |
        |-- Failure --> Retry --> DLT
                          |
                          v
             meter-reading-events-dlt

Kafka vs Azure Service Bus

Feature Kafka Azure Service Bus
Throughput Very High Medium
Replay Events Yes Limited
Event Streaming Yes No
Best Use Event-driven systems Commands & workflows

Kafka Best Practices in .NET

  • Use key-based partitioning
  • Enable idempotent producers
  • Manual offset commits
  • Monitor consumer lag
  • Separate topics by domain

When NOT to Use Kafka

  • Simple CRUD applications
  • Low traffic systems
  • Strong transactional consistency required

Conclusion

Kafka with .NET enables scalable, resilient, and event-driven architectures. When designed correctly, it significantly improves system performance and reliability.

However, Kafka demands discipline, monitoring, and architectural maturity.

Comments

Post a Comment

Popular posts from this blog

How I Solved a Real Entity Framework Deadlock Caused by Different Table Order in High-Transaction Systems

The Day a Deadlock Brought Down My High-Transaction System: My Real Experience with Entity Framework Every developer has one story that sticks with them forever. For me, it was the night my high-transaction system started throwing Entity Framework deadlock errors faster than I could refresh my logs. Everything was running perfectly earlier that day. But by evening, users started reporting issues, transactions were halting, and logs began filling with the dreaded message: “Transaction (Process ID xx) was deadlocked on lock resources…” I knew this wasn’t going to be a normal night. 🔍 It All Started With a Simple Deadlock Error It was late when I saw the first error. In one part of the system, customers were generating PINs. The balance was deducted successfully — but the transaction report wasn’t updating. Instead, the application threw a deadlock exception. Minutes later, more deadlocks appeared. Something was wrong. 🧩 Debugging the Issue: The Unexpected Discovery I be...
Read more

How I Built a JWT Refresh Token Middleware in ASP.NET Core

A few months ago, I was working on a project where our APIs were protected using JWT tokens . Everything was running smoothly… until we realized that our tokens were expiring too quickly . Users were getting logged out, and every client request needed re-authentication. That’s when I decided to implement a refresh token mechanism directly in middleware, and it turned out to be a neat solution. Understanding the Problem JWT tokens are stateless and self-contained , which is great for scalability, but the downside is that once they expire, the user needs a new token. Traditionally, developers handle this on the client side with a separate refresh endpoint. But I wanted a more transparent approach : Whenever a valid token was nearing expiration, the server would automatically issue a new token. This would reduce friction for the user. The new token would be added to the response headers so clients could pick it up without changing the response payload. Step 1: Creating Midd...
Read more

Azure Service Bus vs Apache Kafka in .NET Microservices

A Practical Comparison Guide

When I started building .NET microservices , everything looked clean on architecture diagrams. But as traffic increased, synchronous HTTP calls quickly became a bottleneck. That’s when asynchronous messaging in .NET microservices became unavoidable. The real question then was: Azure Service Bus vs Apache Kafka in .NET microservices , which messaging system is the right choice for scalability, performance, and real-world production workloads? Why Messaging Is Critical in Microservices Loose coupling between services Fault isolation Independent scalability Event-driven workflows Two technologies dominate in .NET microservices: Azure Service Bus and Apache Kafka . Azure Service Bus – Enterprise Messaging Azure Service Bus is ideal for business workflows where reliability, ordering, and security are non-negotiable. Azure Service Bus Architecture font-size="14" x="70" y="115">Order Service font-size="1...
Read more