Kafka C/C++ Clients: librdkafka vs modern-cpp-kafka

For new Kafka code in C or C++, use librdkafka directly. It is Confluent's Kafka client library, and Confluent's Python, Go, and .NET clients are built on it. librdkafka provides native C and C++ APIs. Admin operations require the C API; the C++ API does not expose them.

The C++ wrappers listed in librdkafka's README, cppkafka and modern-cpp-kafka, add a more modern C++ interface with RAII resource management, but neither is a good default for new projects. modern-cpp-kafka was archived in July 2026, and cppkafka's default branch has had no commits since January 2024.

This guide helps you choose between librdkafka's C and C++ APIs and plan a migration from cppkafka or modern-cpp-kafka. It explains which thread runs your callbacks, when offsets are committed, what happens to queued messages at shutdown, and how packaging and Schema Registry integration work.

Quick decision

Kafka C/C++ clients compared

Capabilities verified against each release's source and documentation: 13 Oct 2026

The wrappers link the compiled librdkafka library, even header-only modern-cpp-kafka, so all four options share its native dependencies, such as OpenSSL for TLS. Package builds may leave features out: vcpkg's port enables no optional features by default and Conan's recipe defaults ssl to false, so request librdkafka[ssl] or librdkafka/*:ssl=True if your cluster uses TLS.

librdkafka C API

librdkafka implements the Kafka protocol in C. Its primary interface is rdkafka.h; the native C++ API and both wrappers build on it.

Capabilities and limitations

• Coverage: Idempotent and transactional producers, the high-level consumer, and the Admin API. Admin operations can use an existing producer or consumer handle.
• Threads and callbacks: Delivery-report, error, and statistics callbacks run only when your code serves them, so a producer must call rd_kafka_poll() regularly. In a consumer, call rd_kafka_poll_set_consumer() once so that rd_kafka_consumer_poll() serves them as well. Log and partitioner callbacks are exceptions: librdkafka can call them from its own threads.
• Memory: Your code destroys the objects it owns, such as messages returned by rd_kafka_consumer_poll(), lists it creates, and the handle. Messages passed to delivery-report callbacks and partition lists passed to rebalance callbacks are owned by librdkafka and must not be destroyed by your code. Destroy owned messages before closing the consumer; outstanding references can make rd_kafka_destroy() block.
• Consumer commits: By default, auto-commit runs every 5 seconds, and a message's offset is stored as soon as rd_kafka_consumer_poll() returns it, before your code processes it. For at-least-once processing, set enable.auto.offset.store=false and call rd_kafka_offset_store_message() after successfully processing each record. The call only stores the offset locally; the next automatic commit sends it to the broker. Do not store an offset past an earlier record in the same partition that has not been successfully processed.
• Consumer groups: KIP-848 has been generally available since 2.12.0; enable it with group.protocol=consumer on Kafka 4.0+. Release 2.16.0 still defaults to the classic protocol. If you register a rebalance callback, use rd_kafka_incremental_assign() and rd_kafka_incremental_unassign() under KIP-848 or cooperative-sticky.
• Default partitioner: By default, consistent_random hashes non-empty keys with CRC32. Set partitioner=murmur2_random to match the Java producer's default keyed partitioning.
• Share consumers: A preview share consumer (KIP-932) lets several consumers read from the same partitions, with per-record acknowledgement. It is available only in the C API, needs Kafka 4.2+ with share groups enabled, and is not recommended for production. A share consumer handle must not be used from several threads at once.

When to choose: Choose the C API for C projects, for Admin operations from C++, or as the base of your own RAII wrapper. It is also the only interface for evaluating the preview share consumer.

librdkafka C++ API

The native C++ API in rdkafkacpp.h ships as librdkafka++ and is released alongside librdkafka. Its public interface remains C++03-compatible.

Capabilities and limitations

• Coverage: Producer with idempotence and transactions, KafkaConsumer, metadata, and partition offset queries. There is no Admin API and no share consumer.
• Style: Factory functions such as KafkaConsumer::create() return raw pointers, and callbacks are interfaces you subclass, such as DeliveryReportCb and RebalanceCb. Operation failures are reported through ErrorCode values or RdKafka::Error objects. In C++11 and later code, hold the objects you own in std::unique_ptr, including messages returned by consume(). Destroy consumed messages before calling the consumer's close(), then destroy the consumer.
• Threads and callbacks: Queued callbacks follow the C API's polling model. Producers serve them through poll() and flush(), while KafkaConsumer uses consume(). Consumer close() can also invoke rebalance and offset-commit callbacks.
• Consumer commits: The C API's defaults and rules apply; with enable.auto.offset.store=false, store offsets with Message::offset_store() after successful processing.
• Consumer groups: KIP-848 and cooperative-sticky work as in the C API. If you register a RebalanceCb, check rebalance_protocol(). Use incremental_assign() and incremental_unassign() for COOPERATIVE; use assign() and unassign() for eager protocols.
• Reaching the C API: Handle::c_ptr() exposes the underlying C handle, but calling C APIs through it is not officially supported. For Admin operations, create a separate C API handle.

When to choose: Choose the C++ API for C++ services that produce and consume Kafka records.

cppkafka

cppkafka is a C++11 wrapper over librdkafka's C API. The repository is not archived, but its default branch has had no commits since January 2024.

Capabilities and limitations

• What it adds: RAII ownership of handles and messages, std::chrono timeouts, and a MessageBuilder for producing. Failed synchronous API calls throw exceptions, while consumer errors arrive inside Message. It needs Boost for boost::optional.
• Callbacks: Delivery reports run in the thread that calls poll() or flush(), as in librdkafka. Assignment and revocation callbacks run inside poll().
• Consumer commits: By default, librdkafka stores offsets when poll() returns a message and commits them every 5 seconds. To keep automatic commits, set enable.auto.offset.store=false and call store_offset(msg) after successfully processing each record. To commit manually, set enable.auto.commit=false and call commit(msg) or async_commit(msg) after successful processing. In both cases, do not store or commit an offset past an earlier record in the same partition that has not been successfully processed. store_offset() calls librdkafka's deprecated rd_kafka_offset_store(), which lacks partition leader epoch support.
• Consumer groups: The rebalance handler always calls the eager rd_kafka_assign(), and librdkafka rejects that call under the cooperative protocol. As a result, partition.assignment.strategy=cooperative-sticky and group.protocol=consumer do not work. librdkafka's configuration reference says the default group.protocol will change to consumer in a future release, so set group.protocol=classic explicitly.
• Consumer shutdown: The Consumer destructor clears your assignment and revocation callbacks before closing, so revocation logic does not run during destructor-driven shutdown. If that logic commits offsets or flushes state, do it explicitly before the consumer goes out of scope.
• Producer shutdown: The Producer destructor does not flush, so queued messages are dropped. flush() uses the handle timeout, which defaults to one second, and throws if it times out. Pass an explicit shutdown timeout and handle failure.
• Coverage: The wrapper exposes no transaction methods or Admin API.

When it is still acceptable: Existing cppkafka code can keep running if it uses the classic protocol with eager assignment and needs neither transactions nor Admin operations. Test each librdkafka upgrade, and move before you need cooperative rebalancing or KIP-848.

modern-cpp-kafka

modern-cpp-kafka is a header-only C++ wrapper from Morgan Stanley, built on librdkafka's C API. It adds RAII lifetime management and class and method names modeled on the Java client. The repository was archived in July 2026.

Capabilities and limitations

• Callbacks: During normal operation, the producer and the consumer each start a background thread that runs delivery and offset-commit callbacks by default; during close(), the closing thread can run the remaining ones. Rebalance callbacks run in the thread that calls poll(), subscribe(), unsubscribe(), or close(). With enable.manual.events.poll=true, you run delivery and commit callbacks yourself through pollEvents(), and send() reports an error instead of blocking when the queue is full.
• Producer buffers: send() does not copy the record value by default. Keep its memory valid until the delivery callback finishes, or pass SendOption::ToCopyRecordValue.
• Consumer commits: The wrapper disables librdkafka's auto-commit and ignores enable.auto.offset.store and auto.commit.interval.ms. With enable.auto.commit=true, the default, each poll() asynchronously commits the previous batch's offsets before fetching new records, and close() attempts a synchronous commit of them. This is safe only if every record returned by the previous poll() has been successfully processed before the next poll() or close(). Otherwise, set enable.auto.commit=false and call commitSync(record) or commitAsync(record, callback) after successful processing, without committing past an earlier record in the same partition that has not been successfully processed.
• Group ID: Set a stable group.id explicitly; otherwise, each consumer gets a random group ID and does not resume from the previous instance's committed offsets.
• Subscribing: subscribe() blocks until partitions are assigned and throws when its assignment wait timeout, 30 seconds by default, expires.
• Producer shutdown: close() flushes without an overall timeout by default. Pass an explicit timeout; if it expires, the producer purges the remaining messages.
• Coverage: Idempotent and transactional producers. AdminClient can create, delete, and list topics and delete records. KIP-848 support is not documented.

Risks for existing users: Wrapper fixes now require adopting or maintaining a fork. The README documents compatibility with librdkafka 2.4.0 but not with newer releases, so verify each librdkafka upgrade. A migration also changes callback threads, commit timing, and shutdown behavior; see Moving off cppkafka or modern-cpp-kafka.

Moving off cppkafka or modern-cpp-kafka

Both wrappers use librdkafka. Choose the native API you will migrate to, then review the behavior changes below.

• C++ API: Closest to the wrappers' object model, and enough for producing, consuming, and transactions. Manage owned objects, for example with std::unique_ptr, and replace exception handling with checks of the returned errors.
• C API: Required for Admin operations, including the calls made through modern-cpp-kafka's AdminClient.
• Both: The C++ API for producing and consuming, plus a separate C handle for Admin calls. librdkafka recommends a producer handle for Admin-only use, because it needs no group.id.

Then plan the cutover:

• Group ID: Reuse the old group ID so the new consumers resume from its committed offsets, as long as they are still retained. modern-cpp-kafka consumers without group.id used a random ID: read it with getGroupId() while the old instance runs, and set it explicitly in the new client. If several instances ran that way, each had its own group and read every partition; a shared group.id splits the partitions among them instead. If you start a new group, set auto.offset.reset deliberately: it applies only when there is no valid committed offset, and librdkafka defaults to the latest offset.
• Final commits: Finish processing, commit the completed offsets and check the result, then close the old consumers before starting the new ones. If some polled records are unfinished, make sure automatic commits cannot advance past them during shutdown.
• Commit paths: cppkafka's store_offset() and the native store functions only save offsets locally for the next automatic commit; modern-cpp-kafka's commit methods send the commit directly. On every path, do not store or commit an offset past an earlier record in the same partition that has not been successfully processed.
• Rebalance callbacks: If you register one, check the protocol as librdkafka's examples do. For COOPERATIVE, use incremental_assign() and incremental_unassign() in C++, or the rd_kafka_incremental_* functions in C. For eager protocols, use assign() and unassign() in C++, or rd_kafka_assign() in C. The same callback then keeps working when you switch to KIP-848.
• Order of changes: Upgrade librdkafka, switch APIs, and change the group protocol in separate steps. When moving from cppkafka, keep group.protocol=classic explicitly during the migration and enable KIP-848 later if needed, so you can tell which change caused a difference in behavior.

Schema Registry in C and C++

librdkafka sends and receives bytes; it has no built-in Schema Registry client, serializers, or deserializers. For new code, Confluent recommends libschemaregistry, its C++ Schema Registry client:

• Formats: Serializers and deserializers for Avro, Protobuf, and JSON Schema.
• Independent of the Kafka client: It does not depend on librdkafka. Serialize to a byte buffer and produce it with either librdkafka API; deserialize after consuming. Configure the Schema Registry URL and authentication separately from the Kafka client. librdkafka includes an example, producer_schema_registry.cpp.
• Build: It requires a C++17 compiler; see its README for the build toolchain. Pin the version you use and test upgrades.

According to Confluent's documentation, libserdes, the legacy C library, is discontinued, and libschemaregistry offers only a C++ API, so a C project needs a bridge to use it. One option is a small C++ module that exposes extern "C" serialize and deserialize functions to the rest of the service. Alternatively, build your own integration using C codec libraries and the registry's REST API. You must also implement schema caching and encoding compatible with Confluent's serializers.

Either way, test round trips with your existing producers and consumers: decode the records they write, and check that they decode yours. In Confluent's schema-ID payload format, the prefix is a zero byte and a four-byte big-endian schema ID; Protobuf also adds message indexes before the payload.

GitHub repository metrics

The tables below compare the three repositories' GitHub activity, maintenance, and public interest. librdkafka's C and C++ APIs ship from the same repository, so one column covers both. Snapshot: 13 Oct 2026.

Project overview

Activity

Commit counts cover the default branch. Release figures count GitHub Releases only. Closed issues and merged PRs may have been opened before the reporting window.

Maintenance

PR merge distribution counts non-bot mergedBy accounts, with shares calculated over merges attributed to those accounts; it does not measure reviewer or maintainer headcount. Median PR merge time measures creation to merge for PRs merged in the last 90 days, including automated PRs; unmerged PRs are excluded. Issue closure figures use issues opened 90–180 days before the snapshot and count those closed within 90 days of opening; they measure closure, not first-response time.

Public interest

Overall repository signals

• librdkafka is, in this snapshot, the only one of the three with ongoing commits, releases, and merged PRs, alongside a large backlog of open issues and PRs.
• cppkafka has had no default-branch commits since January 2024 and no merged PRs in the past 90 days; the repository is not archived.
• modern-cpp-kafka is archived and read-only, so further changes to the wrapper require a fork.

Test Kafka C/C++ clients with Kafma

In C and C++, the evidence that Kafka code works usually comes from the client itself: return codes, callbacks, and logs. That evidence has blind spots. A produce() call can succeed even though the record never reaches the topic, and a consumer can commit past a record it failed to process.

Kafma is a desktop Kafka GUI client for checking from outside your process. It shows the records on your test cluster, their raw bytes, and consumer group positions, without joining the group or committing offsets.

Confirm that sent records arrived. Send a few records with unique keys from your producer. In Kafma's Kafka console, load the newest records and search for those keys. If a key is missing, check immediate errors and delivery reports, then inspect a wider range of records. For the records that arrived, compare the partition, and the offset when the report includes a valid one, and check the key, value, headers, and, in Raw, the exact bytes.

Catch commits past a failure. From Kafma's producer panel, put a record your handler will reject between two valid ones, all on one partition. In Watch Group, check whether the committed offset advances beyond the rejected record's offset, and compare it with your application logs. Then restart the consumer with the same group.id and check which records it reads again.

Download Kafma and connect it to the same test cluster as your service.

Frequently asked questions

Should I use librdkafka's C API or C++ API?

For C++ services, use the C++ API for producing, consuming, and transactions, and add a separate C API handle when you need Admin operations. Use the C API for C projects, for the preview share consumer, or as the base of your own RAII wrapper.

Are cppkafka and modern-cpp-kafka still maintained?

modern-cpp-kafka was archived on 28 Jul 2026, so its repository is read-only. cppkafka is not archived, but its default branch has had no commits since 25 Jan 2024, and the 13 Oct 2026 snapshot found no merged PRs in the past 90 days, with 13 open PRs.

Can I use librdkafka's C API from C++?

Yes. rdkafka.h wraps its declarations in extern "C", so C++ code can include it and link -lrdkafka. Manage the objects you own with std::unique_ptr and a deleter that calls the matching destroy function, such as rd_kafka_message_destroy(). For a consumer, destroy the consumed messages you own, call rd_kafka_consumer_close(), then destroy the handle. This is also the supported way to reach the Admin API from a C++ service.

Does librdkafka support KIP-848?

Yes, generally available since librdkafka 2.12.0. Set group.protocol=consumer; the brokers must run Kafka 4.0 or later. Release 2.16.0 still defaults to the classic protocol, and the configuration reference says the default will change to consumer in a future release. Under the new protocol, partition.assignment.strategy, session.timeout.ms, and heartbeat.interval.ms are not used. If you register a rebalance callback, use incremental assignment and unassignment. cppkafka's rebalance handling supports only eager assignment, and modern-cpp-kafka does not document KIP-848 support.

How do I use Schema Registry from C or C++?

Use libschemaregistry, Confluent's C++ client, for Avro, Protobuf, and JSON Schema. It does not depend on librdkafka: serialize to bytes, then produce them with either API. libserdes, the legacy C library, is discontinued, so C code needs a C++ bridge to use libschemaregistry, or a custom integration that handles registry requests, schema caching, and compatible serialization. See Schema Registry in C and C++.

How do I install librdkafka on Windows and macOS?

On macOS, use Homebrew: brew install librdkafka. On Windows, librdkafka's README points to the librdkafka.redist NuGet package. vcpkg works on both platforms, but its port enables no optional features by default, so request the ones you need, for example vcpkg install "librdkafka[ssl,zstd]".

Conclusion

librdkafka is the default choice for new Kafka code in C or C++. Choose the C API for C projects and the native C++ API for producing, consuming, and transactions from C++. Add a separate C API handle for Admin operations, and manage owned C++ objects with std::unique_ptr. For at-least-once consumers, set enable.auto.offset.store=false and store an offset only after that record and every earlier record in its partition have been processed successfully. For producers, check delivery reports and flush with an explicit timeout before exit.

Plan a migration from archived modern-cpp-kafka. Existing cppkafka services can continue within its limitations: keep group.protocol=classic explicit, and migrate before adopting cooperative assignment or KIP-848. During a migration, check callback threads, commit timing, buffer ownership, and shutdown behavior.

This guide is maintained by the team behind Kafma.