Kafka Streams in Golang: Goka vs Watermill vs go-streams

Apache Kafka Streams has no Go API. Go projects commonly discussed as Kafka Streams alternatives solve different problems. Goka maintains per-key state in Kafka-backed tables that can recover after a restart. Watermill organizes event-driven handlers with Kafka as one of several transports. go-streams builds in-process pipelines from sources, transformations, and sinks. Redpanda Connect, formerly Benthos, runs configured data pipelines, usually as a separate service.

This guide compares where each tool runs, how it uses Kafka, and whether its state survives a restart. Capabilities are checked against specific releases, and a dated GitHub snapshot compares maintenance activity. If you only need to produce and consume records, see Kafka Go Clients: franz-go vs kafka-go vs Sarama.

Quick decision

Where state lives

A running total or join needs state from earlier records. If another instance must resume that work after a crash or rebalance, the state needs a recovery path. Of the four tools here, Goka is built around Kafka-backed per-key tables: processor instances keep local copies of the table state for their assigned partitions and restore that state from Kafka when ownership changes. Goka does not provide a built-in window operator.

Watermill organizes message handlers while the application owns their state. go-streams keeps its window data in process memory rather than a Kafka changelog. Redpanda Connect holds window data in memory. After a restart, unacknowledged records are consumed again rather than restoring the previous window; processing-time window boundaries may change. Its Kafka-backed cache is not a co-partitioned processor state store. Choose these tools for their messaging or integration model, rather than expecting Goka-style state recovery.

Goka, Watermill, go-streams, and Redpanda Connect compared

Capabilities checked against project documentation and source on 27 Sep 2026.

All four use a Kafka client underneath. The client and its configuration affect broker compatibility, consumer-group behavior, and partitioning. Check those settings and the record format when combining services. See the Kafka Go client comparison for franz-go vs kafka-go vs Sarama vs confluent-kafka-go.

Goka

Goka attaches a key-value table to a Kafka consumer group. A processor consumes input topics, updates local table state, and can emit further Kafka records. The table is persisted in Kafka so another instance can restore it after a rebalance or crash. A View reads a table; an Emitter writes into a stream.

Capabilities and limitations

• State and recovery: Group tables are the reason to use Goka. Persist stores the group table in a compacted topic named <group>-table, and each instance keeps its partitions' table state in local storage, LevelDB by default. Input processing is blocked until the tables are recovered, so plan disk and restore time for reassignment.
• Delivery: Goka delivers messages at least once. After a failure, a message can be processed again, so a callback that increments a counter can apply the same message twice. Make non-idempotent state updates and external side effects safe to retry, using durable deduplication state when needed.
• Joins and windows: Join reads a co-partitioned table, and Lookup reads a table that need not be co-partitioned. There is no built-in windowing; model windows as keyed table state and emitted events. You write callbacks over decoded messages rather than a Java-style KStream/KTable DSL.
• Client: Goka uses Sarama. Kafka version and consumer-group behavior follow the Sarama release you pin.
• Partitioning: Goka defaults to FNV-1a for producing records and locating keys in views. Producers writing its co-partitioned input topics must place the same key on the same partition. When integrating a producer with a different default, such as franz-go or the Java client, align the partitioners and Goka's WithHasher, WithViewHasher, and, if used, WithEmitterHasher settings. Changing the hasher does not move records already in a topic. See Keep keyed records on the same partitions.
• Maintenance: The repository snapshot shows 7 commits and 2 merged PRs in 90 days, with PR merges spread across three people. Its 30 open issues have a median age of 3.4 years.

When to choose: Use Goka when the service must keep per-key state that survives process failure and is partitioned with a consumer group. Use a Go Kafka client when each record is independent. Use Watermill when the hard part is application choreography, not Kafka tables.

Watermill

Watermill builds event-driven Go services. You publish and subscribe to message.Message values, then use routers and CQRS components to build workflows such as sagas. watermill-kafka is the Kafka adapter; it uses Sarama (Publisher, Subscriber, consumer groups). The same handlers can sit on NATS, RabbitMQ, Redis streams, or SQL.

Capabilities and limitations

• Kafka is a transport: watermill-kafka publishes and consumes through Sarama. Watermill does not create changelog topics or restore aggregates from Kafka. The Kafka adapter does not support exactly-once delivery, so handlers must tolerate redelivery.
• Partitioning: The default marshaler publishes records without a Kafka key, so Sarama assigns them to random partitions and per-key ordering is lost. Use NewWithPartitioningMarshaler with a key function, or a custom marshaler, when consumers rely on per-key order or the topic must be co-partitioned with Goka tables. With a key, Sarama's default FNV-1a partitioner matches Goka's default.
• Acks and retries: Ack marks the offset for commit. Nack redelivers the message after NackResendSleep (100 ms by default), holding back later messages in that partition. Use bounded retries with the Retry middleware and route unprocessable records to a poison queue with PoisonQueue when the partition should continue.
• Portability: Handlers can move to another Pub/Sub, but keys, consumer groups, and Schema Registry framing still need Kafka-specific configuration.
• Maintenance: Watermill has the broadest visible usage of the four in the repository snapshot, but the core repository had 2 commits and 2 merged PRs in 90 days, and one account handled 95% of recorded merges. The snapshot covers the core repository, not watermill-kafka.

When to choose: Use Watermill when you are building sagas, CQRS, or a service that may use Kafka today and another broker later. Do not choose it to replace Kafka Streams tables.

go-streams

go-streams is a pipeline DSL: a Source emits, Flow stages transform (map, filter, throttle, and tumbling, sliding, or session windows), and a Sink consumes. The Kafka example uses the Sarama-based kafka module as source and sink. Other connectors cover Pulsar, NATS, Redis, Aerospike, cloud storage, WebSocket, and files.

Capabilities and limitations

• Windows are in-process: A five-second tumbling window lives in that Go process. If the process stops, the window is gone unless you persist it yourself. That is not Goka's Kafka-backed table and not Kafka Streams changelog restore.
• Offsets and delivery: The Sarama source marks each message as consumed before passing it downstream, so Sarama's auto-commit can record the offset before the flow or sink finishes. If the process stops, records in flight or buffered in a window can be lost rather than redelivered. The Sarama sink also logs a failed SendMessage and moves on, so an output failure can lose a record even while the process keeps running.
• Kafka's role: Connector I/O through Sarama. When the sink forwards a consumed Kafka message, it copies the key and value but not the headers. go-streams does not add a Streams topology or group table.
• Maintenance: The repository snapshot shows no commits or merged PRs in 90 days, and the latest GitHub Release, v0.13.0, dates from May 2025. One account handled all recorded merges.

When to choose: Use go-streams for an in-process pipeline with Kafka at the edges when the built-in connectors' delivery behavior is acceptable, or when you provide your own source and sink with the offset and error handling you need. Use Goka when state must survive restarts. Use Watermill when you need messaging patterns beyond a linear flow.

Redpanda Connect

Redpanda Connect is a stream processor configured in YAML: inputs read from systems such as Kafka, processors transform messages with the Bloblang mapping language, and outputs write the results. It started as Benthos; Redpanda acquired the project in 2024 and renamed it Redpanda Connect. It usually runs as its own binary or container, and Go programs can embed it or add plugins through the public/service API.

Capabilities and limitations

• Kafka clients: Use the redpanda input and output, which run on franz-go. Since v4.68.0, the kafka (Sarama) and kafka_franz inputs and outputs are deprecated and will be removed in the next major version; their functionality moved into redpanda.
• Delivery: With at-least-once inputs and outputs, the normal pipeline acknowledges messages after output delivery without persisting them to disk in transit. A buffer such as system_window can weaken that guarantee by intentionally dropping expired windows under back pressure.
• Partitioning: The redpanda output uses Murmur2 for keyed records by default and has no FNV-1a option. Its key field is optional, so set it when Goka needs per-key routing. Before writing co-partitioned input topics, check that the output's partition placement matches Goka's configured hasher.
• Windows and state: The system_window buffer groups messages into tumbling or sliding windows in memory, by processing time by default. Messages in an unfinished window are not acknowledged, so after a restart they are consumed again and window boundaries may differ. Under sustained back pressure, the buffer can intentionally drop expired windows, so not every input record is guaranteed to reach an output. Caches hold key-value state; the redpanda cache stores it in a Kafka topic but rescans the partition on reads, so its documentation recommends a compacted topic and an in-memory cache in front.
• Licensing: Most components are Apache-2.0; enterprise features use the Redpanda Community License. The Bento fork continues from the codebase before the license change.
• Maintenance: The repository snapshot shows the most activity of the four: 156 commits and 153 merged PRs in 90 days, 83 GitHub Releases in 12 months, and merges spread across 15 people. Its open issue and PR backlogs are also the largest.

When to choose: Use Redpanda Connect when the job is moving and transforming data between Kafka and other systems, and you prefer configuring a pipeline service over writing processing code. Use Goka when per-key state must be co-partitioned with the input and restored from Kafka.

GitHub repository metrics

The tables below compare the four projects' GitHub activity, maintenance, and public usage using the same measurement windows and assessment rules as the Go client comparison. Snapshot: 27 Sep 2026.

Watermill figures cover the core ThreeDotsLabs/watermill repository, not the separate watermill-kafka adapter.

Project overview

Activity

Release figures count GitHub Releases, not Go module tags.

Maintenance

PR merge distribution counts non-bot mergedBy accounts, so automated merges may undercount human reviewers. Issue closure rates use issues opened 90–180 days before the snapshot, giving each issue a full 90-day window.

Public usage and interest

GitHub dependents are approximate public-repository counts. They count repositories that declare the project as a dependency, so deployments of Redpanda Connect as a standalone binary or container are not included.

Overall repository signals

• Goka has modest recent activity, with PR merges spread across three people; its open issues are old.
• Watermill has the broadest visible usage of the four, but little recent activity, and one account handled 95% of recorded PR merges in the past year.
• go-streams had no commits or merged PRs in the last 90 days, and one account handled all recorded PR merges in the past year.
• Redpanda Connect is by far the most active, with frequent releases and merges spread across 15 people; its open issue and PR backlogs are large.

These are maintenance and adoption signals, not evidence of runtime quality or performance.

Test Go stream processing with Kafma

A stream processor that passes unit tests can still write the wrong key, lose state after a restart, or stall while it restores. Those problems show up in topics and consumer-group positions, not in the processor's return values.

Kafma is a desktop Kafka UI that runs beside your Go service or pipeline. Use its Kafka console to drive input topics, inspect output and state topics, and watch the processor's consumer group.

Drive input with keyed test records

Use the producer panel to send records with the keys that should update the same table entry or fall into the same window. Loop keeps records arriving at a fixed interval, which makes window boundaries visible. Then open the output topic and check keys, headers, and payload encoding. With Schema Registry configured, Kafma decodes Confluent-framed records. Its Schema Registry UI lists subjects, versions, and their schema IDs.

Kafma's Auto partitioning does not follow Goka's default FNV-1a hasher. For topics that Goka reads as co-partitioned input, send test input through a Goka Emitter, or set the partition explicitly after computing the one Goka would choose.

Inspect state topics

For Goka, open the group's <group>-table topic. The latest record for each key is the state a new instance will restore, and a tombstone deletes that key. Compare it with the output topic after the same input. For Redpanda Connect's redpanda cache, inspect the cache topic the same way.

A Goka processor updates the table partition that matches the input partition it processed, so comparing an input topic with the group's own table does not prove keys were hashed correctly. To check co-partitioning, compare the partition of a key in an independently produced input topic with the same key in the table that Goka joins, then confirm the view or join result.

Restart and failure tests

Open Watch Group on the processor's consumer group, then restart an instance or add a second one. A Goka processor blocks input processing until its tables recover, so lag grows during restore and should fall once processing resumes. Member status and committed positions show how long recovery takes. Kafma reads this state without joining the group, triggering a rebalance, or committing offsets. The consumer group UI shows lag and partition assignments for each member as ownership moves.

To test restarts, send inputs with a unique ID in each payload, such as an event_id field. Configure the test path to emit one output per input and copy that ID into the output. Stop and restart the processor partway through, wait for the group to catch up in Watch Group, then compare input and output IDs. Duplicate IDs are consistent with at-least-once processing; missing IDs need investigation, and go-streams' early offset marking is one possible cause. Leave system_window pipelines out of this one-to-one test, because the buffer can drop expired windows.

For Watermill, send a record the handler rejects. With the default Nack loop, the committed position for that partition stops advancing and later records stay pending in Watch Group.

Replay production-shaped input

Kafma Data Clone copies a selected range of real records to a test topic, including keys, headers, and tombstones, so a processor can be tested against older schema versions and edge cases that fixtures miss. Copied records receive new offsets and are repartitioned by the target producer, so their placement may not match Goka's hasher. Use clones to test decoding and per-record handling, not Goka's co-partitioned joins. See the Data Clone guide for copy behavior.

Download Kafma and connect to the same test cluster as the application.

Frequently asked questions

Does Go have Apache Kafka Streams?

No. Kafka Streams is a Java (and Scala) library. Goka is the usual Go stand-in when you need Kafka-backed processor state. It is not an API-compatible port.

Is Watermill a Kafka Streams alternative?

No. Watermill is a messaging and application toolkit. Its Kafka module publishes and subscribes through Sarama. It does not manage Kafka Streams-style state stores or changelog recovery.

Is Redpanda Connect a Kafka Streams alternative?

Partly. It covers transformation, routing, enrichment, and in-memory windows between Kafka and other systems, and it runs as its own service. It does not provide Kafka Streams' co-partitioned state stores or changelog restore. Use Goka when per-key state must be restored from Kafka.

What happened to Benthos?

Redpanda acquired Benthos in 2024 and renamed it Redpanda Connect. Most components remain Apache-2.0, while enterprise features use the Redpanda Community License. The Bento fork continues from the codebase before the license change.

Do any of these provide exactly-once processing?

None of the four documents Kafka Streams-style exactly-once processing. Goka and Watermill's Kafka adapter deliver at least once, as does Redpanda Connect in a normal pipeline whose inputs and outputs support at-least-once delivery. go-streams' built-in Kafka connectors can lose records. For at-least-once paths, make state updates and side effects safe to retry. For go-streams, review when offsets are marked and how sink errors are handled before relying on delivery guarantees.

Does Goka support windowing?

Not as a built-in operator. Goka provides group tables, Join, and Lookup, but no timer or punctuation API. Store per-key window state in the group table and close a window when something triggers the callback: a later record for that key, a message you emit on a schedule, or a scheduled call to VisitAll through an experimental Visitor edge, which can block rebalances and shutdown. For in-memory windows without Kafka-backed state, see go-streams or Redpanda Connect's system_window buffer.

What if I need managed windows and joins outside a Go service?

Use a dedicated engine such as Flink or Spark Structured Streaming, which manages state and checkpoints for the job. In the Go ecosystem, Numaflow runs streaming jobs on Kubernetes, and Apache Beam's Go SDK runs pipelines on a runner such as Flink or Dataflow.

Is there a Kafka Streams-style DSL for Go?

tryfix/kstream and related forks aim at a Java Streams-like DSL. They are much smaller than Goka and are not in this comparison. Treat them as experimental unless you have verified maintenance and operations for your own workload.

Can I keep using a Go Kafka client with these libraries?

Yes. Goka, Watermill's Kafka adapter, and go-streams' Kafka module use Sarama. Redpanda Connect's redpanda components use franz-go; its deprecated kafka input and output still use Sarama. Other services can use franz-go, kafka-go, Sarama, or confluent-kafka-go on the same topics as long as serialization matches. Producers writing the same keyed records must also use compatible partitioners; see Keep keyed records on the same partitions.

Conclusion

For Kafka-backed aggregates and recoverable per-key state in Go, start with Goka. Choose Watermill for event-driven handlers when Kafka is one of several transports. Use go-streams for an in-process pipeline when the built-in Kafka connectors' record-loss risk is acceptable. Use Redpanda Connect for configured pipelines that move and transform data between Kafka and other systems.

Before connecting these tools through keyed topics, check partitioning: Goka defaults to FNV-1a, Redpanda Connect's redpanda output uses Murmur2, and Watermill's default Kafka marshaler sends no key unless you configure a partitioning marshaler.

If you are porting a Java Kafka Streams topology, keep it on the JVM when API compatibility matters. Otherwise, redesign windows, joins, and recovery for the Go tool you choose, then test delivery across restarts before replacing the existing pipeline.

This guide is maintained by the team behind Kafma.