[{"data":1,"prerenderedAt":490},["ShallowReactive",2],{"blog-post-\u002Fblog\u002Fcopy-kafka-topics-without-mirrormaker":3},{"id":4,"title":5,"body":6,"date":474,"description":475,"extension":476,"meta":477,"navigation":478,"ogImage":183,"path":479,"pinned":480,"seo":481,"sitemap":482,"stem":483,"tags":484,"__hash__":489},"blog\u002Fblog\u002Fcopy-kafka-topics-without-mirrormaker.md","How to Copy Kafka Topics Without MirrorMaker 2",{"type":7,"value":8,"toc":449},"minimark",[9,14,23,26,48,51,54,65,68,72,75,116,119,123,126,158,161,169,173,176,185,188,193,196,200,208,228,236,247,255,262,266,274,301,304,312,316,319,322,328,332,339,347,355,359,362,378,390,394,398,401,405,408,412,418,422,425,429,432,436,439,443,446],[10,11,13],"h2",{"id":12},"when-you-need-to-move-kafka-data","When You Need to Move Kafka Data",[15,16,17,18,22],"p",{},"For Kafka mirroring between clusters, ",[19,20,21],"strong",{},"MirrorMaker 2 (MM2)"," is often the first tool teams consider.",[15,24,25],{},"MirrorMaker 2 is designed for production scenarios such as:",[27,28,29,36,42],"ul",{},[30,31,32,35],"li",{},[19,33,34],{},"Disaster recovery:"," Keeping a standby cluster current and ready for failover.",[30,37,38,41],{},[19,39,40],{},"Multi-region replication:"," Streaming selected topics across regions or cloud providers.",[30,43,44,47],{},[19,45,46],{},"Cluster migration:"," Moving live applications from an older Kafka cluster to a new one with limited downtime.",[15,49,50],{},"These jobs need the target to stay in sync. MirrorMaker 2 keeps following new records instead of copying a range and stopping.",[15,52,53],{},"Local debugging and tests usually need a much smaller, bounded copy. You may only need to:",[27,55,56,59,62],{},[30,57,58],{},"Reproduce a production consumer crash with real payloads on your local machine.",[30,60,61],{},"Seed a local Kafka cluster with realistic topic structure, schemas, and sample messages for integration testing.",[30,63,64],{},"Copy topic structure and schemas only, then produce your own test messages locally.",[15,66,67],{},"MirrorMaker 2 can move the records, but a continuous replication flow is a poor fit for these one-off workflows.",[10,69,71],{"id":70},"why-mirrormaker-2-is-a-poor-fit-for-dev-and-test","Why MirrorMaker 2 Is a Poor Fit for Dev and Test",[15,73,74],{},"MirrorMaker 2 treats data movement as a long-running replication flow, not an ad-hoc developer task. Point it at local testing and the model gets in the way:",[27,76,77,83,89,95,101],{},[30,78,79,82],{},[19,80,81],{},"Operational overhead:"," Even a small copy requires running and configuring MirrorMaker 2 workers. A typical flow also creates supporting topics for heartbeats, checkpoints, and offset syncs.",[30,84,85,88],{},[19,86,87],{},"Unbounded streaming:"," The flow follows new records until you shut it down. It has no built-in incident time window or \"copy this offset range and finish,\" so a local Kafka cluster can keep filling for as long as the process runs.",[30,90,91,94],{},[19,92,93],{},"Schema Registry stays behind:"," MirrorMaker 2 copies Kafka records, but it does not sync Schema Registry or rewrite schema IDs. With separate registries, Avro, Protobuf, or JSON Schema consumers may fail to deserialize records unless schemas and IDs are coordinated.",[30,96,97,100],{},[19,98,99],{},"No field masking:"," Replication preserves records. There is no built-in field masking, so identifiers, emails, and tokens land on the target as they are.",[30,102,103,106,107,111,112,115],{},[19,104,105],{},"Topic names change by default:"," The default replication policy prefixes remote topics with the source cluster alias (e.g., ",[108,109,110],"code",{},"source.orders","). Local consumers must use the new names unless you configure a different replication policy, such as ",[108,113,114],{},"IdentityReplicationPolicy"," for a one-way flow.",[15,117,118],{},"For development and testing, you do not need a replication pipeline. You need a bounded, schema-aware copy that can mask sensitive fields.",[10,120,122],{"id":121},"what-a-bounded-schema-aware-copy-needs","What a Bounded, Schema-Aware Copy Needs",[15,124,125],{},"To be usable for development and testing, that copy needs to meet several requirements:",[27,127,128,134,140,146,152],{},[30,129,130,133],{},[19,131,132],{},"Explicit boundaries:"," Support a time window, an offset range, or a bounded set of recent records, then stop at the selected boundary.",[30,135,136,139],{},[19,137,138],{},"Schema coordination:"," If the clusters use separate Schema Registries, discover the schemas needed by records in the selected range, register them on the target, and update the schema IDs in the payloads so local consumers can deserialize them.",[30,141,142,145],{},[19,143,144],{},"Field masking:"," Hash, redact, or null selected fields in schema-backed message values before writing the records to a development or test environment.",[30,147,148,151],{},[19,149,150],{},"Flexible topic naming:"," Keep exact source names when local applications expect them, or add a prefix or suffix to prevent collisions when isolating copies or cloning within the same cluster.",[30,153,154,157],{},[19,155,156],{},"Read-only source:"," Reading from production must be read-only. It should not join application consumer groups, trigger partition rebalances, or commit offsets back to the source.",[15,159,160],{},"Depending on the task, you might copy the topic structure and schemas, the messages, or both.",[15,162,163,168],{},[164,165,167],"a",{"href":166},"\u002Ffeatures\u002Fkafka-data-clone","Kafma Data Clone"," is built to meet these requirements.",[10,170,172],{"id":171},"copy-kafka-topics-with-kafma-data-clone","Copy Kafka Topics with Kafma Data Clone",[15,174,175],{},"Data Clone runs in the Kafma desktop client and copies selected topic structure, schemas, and message ranges within one cluster or between clusters. It requires no Kafka Connect deployment or separately managed workers.",[15,177,178],{},[179,180],"img",{"alt":181,"height":182,"src":183,"width":184},"Kafma cloning selected Kafka topics and masked production data to a local cluster",1564,"https:\u002F\u002Fmedia.kafma.app\u002Fchangelog\u002Fv1.0.0\u002Fhomepage-data-clone.png",2740,[15,186,187],{},"Set up a bounded, schema-aware copy in five steps:",[189,190,192],"h3",{"id":191},"step-1-select-the-source-cluster-and-topics","Step 1: Select the source cluster and topics",[15,194,195],{},"Choose a source cluster and one or more topics. Kafma reads the source without joining any application consumer group or committing offsets, and it does not modify source topics, messages, or schemas.",[189,197,199],{"id":198},"step-2-choose-the-copy-scope-and-message-range","Step 2: Choose the copy scope and message range",[15,201,202,203,207],{},"Choose a ",[164,204,206],{"href":205},"\u002Fdocs\u002Fdata-clone\u002Fscope-and-range","scope"," for the selected topics:",[27,209,210,216,222],{},[30,211,212,215],{},[19,213,214],{},"Structure + Data:"," Creates missing target topics, registers required schemas, and appends records in the selected range.",[30,217,218,221],{},[19,219,220],{},"Structure only:"," Creates missing topics and registers schemas without copying messages. Enable message sampling when schemas are not registered by topic name alone.",[30,223,224,227],{},[19,225,226],{},"Data only:"," Appends records to existing target topics without creating topics or registering schemas.",[15,229,230],{},[179,231],{"alt":232,"height":233,"src":234,"width":235},"Kafma Data Clone scope selector with Structure + Data, Structure only, and Data only options",770,"https:\u002F\u002Fmedia.kafma.app\u002Fblog\u002Fcopy-kafka-topics-without-mirrormaker\u002Fscope.png",1324,[15,237,238,239,242,243,246],{},"Then choose a time window, an offset range, ",[19,240,241],{},"Latest 10k msgs \u002F partition",", or ",[19,244,245],{},"All messages",". The selected range is fixed when the clone starts. Messages produced afterward are not copied.",[15,248,249,251,252,254],{},[19,250,241],{}," applies to each partition, so a 20-partition topic can copy up to 200,000 messages. Use ",[19,253,245],{}," only for small topics.",[15,256,257,258,261],{},"With ",[19,259,260],{},"Structure + Data",", Kafma registers the schema versions referenced by messages in the selected range. When the source and target use separate Schema Registries, it updates each copied message to use the corresponding target schema ID.",[189,263,265],{"id":264},"step-3-add-field-masking-rules","Step 3: Add field masking rules",[15,267,268,269,273],{},"For schema-backed Avro, Protobuf, or JSON Schema values, add ",[164,270,272],{"href":271},"\u002Fdocs\u002Fdata-clone\u002Fmasking","masking rules"," for fields that should not reach the target unchanged:",[27,275,276,282,292],{},[30,277,278,281],{},[19,279,280],{},"Hash:"," Replaces a string with a pseudonym that remains stable within the current clone run.",[30,283,284,287,288,291],{},[19,285,286],{},"Redact:"," Replaces a string with ",[108,289,290],{},"***",".",[30,293,294,297,298,291],{},[19,295,296],{},"Drop:"," Sets a nullable field to ",[108,299,300],{},"null",[15,302,303],{},"Masking applies to message values, not keys or headers.",[15,305,306],{},[179,307],{"alt":308,"height":309,"src":310,"width":311},"Kafma Data Clone masking rule for a customer ID with Hash, Redact, and Drop options",962,"https:\u002F\u002Fmedia.kafma.app\u002Fblog\u002Fcopy-kafka-topics-without-mirrormaker\u002Fmasking.png",1638,[189,313,315],{"id":314},"step-4-choose-the-target-topic-names-and-partitions","Step 4: Choose the target, topic names, and partitions",[15,317,318],{},"Select the target cluster, then keep the source topic names or add a prefix or suffix. A prefix or suffix is required when cloning within the same cluster.",[15,320,321],{},"For new target topics, use the source partition count or reduce each topic to one partition. Existing target topics keep their current configuration and partition count.",[15,323,324,327],{},[19,325,326],{},"Match Source"," copies the partition count, not each record's original partition. Kafma repartitions keyed records by key on the target and automatically assigns records without a key.",[189,329,331],{"id":330},"step-5-review-the-pre-flight-plan-and-start-the-clone","Step 5: Review the pre-flight plan and start the clone",[15,333,334,338],{},[164,335,337],{"href":336},"\u002Fdocs\u002Fdata-clone\u002Fpre-flight","Run pre-flight"," to review the topics Kafma will create or append to, schemas it will register, approximate message counts, partition decisions, masking rules, and warnings. Nothing is written to the target until you start the clone. Pre-flight blocks the clone until all masking and Schema Registry warnings are resolved.",[15,340,341],{},[179,342],{"alt":343,"height":344,"src":345,"width":346},"Kafma Data Clone pre-flight execution plan showing topic creation, schema registration, and a validated masking rule",1100,"https:\u002F\u002Fmedia.kafma.app\u002Fblog\u002Fcopy-kafka-topics-without-mirrormaker\u002Fpreflight.png",1896,[15,348,349,350,354],{},"After the clone finishes, inspect the target topics in the ",[164,351,353],{"href":352},"\u002Fblog\u002Fkafka-viewer-debugging-workflow","Kafka viewer",". Then produce extra records if needed and run your consumer or integration tests locally. Running the same clone again appends another copy.",[10,356,358],{"id":357},"when-to-use-mirrormaker-2-or-data-clone","When to Use MirrorMaker 2 or Data Clone",[15,360,361],{},"The deciding question is whether the target must keep following new messages from the source:",[27,363,364,370],{},[30,365,366,369],{},[19,367,368],{},"Use MirrorMaker 2"," for continuous cross-cluster replication, including disaster recovery, multi-region replication, and live cluster migrations.",[30,371,372,377],{},[19,373,374,375],{},"Use ",[164,376,167],{"href":166}," for one-time development or test copies that stop at a defined range and may need target schema registration or field masking.",[15,379,380,381,385,386,291],{},"Need a bounded Kafka dataset for local testing? ",[164,382,384],{"href":383},"\u002Fdownload","Download Kafma"," or see how to ",[164,387,389],{"href":388},"\u002Fdocs\u002Fdata-clone\u002Foverview","clone Kafka topics between clusters",[10,391,393],{"id":392},"faq","FAQ",[189,395,397],{"id":396},"can-mirrormaker-2-copy-only-one-kafka-topic","Can MirrorMaker 2 copy only one Kafka topic?",[15,399,400],{},"Yes. A topic filter can select one topic, but the flow still follows new records until you shut it down. It has no built-in incident time window or \"copy this range and finish\" mode.",[189,402,404],{"id":403},"does-mirrormaker-2-copy-schema-registry-schemas","Does MirrorMaker 2 copy Schema Registry schemas?",[15,406,407],{},"No. MirrorMaker 2 copies Kafka records, while Schema Registry is a separate service. If the target has its own registry, copied Avro, Protobuf, or JSON Schema payloads still carry source schema IDs until the schemas and IDs are coordinated.",[189,409,411],{"id":410},"can-i-copy-a-kafka-topic-without-kafka-connect","Can I copy a Kafka topic without Kafka Connect?",[15,413,414,415,417],{},"Yes. ",[164,416,167],{"href":388}," runs in the desktop client and copies selected topic structure, schemas, and bounded message ranges without Kafka Connect workers.",[189,419,421],{"id":420},"can-i-copy-a-topic-within-the-same-kafka-cluster","Can I copy a topic within the same Kafka cluster?",[15,423,424],{},"Yes, when the target uses a different name. Data Clone requires a prefix or suffix so the copy cannot overlap the source. The target assigns new offsets.",[189,426,428],{"id":427},"can-kafma-mask-plain-json-messages-without-schema-registry","Can Kafma mask plain JSON messages without Schema Registry?",[15,430,431],{},"No. Data Clone masking supports Schema Registry-backed Avro, Protobuf, and JSON Schema message values. It does not mask plain JSON, message keys, or headers.",[189,433,435],{"id":434},"should-i-generate-kafka-test-data-or-copy-production-records","Should I generate Kafka test data or copy production records?",[15,437,438],{},"Generate synthetic data for planned test cases, edge cases, load tests, or environments where real data is prohibited. Copy a small production sample when synthetic messages cannot reproduce the issue. Mask supported sensitive fields before writing that sample to the target.",[189,440,442],{"id":441},"does-data-clone-affect-production-consumers-or-cluster-performance","Does Data Clone affect production consumers or cluster performance?",[15,444,445],{},"Data Clone does not join application consumer groups, commit offsets, or trigger their rebalances. It still reads from the source brokers, so a large range can add broker and network load. Use a narrow time or offset range and review the estimated message count in pre-flight.",[15,447,448],{},"This guide is maintained by the team behind Kafma.",{"title":450,"searchDepth":451,"depth":451,"links":452},"",3,[453,455,456,457,464,465],{"id":12,"depth":454,"text":13},2,{"id":70,"depth":454,"text":71},{"id":121,"depth":454,"text":122},{"id":171,"depth":454,"text":172,"children":458},[459,460,461,462,463],{"id":191,"depth":451,"text":192},{"id":198,"depth":451,"text":199},{"id":264,"depth":451,"text":265},{"id":314,"depth":451,"text":315},{"id":330,"depth":451,"text":331},{"id":357,"depth":454,"text":358},{"id":392,"depth":454,"text":393,"children":466},[467,468,469,470,471,472,473],{"id":396,"depth":451,"text":397},{"id":403,"depth":451,"text":404},{"id":410,"depth":451,"text":411},{"id":420,"depth":451,"text":421},{"id":427,"depth":451,"text":428},{"id":434,"depth":451,"text":435},{"id":441,"depth":451,"text":442},"2026-09-02","Tired of setting up MirrorMaker just to move test data? Learn how to copy topics, schemas and masked messages between clusters with Kafma.","md",{},true,"\u002Fblog\u002Fcopy-kafka-topics-without-mirrormaker",false,{"title":5,"description":475},{"loc":479},"blog\u002Fcopy-kafka-topics-without-mirrormaker",[485,486,487,488],"Kafka Mirroring","Kafka Testing","Kafka MirrorMaker","MirrorMaker 2","mJ3r4IU9km3OKL683bCGnEVMM87lDKcBk_sZZJSEvIQ",1788533765134]