Use AWS Glue Schema Registry in Kafma

Choose AWS Glue in a Kafka connection's Schema Registry settings to browse and manage Avro, JSON Schema, and Protobuf schemas. Kafma uses the official AWS Glue wire format to decode and produce records and can copy schemas and records between registries with Data Clone. Glue support is available on Free and Pro.

Configure AWS Glue

Open the Kafka cluster's settings, expand Schema Registry, and choose AWS Glue. Enter the AWS Region and Registry Name of an existing Glue registry.

Glue credentials are configured separately from Kafka authentication. A Kafka cluster does not need to use MSK or AWS IAM authentication to use Glue.

Optionally enter an Assume role ARN. Kafma uses the chosen credentials to assume that role, then uses the role's credentials for Glue requests.

Select Test, then save the connection. Test calls only GetRegistry and ListSchemas. A successful test confirms those two requests; it does not confirm permission to retrieve schema versions for decoding, or to register, modify, or delete schemas.

Required IAM permissions

The AWS identity used for Glue needs permission for the operations you use:

Decoding a record requires glue:GetSchemaVersion, even when Test succeeds. Writing a schema also involves reading its definition and checking the new version's status, so write actions alone are insufficient.

Scope the Glue permissions to the registry and schema resources you use. See the AWS Glue IAM action and resource reference for the resource types supported by each action.

When using an assumed role, grant the Glue permissions to that role and configure its trust policy to allow your source identity to assume it. See AWS role trust policies.

If you receive Access Denied after a successful test, check the action named in the error, the selected AWS account and Region, and the permissions of the identity or role actually making the request.

Browse schemas and versions

Open Schema Registry to list the schemas in the configured registry. Select a schema to inspect its definition, fields, versions, and compatibility, or use Diff to compare available versions.

Glue assigns each version a numeric version number and a schema version UUID. The UUID identifies the version in encoded records; it is not a Standard registry's numeric schema ID.

When a schema has no available version, Kafma shows its state:

These schemas cannot be used for Mock, Lab, or message production until a version becomes available, and a Pending schema can't take a new version until you refresh it. Refresh a Pending schema after Glue completes its check. For Failed, review the definition and compatibility before publishing again.

Register schemas and publish versions

Select Register Schema, choose Avro, JSON Schema, or Protobuf, then enter the Schema Name, definition, and compatibility setting. Select Register to submit the first version.

For an existing schema, open its details and select New Version. Edit the draft beside the selected version, review the diff, and select Publish. A new version must keep the schema's existing format.

Glue can accept a submission before its compatibility check finishes. If Kafma reports that Glue is still checking, refresh the schema before retrying so you can see whether the version became available.

Glue does not support Standard Schema Registry references. Include the required definitions in the schema instead of referencing other registry subjects.

Once a version is available, use Mock to generate a sample or Lab to validate a JSON payload. See Mock and Lab for the shared workflow.

Set schema compatibility

Compatibility is configured separately for each Glue schema; there is no global compatibility setting or inherited default. Use the edit control on the schema's Compatibility card.

The checkpoint is the version from which Glue applies compatibility checks to later versions. When you change compatibility in Kafma, it moves this checkpoint to the latest version. The change does not revalidate the entire version history. See AWS Glue UpdateSchema.

Schema registration, publishing, compatibility changes, and deletion are unavailable on a Read-Only connection.

Decode and produce messages

In the Console, keep the KEY and VALUE decoders on Automatic. Kafma reads the Glue wire header, retrieves the version by UUID, and decodes the payload with its Avro, JSON Schema, or Protobuf definition, including zlib-compressed records. Kafma reads records from your existing Glue producers, and your Glue consumers can read records produced by Kafma.

Automatic schema decoding follows the registry type configured for the Kafka connection. Confluent-format records on a Glue connection, and Glue-format records on a Standard connection, remain raw bytes instead of being decoded through the other registry format.

When producing messages, Kafma automatically binds the VALUE to an available Glue schema with the same name as the topic, matching the official serializer's default schema name. You can choose a different schema or remove this binding. KEY is not automatically bound to a Glue schema; select it manually when the key is schema-encoded.

Enter the payload as JSON, select the version and any Protobuf message type, then choose Produce. Kafma validates and encodes it in the Glue wire format. See Producing messages.

Clone schemas and records

Data Clone can copy schemas and selected messages between Glue registries, or between Glue and Standard Schema Registry. Schema-backed records are converted to the destination's schema identifiers and wire format; records without a schema are copied unchanged.

A Standard schema can't be registered in a Glue target if it uses references, or if its subject name isn't a valid Glue schema name (1–255 letters, numbers, dots, hyphens, underscores, $, or #). The copy stops for that topic during the run; pre-flight does not flag it.

Glue schemas named <topic>-value or <topic>-key map to one side of the record. A schema named exactly after the topic can serve either side, so Kafma decides from the records; see scope and range. Optional field masking applies to supported schema-backed values before they reach the target.

Delete a schema

Kafma can delete an entire Glue schema, including its versions. It does not offer deletion of an individual Glue version, and there is no Standard-style soft-delete followed by permanent deletion.

Deleting the schema does not delete Kafka topics or their messages. However, once Glue removes the versions, it can no longer resolve the UUIDs referenced by those records. When Console cannot retrieve the referenced version, it shows a decode error and keeps the raw bytes available. Check the producers and consumers that still need the schema before confirming deletion.

Next steps

• Generate mock data and validate payloads
• Decode messages in the Console
• Clone topics between clusters