Outbox Scanner is a scanner for the data streaming outbox pattern. Scanner deals with two separate tables:
- Schema Table
- Outbox Table
Any new record in the Schema table will count as a new schema and will be published by the scanner automatically. Scanner can also respond to requests for a schema by id through Kafka. The requested schema is published back on the Kafka with appropriate topic.
Any records on the Outbox table must have a corresponding schema_id in the Schema table.
This is constrained by a foreign key. Scanner scans for new events stored in the
outbox table and publishes them to Kafka and updates the records with a published_at date.
The scanner currently performs the following tasks:
- Creates necessary Schema and Outbox tables if they don't exist
- Scans the Schema table periodically and sends out new schemas
- Responds to schema requests by consumers
- Scans the outbox table for new events
- Publishes new events and marks them as published
The following functionality is planned for the scanner:
- to verify new schemas have the correct compatibility with older ones
- to verify that the messages conform to the referencing schema
- a leader/follower functionality using Kubernetes API or a locking mechanism using a database to ensure messages are only published once (for performance considerations)