Switch to single-topic-per-workspace services

[cortadocodes]

Epic

End User Goal

Make asynchronous event handling scalable and more flexible.

Overview

Switch to a single-topic-per-workspace infrastructure where all services in a workspace publish and subscribe to a single topic. Subscriptions are filtered to just the events that service requires. This opens up all kinds of ways of consuming events in a more efficient and scalable manner.

In our current specific use-case, this allows a single cloud function (the async events handler) to consume all events in a workspace and store them in BigQuery for later retrieval. Currently, we're blocked by cloud functions only being allowed one subscription while our service infrastructure necessitates a subscription per service.

Contents

• Add/update attributes in event schema #634
• Specify topic name in octue.yaml and/or get it from an environment variable (otherwise use a default)
• Raise error if topic doesn't exist at Service instantiation time
• Add new attributes to published events
• Remove backwards compatibility for event attributes
• Publish to single topic
• Subscribe to single topic using a filtered subscription
• Update tests and mocks
• Test with octue example service
• Remove internal_service_name from Child?
• Remove service_id from Runner?