Feature: Support setting connector state

[lovromazgon]

Feature description

Currently the connector state can be retrieved through the API but it can't be updated. Adding an endpoint to explicitly set the state to some specific value would allow users to change the position before starting the pipeline, giving them the power to control the starting position. This mean that an operator could externalise the storage of connector state if they want or copy over state from one Conduit instance to another.

[lyuboxa]

YES PLEASE

[nickchomey]

I could also see this being useful if, for whatever reason, the source db was restored to a backup. Conduit would think the position is in "the future".

With this feature, you could roll back conduit as well.

Or am I missing how conduit's position works?

[lyuboxa]

Or am I missing how conduit's position works?

The connector positional data is the state of the connector, last known point from which to continue to stream.
If this state is lost, the connector will restart from the beginning. This will allow anyone who is operating conduit to
restart a connector on a different instance with a state from another instance, similarly to migrating or rolling back configuration.

In the case of a source database being rolled back, this will require the state of the connector at the time of the rollback. Almost as both the source and conduit will need to be backed up in lockstep.