Subscribe to Consumer without mentioning JetStream name

[VadimZhiltsov]

Hello,

When we use Nats API to subscribe to some events from Jetstream fist of all we have to specify which Consumer we would like to use, but current SDK APIs require to mention JetStream name in order to identify consumer, as Consumer Id is based on pair of Consumer name + JetStream name.

I find this API a bit redundant and this seems to be the only place where in the code I have to specify Jetstream name, as publish does not require JS, but only subscribe.

I wonder if some unique identifier for Consumer can be created, so we can subscribe only based on it, without mentioning JS name. This enable us to make JS as abstraction so code can be written without specifing stream.

something like:

ver consumer = nats.Consumer("CONSUMER_ID")

consumer.subscribe("subject", func() {})

instead of

js := nats.JetStream("JS_NAME")
consumer := nats.Consumer("CONSUMER_NAME",)

consumer.subscribe("subject", func() {})

Please share your thoughts :)

[piotrpio]

Firstly, I think you may be confusing some concepts - Stream and JetStream are not the same and I think when you're talking about JetStream name you actually mean stream name. JetStream is the NATS server data persistence layer, while Streams are the individual, named message stores within JetStream. Please correct me if you meant something else.

It is not possible to get a consumer using just the consumer name (consumer is always bound to a specific stream and there is no way to bypass it). Using js.Publish() works with just providing the subject, because a stream persists messages published on particular subjects - it has nothing to do with consumers.

Are you using the legacy JetStream API (in nats package) or jetstream package? I think the confusion when using legacy JetStream implementation (i.e. the one in nats package) may be caused by the fact that it exposes Subscribe() set of methods on JetStream object, hiding the underlying consumer management logic. While this is a useful abstraction, in reality getting messages from a consumer does not directly translate to core NATS subscribe.

In the new API, you have 2 ways of getting the consumer instance to start receiving messages from a stream (I'm skipping error handling in these examples):

1. First get the stream instance and from there you can get consumers.

js, _ := jetstream.New(nc)
stream, _ := js.Stream(ctx, "STREAM_NAME")
consumer, _ := stream.Consumer(ctx, "CONSUMER_NAME")
cc, _ :=  consumer.Consume(func(msg jetstream.Msg){})

2. Bypass stream API and get consumer directly (by providing stream name and consumer name):

js, _ := jetstream.New(nc)
consumer, _ := js.Consumer(ctx, "STREAM_NAME", "CONSUMER_NAME")
cc, _ :=  consumer.Consume(func(jetstream.Msg){})

[VadimZhiltsov]

@piotrpio thank you for clarifications on code examples!

As you can see from your code samples we have to specify jetstream all the times when I need to use consumer in the code.
And from DX I find more useful to build apis which can use just consumer name / consumer id to setup consumption process
So basically this

consumer, _ := js.Consumer(ctx, "CONSUMER_NAME_OR_ID")
cc, _ :=  consumer.Consume(func(jetstream.Msg){})

instead of this

js, _ := jetstream.New(nc)
consumer, _ := js.Consumer(ctx, "STREAM_NAME", "CONSUMER_NAME")
cc, _ :=  consumer.Consume(func(jetstream.Msg){})

That helps to keep jstream name as just Nats server reference and don't use it in code at all

[ripienaar]

A stream and consumer is directly related to if you have many streams each can have a consumer called “C”. Names are not unique per jetstream only per stream.

How would it know which one you mean without the stream name?

[VadimZhiltsov]

At current architecture there is no way. But that is what request is about - to create cluster wide identifiers for consumers, so consumers can be referenced in code by that id without stream name