Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_ConsumeKafkaTopic never terminates. #30

Open
robertwb opened this issue Jul 7, 2020 · 1 comment
Open

_ConsumeKafkaTopic never terminates. #30

robertwb opened this issue Jul 7, 2020 · 1 comment

Comments

@robertwb
Copy link

robertwb commented Jul 7, 2020

The _ConsumeKafkaTopic DoFn will read forever (unless the topic is closed), which will not allow the bundle to finish and elements to be checkpointed/processed downstream. This should be re-written as an SFD. (Note also that there's ongoing work to expose Java's KafkaIO in Python as a cross-laungage transform).

for msg in consumer:

@wzorgdrager
Copy link

Any hints on how to implement this @robertwb ? The cross-language KafkaIO transform is not working for me either, see here. I have the exact problem as you described here when using this connector.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants