Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve support for asynchronous questions #631

Closed
20 of 25 tasks
cortadocodes opened this issue Feb 22, 2024 · 0 comments · Fixed by #632
Closed
20 of 25 tasks

Improve support for asynchronous questions #631

cortadocodes opened this issue Feb 22, 2024 · 0 comments · Fixed by #632
Assignees

Comments

@cortadocodes
Copy link
Member

cortadocodes commented Feb 22, 2024

Feature request

Use Case

We'd like to be able to "fire-and-forget" a question and then retrieve its events (including results) separately later.

Current state

Questions can be asked:

  • Synchronously
  • Asynchronously (but their results and events can't be easily retrieved)

Proposed Solution

  • Adapt Service.ask to work with BigQuery subscriptions
  • Provide convenient asynchronous method or argument in Child class
  • Test sending events to BigQuery via a push subscription
  • Add a function to retrieve an async question's events with:
    • Any combination of the following:
      • Inputs
        • Remove sender_type part of filter in bigquery answer subscriptions
      • Output values and manifest
      • Output datasets
      • Log messages
      • Monitor messages
    • Output format options:
      • JSON
      • Just events
      • Events and attributes
      • Events attributes, and Pub/Sub metadata
  • Add a CLI command that does the same (not needed for MVP)
  • Add a function that replays events retrieved asynchronously
  • Decide whether bigquery tables should be specified per question (by the parent) or per child (by the child)
    • If bigquery tables are specified per question, automatically delete BigQuery subscriptions once their result event has been received and stored
  • Set up bigquery table for project
  • Update documentation including adding an info box warning that an event store (e.g. BigQuery) needs to be set up for events from asynchronous questions to be usable
@cortadocodes cortadocodes moved this to In Progress in Octue Board Mar 5, 2024
@cortadocodes cortadocodes self-assigned this Mar 5, 2024
@cortadocodes cortadocodes changed the title Add ability to ask questions asynchronously Improve support for asynchronous questions Mar 12, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Octue Board Apr 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
1 participant