Skip to content
This repository has been archived by the owner on Jul 12, 2022. It is now read-only.

Listens to a queue of documents from a stream, batches them, and sends them down stream

Notifications You must be signed in to change notification settings

NYPL-discovery/discovery-queue-manager

Repository files navigation

Discovery Queue Manager (DEPRECATED)

Listens to a queue of documents from a stream, batches them, and sends them down-stream.

Streaming contract

Expects data like this from stream IndexDocumentQueue (using this avro schema):

<avro encoded> {"uri":"doc1","type":"test"}
<avro encoded> {"uri":"doc2","type":"test"}
<avro encoded> {"uri":"doc2","type":"test"}
<avro encoded> {"uri":"doc3","type":"test"}
<avro encoded> {"uri":"doc3","type":"test"}
<avro encoded> {"uri":"doc3","type":"test"}

Unique document URIs will be passed along down-stream to stream IndexDocument and looks like this (using this avro schema):

<avro encoded> {"uri":"doc1","type":"test"}
<avro encoded> {"uri":"doc2","type":"test"}
<avro encoded> {"uri":"doc3","type":"test"}

Installation

npm install

Setup

Copy event.sample.json data into event.json. It's encoded with avro schema in base64, but will eventually resolve to something like this:

{"uri":"doc1","type":"test"}
{"uri":"doc2","type":"test"}
{"uri":"doc2","type":"test"}
{"uri":"doc3","type":"test"}
{"uri":"doc3","type":"test"}
{"uri":"doc3","type":"test"}

Run test

node-lambda run

Should produce data like this to the output stream (using this avro schema):

<avro encoded> {"uri":"doc1","type":"test"}
<avro encoded> {"uri":"doc2","type":"test"}
<avro encoded> {"uri":"doc3","type":"test"}

Deploy

npm run deploy-[qa|production]

About

Listens to a queue of documents from a stream, batches them, and sends them down stream

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published