RSS for Hackers

Patterns for processing and producing RSS feeds

  

Getting Started

If you want to process an RSS feed programmatically, you have to run code to poll the feed and keep track of items already processed. This isn't hard to write, but it's often not core to your app's logic. You probably just want to run code on each new item in the feed.

// Don't care how I get feed

for (const item of feed) {
  // Just want to run code
}

Pipedream lets you treat an RSS feed as an event source. Pipedream runs the code to poll the feed, emitting new items as the feed produces them:

Then, you can process items from your event source in 3 ways:

• Subscribe to a private SSE stream, which publishes items in real time
• Trigger a Pipedream workflow for each item, running hosted Node.js code (also real time)
• Access items in batch using Pipedream's REST API To get started, create a new RSS source in the Pipedream UI. Name the source and add your feed URL:

You can also visit https://pipedream.com/sources and click the Create Source button, then choose the rss/rss.js source from the modal that appears.

If you don't have a feed handy, you can use https://rss.m.pipedream.net. This generates a new, random item once every 15 minutes.

You can also create an event source using the Pipedream CLI:

  $ pd deploy # Select the rss source, enter your URL

Once deployed, you can retrieve items emitted by the source using the pd eventscommand:

$ pd events -n 1 sample-feed  # Retrieve the newest item

$ pd events -f sample-feed  # Tail the feed in real time

Then, you can trigger a Pipedream workflow— a serverless Node.js script — on every event:

or process items in your own app using the SSE stream or REST API tied to your source.

  

Example Workflows

Workflows are linear sequences of steps — just Node.js code —hosted and run by Pipedream.

You can copy the workflows below and run them in your Pipedream account for free. When you copy these workflows, you'll be asked to add the URL to your RSS feed, and to name the feed something memorable. This creates an RSS event source and runs your workflow every time an item is added to your feed.

If you already created RSS source, and want to run a workflow on that same feed, you can click the Select Another Trigger button and choose from any event source already running in your account.

They should work as is, but you can modify them in any way. For example, you can add a step to run any Node.js code or use pre-built functions to send data to other destinations.

  

SSE and REST APIs

Pipedream provides two other interfaces for accessing events produced by sources:

• A private SSE stream specific to your source
• A REST API

This way, you can run an event source in Pipedream but access its events in your own application.

SSE

SSE — Server-sent events — defines a spec for how servers can send events directly to clients that subscribe to those events, similar to WebSockets and related server-to-client push technologies. If you listen for new items using SSE, you can run any code in your own application, in real time, as soon as those items are added to the feed.

Streams are private by default. You authenticate by passing your Pipedream API key in the Authorization header:

$ curl -H "Authorization: Bearer API_KEY" \

https://api.pipedream.com/sources/SOURCE_ID/sse

See the Pipedream SSE docs for more information on authentication, example Node code, and other details on the interface.

REST API

If you prefer to process items from an RSS feed in batch, you can fetch them using Pipedream's REST API:

$ curl -H "Authorization: Bearer API_KEY" \

https://api.pipedream.com/v1/sources/SOURCE_ID/events?limit=1

Note the ?limit=1 query string. You can vary the number of events returned (most recent first) by setting this param.

Questions or Feedback?

Please reach out with any questions or feedback. We're happy to add other RSS-specific developer resources to this list, and we'd love to hear what can be improved about event sources or the example workflows.

  
  

---

Pipedream, Inc. — San Francisco, CA

Twitter | Community | Docs | Terms | Privacy

---