Memento

Intro

Memento is a single-tenant, personal aggregator for information you “save” on different platforms, making that content searchable and partially backed up. Aggregation is automatic every 5 minutes.

It currently supports:

• Twitter favourites
• Instapaper articles
• Pinboard bookmarks
• GitHub starred repositories

All data gets saved in a single Postgresql table (for easy search and backup).

The application scope is small, but one of the goals is to make it extremely robust, with all bells and whistles you need for production code.

Application structure

Memento has four main components:

• a capture supervision tree, where every source of content is monitored via a feed process
• permanent storage powered by Ecto and Postgresql
• a very simple http api to serve and search stored content
• a UI (written in Elm)

Capture

The capture workflow is structured around two ideas: Memento.Capture.Feed and Memento.Capture.Handler.

The Feed (implemented as a state machine via gen_statem) represents all common steps used to get data from any source:

• Initial authentication (where needed)
• Try to fetch the latest changes
• On success, store the data, wait 5 minutes and try again
• On failure, wait 30 seconds and try again

What changes between two sources is how some specific steps are performed and this where the Handler comes in: defined as a behaviour, it’s implemented by every source in their own way, as all APIs are different.

Storage and search

All entries are stored on Postgresql. For search, we define a entries_search_index materialized view where we use Postgresql's full text search functionality. This view is updated via a database trigger (all of this is defined as database migrations, see the migrations folder for more details.

HTTP api

This is implemented via Plug, defined in Memento.API.Router.

UI

The UI is contained in the frontend folder and uses elm.mk for compilation/watch. It's served via the parent Elixir application.

Configuration

Please check config/config.exs, configuration options are documented there.

Development

You will need a series of environment variables (for authentication against the sources APIs).

• INSTAPAPER_USERNAME
• INSTAPAPER_PASSWORD
• INSTAPAPER_OAUTH_CONSUMER_KEY
• INSTAPAPER_OAUTH_CONSUMER_SECRET
• PINBOARD_API_TOKEN
• TWITTER_CONSUMER_KEY
• TWITTER_CONSUMER_SECRET

The project is setup with Nanobox. Once you have it installed, you can:

• use nanobox evar to setup the environment variables
• use nanobox run to open a shell session to the container
• call nanobox dns add local memento.local so that you can visit http://memento.local to access the UI

Once inside the container you can:

• call mix do deps.get, ecto.migrate to fetch elixir dependencies, compile and setup the db.
• call iex -S mix to open an iex session with the running project (and then open http://memento.local)
• call MIX_ENV=test mix do ecto.create, ecto.migrate, test to run tests for the first time (after that, it's only mix test )

To work on the UI, open another container session in a different terminal, cd frontend and make watch. Any change on frontend files will trigger a build, so you just need to refresh the browser to see your changes.

Deployment

Assuming you have Nanobox setup for deployment, you just need to call nanobox deploy. Note that the project is also setup for Travis.ci. Note that Travis needs the same environment variables you setup locally to function.