Skip to content

CLI tool for indexing local markdown files which can then be searched for using natural language queries; a duck duck go for your local files

License

Notifications You must be signed in to change notification settings

ssosik/markdown_query

Repository files navigation

MarkdownQuery: mdq

A local text file Xapian indexer and query tool. Started life as a minimal Zettelkasten-like Markdown+FrontMatter document indexer and query interface: Keep lots of small notes and then quickly find them again using Information Retrival-style natural language queries.

I would like to extend this to index other bodies of text on a local filesystem, such as man/info pages. TODO

Example

For demonstration purposes, here I indexed a bunch of articles from Wikipedia. asciicast

About

How do you do personal notes? Grepping through a folder full of notes isn't the most effective approach; using Xapian is a better way.

I originally implemented this as a Vim plugin using some Python code built around Xapian, called tika. I decided to rewrite this in Rust, and after exploring other options for Information Retrieval based search tools like Tantivy and Fuzzy-Finder, I went back to Xapian: it's the best.

Note

This is my first Rust project, I welcome any comments/PRs/issues related to improving the code here.

There's still a lot I'd like to do here, but this version should at least be usable.

Wishlist of TODOs

Installation

For now, clone this repo and run the makefile.

Usage

# Index a source directory, note the single quotes used here to prevent the
# shell from expanding the wildcard here
mdq [db dir] update '/path/to/markdown-directory'

# Run an interactive query against an index
mdq [db dir]

Note on Markdown+Frontmatter format

I would like to make this pluggable, but for now it's hardcoded to look for Markdown+Frontmatter files like this:

---
author: Steve Sosik
date: 2021-01-15T08:23:24-05:00
tags:
- vim
title: How to grep open buffers in Vim
---

Run this command

```
:bufdo vimgrepadd [search] % | copen
```

Requirements

lightly patched version of xapian-rusty, included here as a submodule.

zlib and xapian-core, which are bundled here.

To get started in a ubuntu-18.04 docker image:

apt-get update --yes
apt-get upgrade --yes
apt-get install --yes build-essential git
git clone --recurse-submodules https://github.com/ssosik/xapiary.git
cd xapiary
make

Development

Any modern standard Rust installation should probably work.

I use NixOS along with Direnv and Direnv Nix Integration

git clone --recurse-submodules [email protected]:ssosik/xapiary.git
cd xapiary
direnv allow
# Wait some time for Nix to install all of the Rust tooling
rustup install stable

make test
make run

# hack, hack, hack

About

CLI tool for indexing local markdown files which can then be searched for using natural language queries; a duck duck go for your local files

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published