A local text file Xapian indexer and query tool. Started life as a minimal Zettelkasten-like Markdown+FrontMatter document indexer and query interface: Keep lots of small notes and then quickly find them again using Information Retrival-style natural language queries.
I would like to extend this to index other bodies of text on a local filesystem, such as man/info pages. TODO
For demonstration purposes, here I indexed a bunch of articles from Wikipedia.
How do you do personal notes? Grepping through a folder full of notes isn't the most effective approach; using Xapian is a better way.
I originally implemented this as a Vim plugin using some Python code built around Xapian, called tika. I decided to rewrite this in Rust, and after exploring other options for Information Retrieval based search tools like Tantivy and Fuzzy-Finder, I went back to Xapian: it's the best.
This is my first Rust project, I welcome any comments/PRs/issues related to improving the code here.
There's still a lot I'd like to do here, but this version should at least be usable.
- fix preview output
- use pest for query parsing
- Colorize preview
- Support platforms besides linux, mac
- Delete entries from local cache
- Query/filter on time range
- Vim Plugin
- TUI start list at the bottom instead of the top
- TUI select many
- searching/jumping-to/highlighting in preview
- pageup/down; ctrl-w
- Cache MD5 hashes of files using kv to skip indexing unchanged files
- Keep track of document access count in KV and use that as a weighting factor in query results
- Keep track of all Tags to be used for autocompletion
- cleanups, refactoring, rust-analyze, clippy and linting
- Optimize binary and compile times
- Prune dependencies: https://pingcap.com/blog/rust-compilation-model-calamity#recent-work-on-rust-compile-times
- CLI option for passing in starting query for interactive mode
- CLI option to emit JSON instead of filename
- import man/info pages and other canonical documentation for indexing and IR
- Add URL tag, support multiple?
- Support multiple Author tags
- Add tests
- Query/filter on tags (make sure this is working properly)
- statically link xapian-core
For now, clone this repo and run the makefile.
# Index a source directory, note the single quotes used here to prevent the
# shell from expanding the wildcard here
mdq [db dir] update '/path/to/markdown-directory'
# Run an interactive query against an index
mdq [db dir]
I would like to make this pluggable, but for now it's hardcoded to look for Markdown+Frontmatter files like this:
---
author: Steve Sosik
date: 2021-01-15T08:23:24-05:00
tags:
- vim
title: How to grep open buffers in Vim
---
Run this command
```
:bufdo vimgrepadd [search] % | copen
```
lightly patched version of xapian-rusty, included here as a submodule.
zlib and xapian-core, which are bundled here.
To get started in a ubuntu-18.04 docker image:
apt-get update --yes
apt-get upgrade --yes
apt-get install --yes build-essential git
git clone --recurse-submodules https://github.com/ssosik/xapiary.git
cd xapiary
make
Any modern standard Rust installation should probably work.
I use NixOS along with Direnv and Direnv Nix Integration
git clone --recurse-submodules [email protected]:ssosik/xapiary.git
cd xapiary
direnv allow
# Wait some time for Nix to install all of the Rust tooling
rustup install stable
make test
make run
# hack, hack, hack