This repository has been archived by the owner on Dec 7, 2023. It is now read-only.

Name		Name	Last commit message	Last commit date
parent directory ..
src		src
tests		tests
tokenizations		tokenizations
.gitignore		.gitignore
Cargo.toml		Cargo.toml
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml
pytest.ini		pytest.ini
setup.cfg		setup.cfg
tox.ini		tox.ini

README.md

pytokenizations

Installation

$ pip install pytokenizations

Install from source

This library uses maturin to build.

$ git clone https://github.com/tamuhey/tokenizations
$ cd python
$ pip install maturin
$ maturin build

Now wheel is built in python/target/wheels directory. You can install it with pip install *whl.

Usage

See the README.md