tok
cranv0.2.2Fast Text Tokenization. Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
License MIT + file LICENSE0 versions1 maintainers2 deps3,452 weekly dl
mlverse/tok58
/ 100
Health
safe to use
[email protected] is safe to use (health: 58/100)
Health breakdown0 – 100
25/25
maintenance
6/20
popularity
25/25
security
0/15
maturity
2/15
community
Vulnerabilities
0
none known
Health History
Dependency Tree
License Audit
API access
Get this data programmatically — free, no authentication.
curl https://depscope.dev/api/check/cran/tokFirst published · 2026-04-22 15:55:35
Last updated · 2026-04-22T13:40:02+00:00