Is tokenizer safe to use?

tokenizer has no known vulnerabilities on the latest version (0.3.0). DepScope rates its health at 46/100 (high risk).

What is the latest version of tokenizer?

The latest version of tokenizer is 0.3.0, released 2016-01-20.

Does tokenizer have known vulnerabilities?

No known vulnerabilities on the latest version of tokenizer.

Is tokenizer deprecated?

No, tokenizer is actively maintained. Latest release: 0.3.0 (2016-01-20).

What are alternatives to tokenizer?

Popular alternatives to tokenizer in RubyGems include: jmespath, netrc, thread_safe, crass, tzinfo.

What license does tokenizer use?

tokenizer is licensed under MIT.

tokenizer

rubygemsv0.3.0

A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.

License MITpermissive6 versions0 deps185,585 weekly dl

arbox/tokenizer

/ 100

Health

safe to use

[email protected] is safe to use (health: 46/100)

Health breakdown0 – 100

0/25

maintenance

14/20

popularity

25/25

security

6/15

maturity

1/15

community

Vulnerabilities

none known

Health History

Dependency Tree

License Audit

API access

Get this data programmatically — free, no authentication.

curl https://depscope.dev/api/check/rubygems/tokenizer