package names descriptions source package names package contents all options

[  stonking  ]

Links for tokenizers

Ubuntu Resources:

• Bug Reports
• Ubuntu Changelog
• Copyright File
• Debian Source Repository (Git)

Maintainer:

• Ubuntu MOTU Developers (Mail Archive)

Please consider filing a bug or asking a question via Launchpad before contacting the maintainer directly.

Original Maintainers (usually from Debian):

• Debian Deep Learning Team (Mail Archive)
• Kohei Sendai
• Mo Zhou

It should generally not be necessary for users to contact the original maintainer.

External Resources:

• Homepage [github.com]

The following binary packages are built from this source package:

python3-tokenizers

Fast State-of-the-Art Tokenizers for Research and Production

Other Packages Related to tokenizers

build-depends

build-depends-indep

• adep: debhelper-compat (= 13)

  Package not available

• adep: dh-sequence-python3

  virtual package provided by dh-python

• adep: cargo

  Rust package manager

• adep: librust-compact-str-dev

  Memory efficient string type - Rust source code

• adep: librust-daachorse-dev

  Daachorse: Double-Array Aho-Corasick - Rust source code

• adep: librust-dary-heap-dev

  D-ary heap - Rust source code

• adep: librust-derive-builder-dev

  Rust macro to automatically implement the builder pattern for arbitrary structs - Rust source code

• adep: librust-env-logger-dev

  Log implementation configured via environment variable - Rust source code

• adep: librust-esaxx-rs-dev

  Wrapping around sentencepiece's esaxxx library - Rust source code

• adep: librust-indicatif-dev

  Progress bar and cli reporting library for Rust - Rust source code

• adep: librust-itertools-dev

  Extra iterator adaptors, iterator methods, free functions, and macros - Rust source code

• adep: librust-libc-dev

  Rust bindings to libc - Rust source code

• adep: librust-macro-rules-attribute-dev

  Use declarative macros in attribute or derive position - Rust source code

• adep: librust-monostate-dev

  Type that deserializes only from one specific value - Rust source code

• adep: librust-ndarray-dev

  N-dimensional array for general elements and for numerics - Rust source code

• adep: librust-numpy-dev

  PyO3-based Rust bindings of the NumPy C-API - Rust source code

• adep: librust-onig-dev

  Bindings for the Oniguruma regex library - Rust source code

• adep: librust-pyo3-async-runtimes-dev

  PyO3 bridges from Rust runtimes to Python's Asyncio library - Rust source code

• adep: librust-pyo3-dev

  Bindings to Python interpreter - Rust source code

• adep: librust-rand-dev

  Randomness functionality, number generators - Rust source code

• adep: librust-rayon-cond-dev

  Experimental iterator wrapper that is conditionally parallel or serial - Rust source code

• adep: librust-rayon-dev

  Simple work-stealing parallelism for Rust - Rust source code

• adep: librust-regex-dev

  Regular expressions for Rust - Rust source code

• adep: librust-regex-syntax-dev

  Regular expression parser - Rust source code

• adep: librust-serde-derive-dev

  Macros 1.1 #[derive(Serialize, Deserialize)] - Rust source code

• adep: librust-serde-dev

  Generic serialization/deserialization framework - Rust source code

• adep: librust-serde-json-dev

  JSON serialization file format - Rust source code

• adep: librust-spm-precompiled-dev

  This crate aims to emulate https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer - Rust source code

• adep: librust-thiserror-1-dev

  Derive(Error) - Rust source code

• adep: librust-unicode-categories-dev

  Query Unicode category membership for chars - Rust source code

• adep: librust-unicode-normalization-alignments-dev

  Functions for normalization of Unicode strings, including Canonical and Compatible Decomposition and Recomposition, as described in Unicode Standard Annex #15 - Rust source code

• adep: pybuild-plugin-pyproject

  Debian helper tools for packaging Python libraries using PEP517

• adep: python3-all

  package depending on all supported Python 3 runtime versions

• adep: python3-filelock

  platform independent file locking module

• adep: python3-fsspec

  specification that Python filesystems should adhere to (Python 3)

• adep: python3-huggingface-hub

  Official Python client for the Huggingface Hub.

• adep: python3-maturin

  Build and publish crates with pyo3

• adep: python3-packaging

  core utilities for python3 packages

• adep: python3-requests

  elegant and simple HTTP library for Python3, built for human beings

• adep: python3-setuptools

  Python3 Distutils Enhancements

• adep: python3-tqdm

  fast, extensible progress bar for Python 3 and CLI tool

• adep: python3-typing-extensions

  Backported and Experimental Type Hints for Python

• adep: python3-yaml

  YAML parser and emitter for Python3

• adep: rustc

  Rust systems programming language

Download tokenizers

File	Size (in kB)	MD5 checksum
tokenizers_0.23.1+dfsg-2.dsc	3.1 kB	945063b3690d88bb9a0c9fdb7d72d479
tokenizers_0.23.1+dfsg.orig.tar.xz	1,155.4 kB	6baf3ed9bef2c119c2ac88ebc630abff
tokenizers_0.23.1+dfsg-2.debian.tar.xz	5.4 kB	0c9cc4f219fa416d1465e3f086f84554

Debian Package Source Repository (VCS: Git)

https://salsa.debian.org/deeplearning-team/tokenizers.git

Debian Package Source Repository (Browsable)

https://salsa.debian.org/deeplearning-team/tokenizers