ソースパッケージ: tokenizers (0.23.1+dfsg-2) [universe]

tokenizers に関するリンク

Ubuntu の資源:

• バグ報告
• Ubuntu での変更履歴
• 著作権ファイル
• Debian ソースリポジトリ (Git)

メンテナ:

• Ubuntu MOTU Developers (メールアーカイブ)

Please consider filing a bug or asking a question via Launchpad before contacting the maintainer directly.

Original Maintainers (usually from Debian):

• Debian Deep Learning Team (メールアーカイブ)
• Kohei Sendai
• Mo Zhou

It should generally not be necessary for users to contact the original maintainer.

外部の資源:

• ホームページ [github.com]

python3-tokenizers

Fast State-of-the-Art Tokenizers for Research and Production

その他の tokenizers 関連パッケージ

• adep: debhelper-compat (= 13)

  パッケージは利用できません

• adep: dh-sequence-python3

  以下のパッケージによって提供される仮想パッケージです: dh-python

• adep: cargo

  Rust package manager

• adep: librust-compact-str-dev

  Memory efficient string type - Rust source code

• adep: librust-daachorse-dev

  Daachorse: Double-Array Aho-Corasick - Rust source code

• adep: librust-dary-heap-dev

  D-ary heap - Rust source code

• adep: librust-derive-builder-dev

  Rust macro to automatically implement the builder pattern for arbitrary structs - Rust source code

• adep: librust-env-logger-dev

  Log implementation configured via environment variable - Rust source code

• adep: librust-esaxx-rs-dev

  Wrapping around sentencepiece's esaxxx library - Rust source code

• adep: librust-indicatif-dev

  Progress bar and cli reporting library for Rust - Rust source code

• adep: librust-itertools-dev

  Extra iterator adaptors, iterator methods, free functions, and macros - Rust source code

• adep: librust-libc-dev

  Rust bindings to libc - Rust source code

• adep: librust-macro-rules-attribute-dev

  Use declarative macros in attribute or derive position - Rust source code

• adep: librust-monostate-dev

  Type that deserializes only from one specific value - Rust source code

• adep: librust-ndarray-dev

  N-dimensional array for general elements and for numerics - Rust source code

• adep: librust-numpy-dev

  PyO3-based Rust bindings of the NumPy C-API - Rust source code

• adep: librust-onig-dev

  Bindings for the Oniguruma regex library - Rust source code

• adep: librust-pyo3-async-runtimes-dev

  PyO3 bridges from Rust runtimes to Python's Asyncio library - Rust source code

• adep: librust-pyo3-dev

  Bindings to Python interpreter - Rust source code

• adep: librust-rand-dev

  Randomness functionality, number generators - Rust source code

• adep: librust-rayon-cond-dev

  Experimental iterator wrapper that is conditionally parallel or serial - Rust source code

• adep: librust-rayon-dev

  Simple work-stealing parallelism for Rust - Rust source code

• adep: librust-regex-dev

  Regular expressions for Rust - Rust source code

• adep: librust-regex-syntax-dev

  Regular expression parser - Rust source code

• adep: librust-serde-derive-dev

  Macros 1.1 #[derive(Serialize, Deserialize)] - Rust source code

• adep: librust-serde-dev

  Generic serialization/deserialization framework - Rust source code

• adep: librust-serde-json-dev

  JSON serialization file format - Rust source code

• adep: librust-spm-precompiled-dev

  This crate aims to emulate https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer - Rust source code

• adep: librust-thiserror-1-dev

  Derive(Error) - Rust source code

• adep: librust-unicode-categories-dev

  Query Unicode category membership for chars - Rust source code

• adep: librust-unicode-normalization-alignments-dev

  Functions for normalization of Unicode strings, including Canonical and Compatible Decomposition and Recomposition, as described in Unicode Standard Annex #15 - Rust source code

• adep: pybuild-plugin-pyproject

  Debian helper tools for packaging Python libraries using PEP517

• adep: python3-all

  package depending on all supported Python 3 runtime versions

• adep: python3-filelock

  platform independent file locking module

• adep: python3-fsspec

  specification that Python filesystems should adhere to (Python 3)

• adep: python3-huggingface-hub

  Official Python client for the Huggingface Hub.

• adep: python3-maturin

  Build and publish crates with pyo3

• adep: python3-packaging

  core utilities for python3 packages

• adep: python3-requests

  elegant and simple HTTP library for Python3, built for human beings

• adep: python3-setuptools

  Python3 Distutils Enhancements

• adep: python3-tqdm

  fast, extensible progress bar for Python 3 and CLI tool

• adep: python3-typing-extensions

  Backported and Experimental Type Hints for Python (Python3 version)

• adep: python3-yaml

  YAML parser and emitter for Python3

• adep: rustc

  Rust systems programming language

Download tokenizers

Debian パッケージソースリポジトリ (VCS: Git)

https://salsa.debian.org/deeplearning-team/tokenizers.git

Debian パッケージソースリポジトリ (ブラウザで表示可能)

https://salsa.debian.org/deeplearning-team/tokenizers