Tessl Tile for cargo/tokenizers@0.22.0

or run

npx @tessl/cli init

Version

Tile

Overview

Evals

Files

docs

core-tokenization.md decoders.md index.md models.md normalizers.md post-processors.md pre-tokenizers.md training.md utilities.md

tile.json

{
  "name": "tessl/cargo-tokenizers",
  "version": "0.22.0",
  "docs": "docs/index.md",
  "describes": "pkg:cargo/tokenizers@0.22.0",
  "summary": "High-performance tokenization library with BPE, WordPiece, WordLevel, and Unigram models for natural language processing."
}