Trending repositories: tokenizer

2 tracked repositories tagged with tokenizer, ordered by stars. Use the topic filters below to narrow further.

Filter by topic

2 of 2 repositories

  • rasbt/LLMs-from-scratch

    Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

    AI summary: An educational repository detailing how to build a Large Language Model entirely from scratch.

    100,788learningJupyter NotebookOther

  • marcelroed/gigatoken

    Language model tokenization at GB/s

    AI summary: Gigatoken is a high-performance tokenizer for language models capable of processing text at GB/s.

    3,933ai-mlRustMIT