AIJupyter Notebook★ 2,960⑂ 286 forks
datawhalechina/diy-llm
DIY-LLM: FULL LLM COURSE WITH CODE
The diy-llm project offers six step‑by‑step Jupyter notebooks covering data pre‑training, tokenization, Transformer, MoE, distributed training, scaling laws, inference and LLM alignment. Each assignment progressively increases in difficulty, enabling you to build a full‑stack LLM from scratch.
// KEY FEATURES
- Creates and trains a custom tokenizer on any text.
- Implements Transformer blocks with Mixture‑of‑Experts (MoE) layers.
- Trains models on multiple GPUs using PyTorch Distributed and DeepSpeed.
- Evaluates scaling laws, performs inference and aligns LLMs via RLHF.
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram