LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU

Share:

Similar Tracks

Attention is all you need (Transformer) - Model explanation (including math), Inference and Training Umar Jamil

Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNorm Umar Jamil

Variational Autoencoder - Model, ELBO, loss function and maths explained easily! Umar Jamil

How DeepSeek Rewrote the Transformer [MLA] Welch Labs

How Rotary Position Embedding Supercharges Modern LLMs Jia-Bin Huang

Rotary Positional Embeddings: Combining Absolute and Relative Efficient NLP

Quantization explained with PyTorch - Post-Training Quantization, Quantization-Aware Training Umar Jamil

How diffusion models work - explanation and code! Umar Jamil

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratch Umar Jamil

How to make LLMs fast: KV Caching, Speculative Decoding, and Multi-Query Attention | Cursor Team Lex Clips

Coding a Transformer from scratch on PyTorch, with full explanation, training and inference. Umar Jamil

Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math Umar Jamil

Code Your Own Llama 4 LLM from Scratch – Full Course freeCodeCamp.org

LoRA explained (and a bit about precision and quantization) DeepFindr

Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer Umar Jamil

But what are Hamming codes? The origin of error correction 3Blue1Brown

UML use case diagrams Lucid Software

LLM inference optimization: Architecture, KV cache and Flash attention YanAITalk