Paper Breakdowns

3D Gaussian Splatting for Real-Time Radiance Field Rendering

The 2023 paper that dethroned NeRFs for novel view synthesis by replacing the neural network with millions of explicit 3D splats, achieving real-time 1080p rendering.

Explaining and Harnessing Adversarial Examples

The 2014 paper by Goodfellow et al. that exposed a terrifying flaw in neural networks: adding invisible, mathematically calculated noise to an image causes the model to confidently misclassify it.

Highly Accurate Protein Structure Prediction with AlphaFold

The 2021 DeepMind paper that solved a 50-year-old grand challenge in biology by using AI to predict the 3D structure of a protein from its 1D amino acid sequence.

Are Emergent Abilities of Large Language Models a Mirage?

A powerful rebuttal arguing that the 'sudden' appearance of capabilities in LLMs is merely a statistical illusion caused by researchers choosing non-linear, discontinuous metrics.

Attention Is All You Need

The landmark 2017 paper that introduced the Transformer architecture, replacing RNNs with self-attention and launching the modern era of large language models.

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Introduced Activation-aware Weight Quantization, a method that preserves LLM performance by identifying and protecting a tiny fraction of highly salient weights based on activation magnitudes.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Introduced bidirectional pretraining by masked language modelling, setting a new standard for natural language understanding tasks.

Better & Faster Large Language Models via Multi-token Prediction

Proposed training LLMs to predict the next $N$ tokens simultaneously rather than just the single next token, improving reasoning capabilities and creating a built-in draft model for fast speculative decoding.

Billion-Scale Similarity Search with GPUs

The 2017 paper by Facebook AI Research that introduced FAISS, making massive-scale vector similarity search practical.

BitNet: Scaling 1-bit Transformers for Large Language Models

The 2023 Microsoft paper that challenged the fundamental math of neural networks, proving you can train LLMs where weights are just +1 or -1, eliminating matrix multiplication entirely.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

The 2022 Google Brain paper that unlocked complex reasoning in LLMs by simply prompting them to show their work step-by-step before answering.

Training Compute-Optimal Large Language Models

The 2022 DeepMind paper that redefined how LLMs are scaled, proving that most existing models were too large and under-trained on too little data.

Classifier-Free Diffusion Guidance

The 2022 paper that unlocked high-quality text-to-image generation by teaching diffusion models to heavily prioritize the text prompt over the unconditional image prior.

The Claude 3 Model Family: Opus, Sonnet, Haiku

The 2024 Anthropic paper detailing a family of models that pushed the frontier of AI capabilities, heavily utilizing Constitutional AI and Constitutional alignment.

Learning Transferable Visual Models From Natural Language Supervision

The 2021 OpenAI paper that aligned text and images in a shared embedding space, unlocking zero-shot classification and the modern generative image era.

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

The 2020 paper that introduced Late Interaction, bridging the speed of dual-encoders with the accuracy of cross-encoders for neural retrieval.

Concrete Problems in AI Safety

The 2016 paper that defined the modern field of AI Alignment by clearly categorizing how optimization processes can go disastrously wrong in the real world.

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

The 2021 tutorial that popularized Conformal Prediction, a mathematical framework to add rigorous uncertainty bounds (confidence intervals) to any machine learning model.

Consistency Models

The 2023 paper from OpenAI that enables high-quality diffusion generation in just a single step, bypassing the slow iterative sampling process entirely.

Constitutional AI: Harmlessness from AI Feedback

The 2022 Anthropic paper that introduced a method to align models using AI-generated feedback rather than human labels, scaling alignment securely and transparently.

Adding Conditional Control to Text-to-Image Diffusion Models

The 2023 paper that introduced a way to add precise spatial control (like edge maps or human poses) to large text-to-image diffusion models without retraining them.

A ConvNet for the 2020s

The 2022 paper that modernized the standard ResNet architecture to prove CNNs could still match or beat Vision Transformers on accuracy and scalability.

Extracting Training Data from Large Language Models

The 2020 paper that exposed a critical privacy flaw in LLMs, proving that massive generative models verbatim memorize their training data, which can be easily extracted by users.

Denoising Diffusion Implicit Models

The 2020 paper that dramatically accelerated diffusion model sampling, reducing generation time from thousands of steps to a few dozen without retraining.

Denoising Diffusion Probabilistic Models

The 2020 paper that proved diffusion models could generate high-quality images by learning to reverse a gradual noising process.

Deep Double Descent: Where Bigger Models and More Data Hurt

The 2019 paper that broke classical statistical theory, proving that making a neural network "too big" actually causes its error rate to drop again after an initial spike.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

The 2025 breakthrough that proved 'aha' moments and advanced reasoning capabilities could emerge purely from large-scale reinforcement learning, without needing millions of human-written reasoning examples.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

The 2024 paper that disrupted the AI pricing landscape by introducing Multi-Head Latent Attention (MLA), drastically reducing the memory required for the KV Cache.

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

The 2024 paper that introduced Group Relative Policy Optimization (GRPO), a highly efficient alternative to PPO that eliminates the need for a separate value model during reinforcement learning.

Dense Passage Retrieval for Open-Domain Question Answering

The 2020 paper that proved dense neural embeddings could outperform traditional lexical search (like BM25) for open-domain question answering.

Scalable Diffusion Models with Transformers

The 2022 paper that proved Transformers could replace the U-Net as the backbone for diffusion models, unlocking predictable scaling laws for image generation.

DINOv2: Learning Robust Visual Features without Supervision

The 2023 paper from Meta that produced state-of-the-art self-supervised visual features, matching supervised models without using any labels or text.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

The 2023 Stanford paper that revolutionized LLM alignment by mathematically proving you can skip the complex Reward Model and PPO phases of RLHF, aligning models directly from preference data.

Double/Debiased Machine Learning for Treatment and Structural Parameters

The 2016 economics and statistics paper that proved how to use highly flexible machine learning models to infer true causal effects, without bias.

Deep Learning with Differential Privacy

The 2016 Google paper that proved you can train deep neural networks while mathematically guaranteeing the privacy of the individuals in the training dataset.

Efficient Memory Management for Large Language Model Serving with PagedAttention

Introduced PagedAttention and the vLLM engine, applying operating system paging concepts to the KV cache to eliminate memory fragmentation and drastically increase serving throughput.

Efficient Streaming Language Models with Attention Sinks

Discovered that LLMs naturally use the first few tokens as 'attention sinks' to dump excess attention scores, enabling a simple fix to keep models generating text infinitely without crashing.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

The 2019 paper that introduced compound scaling, a principled way to scale up convolutional networks across depth, width, and resolution.

Emergent Abilities of Large Language Models

Argued that certain complex capabilities in LLMs appear suddenly and unpredictably only after the model crosses a specific scale threshold, becoming a central tenet of the AI scaling hypothesis.

Fast Inference from Transformers via Speculative Decoding

Introduced Speculative Decoding, a technique that drastically speeds up LLM inference by using a small, fast draft model to generate tokens, which a larger model then verifies in parallel.

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

The 2017 paper that introduced Fréchet Inception Distance (FID), solving the critical problem of how to mathematically measure the quality of AI-generated images.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

The 2022 Stanford paper that rewrote the attention algorithm to be hardware-aware, drastically speeding up Transformers and unlocking massive context windows.

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

An optimized iteration of FlashAttention that better partitions work across GPU thread blocks, reaching closer to the theoretical maximum speed of the hardware.

Flow Matching for Generative Modeling

The 2022 paper that generalized diffusion models into a simpler, simulation-free framework based on Continuous Normalizing Flows and vector fields.

Gemini: A Family of Highly Capable Multimodal Models

The 2023 Google DeepMind paper introducing a natively multimodal model family built from the ground up to reason seamlessly across text, images, audio, and video.

GPT-4 Technical Report

The 2023 technical report from OpenAI detailing the model that defined the frontier of AI capabilities, demonstrating human-level performance on professional benchmarks.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Introduced a highly efficient post-training quantization method that can compress 175B parameter models down to 3 or 4 bits per weight with negligible accuracy degradation.

GQA: Training Generalized Multi-Query Attention

Introduced Grouped-Query Attention, striking an optimal balance between the high quality of Multi-Head Attention and the fast inference speed of Multi-Query Attention.

Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs

The 2016 paper that introduced the Navigable Small-World graph, the reigning standard for extremely fast approximate nearest neighbor search.

Training language models to follow instructions with human feedback

The 2022 paper that detailed the RLHF methodology used to align GPT-3 into a helpful assistant, directly laying the groundwork for ChatGPT.

Language Models are Few-Shot Learners

The GPT-3 paper that formalized in-context learning, showing that massive scale allows models to learn new tasks simply from examples in the prompt.

Language Models are Unsupervised Multitask Learners

The paper introducing GPT-2, demonstrating that scaling up a language model allows it to perform various downstream tasks zero-shot without explicit fine-tuning.

High-Resolution Image Synthesis with Latent Diffusion Models

The 2021 paper that brought diffusion models to the masses by running the generative process in a compressed latent space, drastically reducing compute requirements.

Let's Verify Step by Step

A 2023 OpenAI paper that showed training reward models to evaluate every single step of a reasoning chain (Process Supervision) drastically outperforms models trained only to evaluate the final answer (Outcome Supervision).

Why Should I Trust You?: Explaining the Predictions of Any Classifier

The 2016 paper that introduced Local Interpretable Model-agnostic Explanations, a method to figure out exactly why a "black box" AI made a specific decision.

LLaMA: Open and Efficient Foundation Language Models

The 2023 paper from Meta that introduced the first highly capable, open-weights foundation model, sparking the open-source LLM revolution.

Llama 2: Open Foundation and Fine-Tuned Chat Models

The 2023 paper from Meta that introduced a commercially viable, chat-tuned foundation model, detailing the immense effort required for RLHF and safety alignment.

The Llama 3 Herd of Models

The 2024 Meta technical report detailing a massive scale-up in data and compute, proving that dense models can reach frontier-level capabilities through sheer scale and data quality.

Visual Instruction Tuning

The 2023 paper that demonstrated how to turn an open-source text LLM into a powerful multimodal model by projecting visual features into the text token space.

LoRA: Low-Rank Adaptation of Large Language Models

The 2021 Microsoft paper that made fine-tuning massive models accessible to anyone by training only a tiny fraction of the parameters.

LoRA: Low-Rank Adaptation of Large Language Models

The 2021 paper from Microsoft that introduced Low-Rank Adaptation, allowing massive models to be fine-tuned quickly and cheaply by only training a tiny fraction of the parameters.

Lost in the Middle: How Language Models Use Long Contexts

A landmark empirical study revealing that despite massive context windows, LLMs systematically fail to retrieve information hidden in the middle of long documents.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

The 2018 MIT paper that proved massive neural networks contain tiny, sparse subnetworks that can learn just as fast and achieve the exact same accuracy as the giant model.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Introduced a new Selective State Space Model architecture that achieves Transformer-level quality with linear-time inference and hardware-aware scaling.

Matryoshka Representation Learning

The 2022 paper that introduced a way to train embedding models so their output vectors can be truncated to smaller sizes without retraining.

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Introduced an elegant technique for Tensor Parallelism, allowing massive Transformer models to be split across multiple GPUs by partitioning the matrix multiplications themselves.

Mixtral of Experts

The 2024 paper from Mistral AI that brought Sparse Mixture-of-Experts (MoE) to the open-source community, enabling a 47B parameter model to run at the speed of a 14B model.

Model Cards for Model Reporting

The 2018 paper that established the industry standard for AI transparency, proposing that every machine learning model should be accompanied by a 'nutrition label' detailing its performance, limits, and biases.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

The 2020 paper that revolutionized 3D computer vision by proving a neural network could "memorize" a 3D scene and render photorealistic novel views.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

The 2023 Microsoft paper that improved upon synthetic data distillation by teaching the smaller model the *reasoning process* of the teacher model, not just the final answer.

A Theory of the Learnable

The 1984 paper that mathematically defined what it actually means for a computer to 'learn', laying the foundational theory for modern machine learning.

Textbooks Are All You Need

The 2023 Microsoft paper that proved massive parameter counts aren't necessary if the training data is of extremely high, 'textbook' quality.

QLoRA: Efficient Finetuning of Quantized LLMs

The 2023 paper that combined 4-bit quantization with LoRA, proving you could fine-tune a massive 65B parameter model on a single consumer GPU without losing performance.

QLoRA: Efficient Finetuning of Quantized LLMs

A 2023 breakthrough that combined 4-bit quantization with LoRA, making it possible to fine-tune massive 65-billion parameter models on a single consumer GPU.

ReAct: Synergizing Reasoning and Acting in Language Models

A 2022 framework that interleaves reasoning (Chain-of-Thought) with acting (using external tools like Wikipedia or APIs), forming the basis of modern LLM agents.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

The 2020 paper that introduced a general-purpose fine-tuning recipe for combining pre-trained parametric and non-parametric memory.

RoFormer: Enhanced Transformer with Rotary Position Embedding

Introduced Rotary Position Embedding (RoPE), a method that mathematically integrates absolute positional information with relative distances, becoming the standard for modern LLMs.

Scaling Laws for Neural Language Models

Demonstrated that language model loss decreases predictably as a power law with compute, dataset size, and parameter count, establishing the mathematical foundation for the LLM scaling race.

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Anthropic's 2024 paper successfully applied Sparse Autoencoders to Claude 3 Sonnet, extracting millions of high-level, interpretable concepts (like 'Golden Gate Bridge' or 'security vulnerabilities') from a state-of-the-art production model.

Score-Based Generative Modeling through Stochastic Differential Equations

The 2020 paper that unified diffusion models and score-matching models into a single, elegant framework based on continuous-time Stochastic Differential Equations (SDEs).

Segment Anything

The 2023 Meta paper that introduced a promptable foundation model for image segmentation, capable of zero-shot segmentation of any object.

Self-Consistency Improves Chain of Thought Reasoning in Language Models

A 2022 paper that improved upon Chain-of-Thought prompting by generating multiple reasoning paths and taking a majority vote on the final answer.

Self-Instruct: Aligning Language Models with Self-Generated Instructions

The 2022 paper that demonstrated how to use a large, proprietary LLM to generate synthetic training data to fine-tune smaller, open-source models.

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

The 2019 paper that adapted BERT for generating semantically meaningful sentence embeddings that can be compared using cosine similarity.

Sigmoid Loss for Language Image Pre-Training

The 2023 paper that replaced CLIP's softmax contrastive loss with a simpler sigmoid loss, allowing for massive scaling and better performance.

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

The 2018 dataset paper that defined the cross-domain text-to-SQL task and became the standard benchmark for natural language database querying.

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

The 2021 paper that brought hierarchical structure and local windows to Vision Transformers, making them practical for high-resolution tasks like object detection and segmentation.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Scaled a sparse Mixture of Experts (MoE) model to a trillion parameters, proving that massive parameter scaling is possible without a proportional increase in compute.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Introduced T5, a framework that reframes every NLP task into a text-to-text format, enabling a single model architecture to handle diverse tasks.

Toolformer: Language Models Can Teach Themselves to Use Tools

A 2023 Meta paper that taught models to use external tools by automatically generating their own training data, fine-tuning the model to natively call APIs via special text tokens.

Towards Monosemanticity: Decomposing Neural Networks with Sparse Autoencoders

A 2023 mechanistic interpretability paper by Anthropic that successfully used Sparse Autoencoders to extract understandable, human-readable concepts from the impenetrable 'black box' of neural network activations.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Introduced ALiBi (Attention with Linear Biases), a simple positional encoding scheme that allowed models to extrapolate to context lengths far longer than what they were trained on.

Training Compute-Optimal Large Language Models

The 'Chinchilla paper' that proved models should be scaled equally with training data, overturning the previous consensus that large models could be trained on relatively little data.

Training language models to follow instructions with human feedback

The 2022 paper from OpenAI that popularized RLHF (Reinforcement Learning from Human Feedback), creating models like InstructGPT and paving the way for ChatGPT.

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

A 2023 paper that framed LLM reasoning as a search algorithm over a tree of possible thoughts, allowing models to evaluate their own progress, backtrack, and look ahead.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

The 2020 paper that proved Transformer architectures could replace CNNs for computer vision by treating image patches as a sequence of words.

Neural Discrete Representation Learning

The 2017 DeepMind paper that introduced the Vector Quantized Variational Autoencoder, turning continuous images into sequences of discrete, learnable 'tokens'.

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

The 2020 Facebook AI paper that successfully applied self-supervised learning to audio, allowing speech recognition models to be trained with vastly less human-transcribed data.

Robust Speech Recognition via Weak Supervision

The 2022 OpenAI paper that achieved human-level robustness in speech recognition by training on a massive 680,000-hour dataset of noisy, weakly supervised web audio.

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Introduced the Zero Redundancy Optimizer, a memory optimization technology that partitions model states across GPUs, making it possible to train models with hundreds of billions of parameters.

Deep Residual Learning for Image Recognition

The 2015 paper that introduced residual skip connections, solving the vanishing gradient problem and making extremely deep networks trainable.

ImageNet Classification with Deep Convolutional Neural Networks

The 2012 paper (AlexNet) that proved deep convolutional neural networks trained on GPUs could crush traditional computer vision methods.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Introduced Batch Normalization, a technique that normalized activations across a mini-batch, making networks faster and dramatically more stable to train.

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Introduced Dropout, a remarkably simple yet highly effective regularization technique that prevents neural networks from overfitting by randomly disabling neurons.

Adam: A Method for Stochastic Optimization

Introduced Adam, an adaptive optimization algorithm that combined the best properties of momentum and RMSProp, becoming the default optimizer for deep learning.

Layer Normalization

Introduced Layer Normalization, which normalizes activations across the features of a single data point rather than across a batch, making it ideal for sequence models.

Sequence to Sequence Learning with Neural Networks

Introduced the Seq2Seq encoder-decoder architecture, allowing neural networks to map input sequences to output sequences of entirely different lengths.

Neural Machine Translation by Jointly Learning to Align and Translate

Introduced the Attention mechanism, allowing Sequence-to-Sequence models to dynamically 'look back' at relevant parts of the input sentence instead of relying on a single context vector.

U-Net: Convolutional Networks for Biomedical Image Segmentation

Introduced U-Net, an elegant, symmetric architecture for image segmentation that remains the foundational backbone for modern diffusion models.

Efficient Estimation of Word Representations in Vector Space

Introduced Word2Vec, a highly efficient method for learning dense word embeddings that captured semantic meaning and analogical relationships.

Random Forests

Formalized Random Forests, an ensemble learning method that combines bagging and random feature selection to build highly robust and accurate predictive models.

XGBoost: A Scalable Tree Boosting System

Introduced XGBoost, a highly scalable and regularized gradient boosting library that utterly dominated Kaggle competitions and tabular data modeling.

Generative Adversarial Nets

Introduced the GAN, an elegant framework where two neural networks—a generator and a discriminator—compete against each other to create hyper-realistic synthetic data.

Auto-Encoding Variational Bayes

Introduced the Variational Autoencoder, a generative model that learns a continuous, structured latent space through the mathematical 'reparameterization trick'.

Proximal Policy Optimization Algorithms

Introduced PPO, an incredibly stable and sample-efficient Reinforcement Learning algorithm that became the default standard, eventually powering RLHF in ChatGPT.

A Unified Approach to Interpreting Model Predictions

Introduced SHAP, a unified framework for interpreting complex machine learning models by applying game theory to calculate the exact marginal contribution of each feature.

Distilling the Knowledge in a Neural Network

Formalized Knowledge Distillation, a method for training small, fast 'student' models to mimic the nuanced behavior of massive 'teacher' models.