Core ML
Classical ML, deep learning, training patterns, and foundations
Production Ml And Llmops
- Core ML
A/B Testing Models
How to route live traffic for model comparison. Canary deployment asks 'is the new model safe?' A/B testing models asks 'is the new model better?' The distinction matters because something can be safe — it doesn't crash, it's not slower — but still fail to move the metric you care about.
- Core ML
Cost Per Token Engineering
You built an AI app that generates poems. It goes viral! You have 100,000 users. Then you get your AWS bill, and you owe $50,000. If it costs you $0.10 to generate a poem, but you only charge users $0.05, going viral will bankrupt you.
- Core ML
LLM Application Architecture
Calling an LLM API directly is not a production system. A real LLM application wraps that call in a stack of layers — gateway, cache, router, guardrails, and observability — each one solving a problem that raw inference can't.
- Core ML
LLM Caching
LLM inference is expensive. Caching is how you make some requests free. The trick is knowing which of the three caching strategies to use, because each one saves money in a different situation — and each one has a failure mode the others don't.
- Core ML
LLM Observability
Debugging an LLM application without observability is like debugging a program without a stack trace. LLM observability gives you a structured, searchable record of exactly what happened at every layer of every request — tokens used, latency, prompt version, tool calls, and cost.
- Core ML
Model Monitoring
Deploying a model is the beginning of the work, not the end. Model monitoring watches the metrics that warn you something is going wrong — before accuracy drops, before costs spike, before users leave.
- Core ML
Model Quantization for Serving
Instead of storing every decimal number in the model with extreme 16-decimal-place precision, we chop off the ends of the decimals. The model's file size shrinks by 4x, it runs much faster, and surprisingly, it barely loses any intelligence.
- Core ML
Model Registry
You've trained a great model. How does the software engineering team actually get it to put it on the website? A Model Registry is the central 'app store' for your company's models, handling stages like Staging, Production, and Archive.
- Core ML
Model Routing
Not every question deserves a $20 answer. Model routing looks at each incoming request, decides how hard it is, and sends it to the cheapest model that can handle it — saving 40-60% on inference costs in real applications without users noticing.
- Core ML
Model Serialization
You trained a model in Python using PyTorch, but the production web server is written in C++ or Rust for maximum speed. You need a way to save the model so that any language can load it and run it.
- Core ML
Model Versioning
In standard software, version 2.0 is just new code. In ML, version 2.0 is new code, new data, and a new environment. If you don't version all three together, you can never reproduce a broken model to debug it.
- Core ML
Prompt and Model Versioning
A prompt buried in your source code is a deployment waiting to go wrong. Treating prompts as versioned artefacts — with a registry, a hash, and a rollback path — gives you the same control over your LLM's behaviour that you already have over your model weights.
- Core ML
Activation Checkpointing
Instead of saving every intermediate calculation in memory during the forward pass, we throw most of them away. When we need them for the backward pass, we just recalculate them from scratch. We trade compute for memory.
- Core ML
Batch vs Real-Time Inference
Do you need the model to give you an answer right this second, or can you just let it crunch all the data overnight while you sleep? The choice fundamentally changes how you build your infrastructure.
- Core ML
Continuous Batching
Instead of waiting for an entire batch of users to finish their long text generation before starting a new batch, the server ejects finished users instantly and injects new users mid-flight. It's like a revolving door for GPU memory.
- Core ML
Dynamic Batching
GPUs are terrible at processing one request at a time, but great at processing 50 at once. When users send requests via an API at random times, the server forces the first user to wait a few milliseconds, gathers 49 more requests from other users, and sends them to the GPU together.
- Core ML
GPU Utilization and Profiling
You rented a $30,000 GPU and ran your code. The GPU says it is at 100% usage, but your code is still slow. Profiling allows you to see exactly what the GPU is actually doing every microsecond, revealing that it's just spinning its wheels waiting for memory.
- Core ML
Autoscaling Inference
You don't want to pay for 10 GPUs at 3:00 AM when nobody is using your app. Autoscaling automatically spins up more servers when traffic spikes, and destroys them when traffic drops, saving you massive amounts of money.
- Core ML
CI/CD for Machine Learning
In software, merging code to 'main' triggers a server deployment. In ML, merging code to 'main' should trigger a training pipeline, not a deployment. The model itself requires a second, separate CI/CD loop.
- Core ML
Compilation and Kernel Fusion
Running 10 small math equations requires the GPU to look at the instructions 10 separate times. Kernel fusion mathematically combines those 10 small equations into 1 massive equation, allowing the GPU to do it all at once.
- Core ML
Concept Drift
Your fraud detection model was perfect when you trained it. Then fraudsters adapted. The same transactions that were safe last year are fraud this year, and vice versa. The inputs didn't change — the world did. That's concept drift.
- Core ML
Data Drift Detection
A model trained on last year's data meets this year's users. If the data coming in today looks different from what the model trained on, accuracy is going to drop — and data drift detection is the statistical radar that catches this shift before it becomes a crisis.
- Core ML
Data Parallelism
If you have 8 GPUs, put an identical copy of the model on every GPU. Give each GPU a different slice of the dataset. They each calculate how the model should change, average their answers together, and update simultaneously. You just trained 8x faster.
- Core ML
Distributed Training
You cannot fit a 70-billion parameter model into a single GPU. It physically doesn't fit in the memory. Distributed training is the art of breaking the model, the data, or the optimizer across hundreds of GPUs so they can train together as one massive brain.
- Core ML
Experiment Tracking
When you train 50 different models in a week, you will forget which hyperparameters produced the best one. Experiment tracking is a digital lab notebook that automatically records exactly how every model was built.
- Core ML
Expert Parallelism
When a Mixture of Experts model has too many experts to fit on one GPU, you distribute the experts across multiple GPUs. But now, every token must be sent across the network to its assigned expert.
- Core ML
Fallbacks and Retries
LLM APIs fail. Rate limits hit, servers go down, and timeouts happen. Fallbacks and retries are the engineering patterns that ensure your users always get a response even when the primary model is unavailable.
- Core ML
Fault-Tolerant Training
When you train a model on 10,000 GPUs for a month, it is mathematically guaranteed that hardware will break. Fault-Tolerant Training is how you ensure that a single broken cable doesn't waste three weeks of million-dollar compute.
- Core ML
Feature Stores
Data Scientists compute features slowly in a data warehouse (Offline). Software Engineers need to fetch those same features in 10 milliseconds for the live app (Online). A Feature Store bridges this gap, guaranteeing that the 'average_spend_30d' calculated during training is mathematically identical to the one fetched during live inference.
- Core ML
Incident Response for ML
A software bug is either broken or not. An ML incident is messier: the model is still returning responses, they're just quietly wrong for some slice of users. Incident response for ML adds the steps — rollback, replay, and blameless postmortem — that standard DevOps runbooks skip.
- Core ML
Inference Engines
You don't serve production traffic by running a Python script. You use a massive, hyper-optimized C++ server designed specifically to squeeze every last drop of performance out of a GPU.
- Core ML
KV Cache Management
Instead of forcing the LLM to re-read the entire conversation from scratch every time it wants to generate a new word, the server saves the mathematical summary of the conversation in the GPU's memory. PagedAttention manages this memory just like a computer's operating system.
- Core ML
Latency Budgets
You can't just build an ML model and hope it's fast enough. You must establish a strict time limit (a budget) that the model is allowed to take before the user gets frustrated and leaves your app.
- Core ML
Memory Offload (CPU & NVMe)
When a model is too big to fit into the ultra-fast GPU memory, you can temporarily 'park' parts of the model in the slower CPU RAM or even on the hard drive, and swap them back into the GPU exactly when they are needed for math.
- Core ML
ML Pipeline Architecture
A Jupyter Notebook is not software. An ML Pipeline turns a messy, manual research script into a repeatable, automated factory that ingests data and produces deployable models.
- Core ML
ML System Design Framework
Every ML system design interview asks the same question in a different costume. This framework gives you a six-phase scaffold — requirements, metric, data, model, serving, monitoring — that produces a defensible, complete answer to any of them.
- Core ML
Multi-Tenant Inference
Instead of renting 5 separate GPUs to run 5 separate ML models for 5 separate customers, you load all 5 models onto a single GPU and share the hardware. It's like an apartment building for neural networks.
- Core ML
On-Device Inference
Instead of paying AWS thousands of dollars to run your model in the cloud, you compress the model and send it to the user's iPhone. The iPhone's processor does the math. Your cloud hosting cost drops to exactly $0.00.
- Core ML
Online Evaluation
Offline evals tell you how your model performs on a fixed test set. Online evaluation tells you how it's performing right now, on real user queries — which are messier, weirder, and more important than any test set you'll ever write.
- Core ML
Pipeline Parallelism
If a model has 80 layers and you have 8 GPUs, you can put Layers 1-10 on GPU 1, and Layers 11-20 on GPU 2. GPU 1 does some math, hands the result to GPU 2, and so on. It's an assembly line for neural networks.
- Core ML
Rate Limits and Quotas
LLM providers set hard ceilings on how many requests and tokens you can send per minute. Hit the ceiling and your app gets a 429 error. The job of rate-limit engineering is to stay under the ceiling without slowing your users down.
- Core ML
Real-time Feature Computation
Batch features update once a night. But if a user clicks a product 5 seconds ago, the recommendation model needs to know *right now*. Real-time feature computation uses streaming architectures to calculate features instantly as events happen.
- Core ML
Reproducibility in Machine Learning
If you compile a Java app twice, you get the same binary. If you train a neural network twice with the exact same code and data, you might get two slightly different models. Reproducibility is the battle to make ML training deterministic.
- Core ML
REST API Serving
A model is just a math equation. To actually let users interact with it, you must wrap it in a standard web server so that mobile apps and websites can send it data over the internet and receive predictions back.
- Core ML
Sequence Parallelism
When your context window gets so large that the activations for a single sequence no longer fit on one GPU, you must split the sequence itself across multiple GPUs.
- Core ML
Shadow and Canary Deployments
You'd never drive a new car off the lot before test-driving it. Shadow and canary deployments are how you test-drive a new model on real traffic — one without any users seeing the results, the other with just a small fraction — before committing to the full switch.
- Core ML
Tensor Parallelism
When a single mathematical matrix is too large to fit in one GPU's memory, you have to split the math itself. Tensor Parallelism cuts a matrix in half, gives half to GPU 1 and half to GPU 2, and merges their answers at the end of the calculation.
- Core ML
Testing ML Systems
You can't write a unit test that says 'assert model.predict(image) == dog', because the model is probabilistic. Testing ML systems requires testing the data going in, checking specific behavioral invariants, and tracking aggregate metrics on golden sets.
- Core ML
Training Cost Estimation
Before you spin up 1,000 GPUs, you need to know if the bill will be $50,000 or $5,000,000. Cost estimation is a math formula that predicts the financial budget based on the model's size, the dataset's size, and the hardware's theoretical speed.
- Core ML
ZeRO & FSDP
Instead of having every GPU hold a copy of the entire model, we can slice the model like a pie. Every GPU only holds one slice, and they pass the slices around right when they are needed for the math.
Attention
- Core ML
ALiBi Attention Bias
ALiBi skips positional embeddings entirely and instead subtracts a penalty proportional to distance straight from the attention scores, before softmax runs.
- Core ML
Attention Complexity
Every query scores against every key, so both the compute and the memory attention needs grow with the square of sequence length, not with the length itself.
- Core ML
Cross-Attention
Take queries from one sequence and keys and values from a different sequence entirely, so one sequence can read another without the two ever being merged into one.
- Core ML
FlashAttention
Standard attention writes a huge score matrix to GPU RAM and reads it back for the softmax — FlashAttention tiles the computation so the score matrix never leaves the small, fast on-chip cache, making attention faster without changing the result by a single bit.
- Core ML
GPT & Autoregressive Models
GPT reads strictly left to right and trains on exactly one job: guess the next word, over and over, which turns out to teach it far more than that sounds.
- Core ML
Grouped-Query Attention
Instead of every head keeping its own keys and values or all heads sharing one pair, grouped-query attention splits heads into a handful of groups where each group shares a KV pair — most of the cache saving, almost none of the quality loss.
- Core ML
Hierarchical Vision Transformers
Hierarchical Vision Transformers attend within small local windows and merge patches across stages, rebuilding a CNN-like pyramid of image resolutions.
- Core ML
Hybrid Attention-SSM Architectures
Interleaving attention layers with SSM layers in the same model captures precise long-range recall where attention is strong and constant-time recurrence where SSM is efficient, so neither layer type has to do the job the other does better.
- Core ML
Linear Attention
By replacing the softmax with a kernel function, attention can be rewritten so keys and values are accumulated into a fixed-size running state, making the computation linear in sequence length rather than quadratic.
- Core ML
Mamba
Mamba makes state space models input-dependent: instead of using the same fixed transition for every token, it learns separate selection gates that let each token control how much it updates the state and how much of the output to reveal.
- Core ML
Multi-Head Attention
Split the model dimension into several smaller subspaces and run independent attention in each one, so different heads can specialize in different kinds of relationship.
- Core ML
Multi-Head Latent Attention
Instead of caching full key and value tensors for every head, compress them into a tiny shared latent vector and reconstruct the per-head K and V on the fly — the cache stores the latent, not the expanded tensors.
- Core ML
Multi-Query Attention
All query heads share a single set of keys and values instead of each keeping their own, which shrinks the KV cache by as many times as there are heads.
- Core ML
Multi-Token Prediction
During training a model grows several extra heads that guess tokens further ahead at once, not just the next one, then those extra heads get thrown away.
- Core ML
Patch Embeddings
Patch embeddings flatten each fixed-size image patch into a vector, then project it once into a token dimension a transformer block can actually read.
- Core ML
Ring Attention
To handle a sequence too long for one GPU, ring attention shards it across devices arranged in a ring, streaming key-value blocks around the ring while each device computes its local attention against whichever block passes through — every device sees all keys and values without storing them all.
- Core ML
Rotary Position Embeddings
Instead of adding a position vector to every token, rotary embeddings rotate a token's query and key vectors by an angle tied to its position in the sequence.
- Core ML
Self-Attention
Every token builds a query, a key, and a value, then scores its query against every key in the sequence to decide how much of each other token's value to blend in.
- Core ML
Sliding Window Attention
Each token only attends to the nearest W tokens around it instead of every token in the sequence, so attention cost scales linearly with sequence length rather than quadratically.
- Core ML
Sparse Attention
Instead of scoring every token against every other token, sparse attention computes only a structured subset of those scores — nearby tokens for local context, a few strided positions for long-range reach, and a handful of global tokens that everything can see.
- Core ML
State Space Models
A state space model compresses the entire past into a fixed-size hidden state that is updated one input at a time, giving linear-time inference without the quadratic score matrix that attention requires.
- Core ML
T5 & Text-to-Text Transformers
T5 reframes every NLP task as text in, text out, so classification becomes generating a label word instead of building a separate output head per task.
- Core ML
Transformer Architecture
Stack blocks of self-attention plus a feed-forward network, each wrapped in a residual connection and normalization, and repeat that same block many times to build the whole model.
- Core ML
Vision Transformers
Vision Transformers slice an image into fixed-size patches, treat each one as a token, and run ordinary self-attention over that sequence of image tokens.
- Core ML
BERT & Masked Language Modeling
BERT reads a sentence in both directions at once and learns by guessing words hidden behind a mask, building representations rather than generating new text.
- Core ML
Causal Masking
Force every future position's attention score to negative infinity before the softmax, so a token can only ever attend to itself and whatever came before it.
- Core ML
Encoder-Decoder Architectures
Three ways to arrange the very same transformer block: read the whole input at once, generate one token at a time, or bridge both with cross-attention.
- Core ML
Feed-Forward Networks
Expand every position to a wider hidden dimension, apply a nonlinearity, then project back down — the transformer's second sub-layer, working on each position entirely on its own.
- Core ML
Layer Normalization
Standardise every feature of one example against its own mean and spread, never against the batch, so the same example normalises identically whether it arrives alone or with ninety-nine others.
- Core ML
Positional Encoding
Add a unique signal built from sine and cosine waves at different frequencies to each token's embedding, since self-attention has no way to tell one position from another on its own.
- Core ML
Pre-Norm vs Post-Norm
Normalizing a sub-layer's input before it runs keeps the residual stream itself untouched at every layer, which trains far more stably at depth than normalizing after adding the sub-layer's output.
- Core ML
RMSNorm
Skip the mean-subtraction step layer normalization performs and rescale a vector by its root-mean-square magnitude alone — cheaper, with one fewer learned parameter, and it matches layer norm's accuracy in most large language models.
- Core ML
SwiGLU
SwiGLU swaps a plain feed-forward activation for a gated one: one projection decides how much of a second projections output actually gets through it.
Recurrent
- Core ML
Attention Mechanism
Instead of reading one fixed summary vector, let the decoder look back at every encoder state and weigh each one by how relevant it is to the current step.
- Core ML
Bidirectional RNNs
Run one recurrent chain forward and a second one backward over the same sequence, then combine them, so every position sees context from both directions at once.
- Core ML
Vanilla RNN
Reuse the same small network at every time step, and let a hidden state carry forward whatever happened earlier so order in the sequence actually matters.
- Core ML
Gated Recurrent Unit
Merge the LSTM's two states into one and cut three gates down to two, trading a little modeling flexibility for a noticeably smaller, faster-to-train cell.
- Core ML
Long Short-Term Memory
Give the network a separate memory line that gates control instead of overwrite, so relevant information from many steps back can survive untouched until it's needed.
- Core ML
Sequence-to-Sequence
An encoder compresses an input sequence into one fixed-size vector, and a decoder generates an output sequence from that vector alone — the shape behind translation and summarization.
- Core ML
Temporal Convolutional Networks
Stack causal, dilated convolutions instead of a recurrent loop, so a sequence model can be trained with every position processed in parallel rather than one step at a time.
Time Series Modelling
- Core ML
Deep Forecasting Models
Neural networks purpose-built for forecasting, capable of ingesting massive datasets, finding complex non-linear patterns, and predicting multiple future steps simultaneously.
- Core ML
Time Series Foundation Models
Massive, pre-trained neural networks that have already 'read' millions of time series. You can use them to forecast your own data without training them from scratch (Zero-Shot forecasting).
- Core ML
Gradient Boosting for Forecasting
Using powerful tabular models like XGBoost to forecast time series by manually converting the time element into columns of historical features.
- Core ML
Hierarchical Forecasting
A mathematical framework that ensures predictions made at the bottom level of a business (e.g., individual store sales) perfectly add up to the predictions made at the top level (e.g., total national sales).
- Core ML
Probabilistic Forecasting
Instead of predicting exactly what a single number will be tomorrow, probabilistic models predict a range of possibilities, outputting a distribution with confidence intervals (e.g., 'we are 90% sure sales will be between 50 and 80').
Unsupervised
- Core ML
Gaussian Mixture Models
Your data came from picking one of k bell curves and drawing a point from it. Fitting means recovering the curves, and every point gets a share of each.
- Core ML
Anomaly Detection
Finding rare rows that do not belong differs from finding outliers in one column, and the wrong metric can make a detector that misses everything look perfect.
- Core ML
Association Rule Mining
Support counts how often items appear together, confidence measures how reliably one predicts another, and lift checks whether that beats pure chance.
- Core ML
Clustering Evaluation
There is no answer key, so every score you can compute encodes somebody's definition of a cluster and quietly rewards whichever algorithm happens to share it.
- Core ML
DBSCAN
A point with enough neighbours close by anchors a cluster, and clusters spread through those anchors, so shape is free and leftover points are called noise.
- Core ML
Hierarchical Clustering
Merge the two closest groups, over and over, recording how far apart they were each time. The word closest is doing far more work here than it looks like.
- Core ML
Independent Component Analysis
PCA finds uncorrelated directions. ICA finds statistically independent ones, a stronger condition that actually recovers the original mixed-together signals.
- Core ML
Isolation Forest
Random cuts isolate an odd point in few splits and a typical point in many, so isolation depth becomes the anomaly score, with no distance or density computed.
- Core ML
K-Means
Point every observation at its nearest centre, move each centre to the mean of its points, repeat. It always settles, and not always in the right place.
- Core ML
Linear Discriminant Analysis
PCA rotates axes toward the direction data spreads most. LDA rotates toward whatever direction best separates classes you know instead, rarely the same axis.
- Core ML
Matrix Factorization
Reconstruct a matrix as the product of two smaller ones, and missing entries fill in for free, since the two factors were fit to explain only entries you have.
- Core ML
Mean Shift
Every point drifts uphill toward wherever the local density peaks, and where it stops is its cluster's mode. No k, no centroid, just the data's own shape.
- Core ML
One-Class SVM
Learn one boundary around normal data using only normal examples, then flag anything outside it as novel, without seeing an example of what abnormal looks like.
- Core ML
Principal Component Analysis
Turn the coordinate axes until the first runs along the direction your data spreads most, then delete the axes that barely move. That is the entire method.
- Core ML
t-SNE
It keeps neighbours together and protects nothing else, so the gaps between the clusters you can see, and the sizes of those clusters, carry no information.
- Core ML
UMAP
Build a neighbour graph, scale each point's edges by its own nearest neighbour, then lay the graph out. Faster than t-SNE, and the gaps still aren't distances.
Regression
- Core ML
Generalized Linear Models
Least squares and logistic regression are the same machine with two settings, a link function and a noise family. Swap them and count data starts working.
- Core ML
Bayesian Linear Regression
Keep a full distribution over plausible coefficients that narrows as data arrives, so every prediction comes with honest, data-driven uncertainty attached.
- Core ML
Gaussian Process Regression
Put a prior directly over the space of possible functions, rather than a fixed set of coefficients, and let the data narrow it to the ones that fit.
- Core ML
Linear Regression
Prediction is a weighted sum plus an intercept, fitted to make the leftover errors smallest. Those leftovers are the most useful thing the model hands back.
- Core ML
Multicollinearity
When two input features move almost identically, a regression splits credit between them almost arbitrarily — coefficients swing wildly, but predictions barely move.
- Core ML
Ordinary Least Squares
Least squares drops a perpendicular from your data onto everything your features can build. One picture that explains the algebra and where it stops holding.
- Core ML
Polynomial Regression
Add powers of the input as features and a straight-line model can bend into any curve — bend it too far and it memorizes noise instead of the trend.
- Core ML
Quantile Regression
Fit a line that a chosen fraction of outcomes fall below, instead of the average outcome — predicting a spread instead of one collapsed number.
Ethics And Safety
- Core ML
Model Cards and Transparency
You wouldn't eat packaged food without an ingredients list, and you shouldn't deploy an ML model without documentation explaining exactly what it was trained on, how it performs, and where it fails.
- Core ML
AI Governance Frameworks
Governance is how you translate vague ethical principles into concrete organizational rules so that a company doesn't accidentally build or buy harmful AI.
- Core ML
Fairness Mitigation
Finding bias is step one. Step two is fixing it by altering the training data, tweaking the learning process itself, or adjusting the predictions after the fact.
- Core ML
Responsible AI in Practice
You cannot staple ethics onto a model after it is built. Responsible AI means embedding checks for fairness, privacy, and safety into every single step of the machine learning lifecycle.
Evaluation
- Core ML
Model Evaluation
Scoring a model is four decisions and not one number: what it never saw, which mistake you count, where you cut the score, and who it quietly fails for.
- Core ML
Classification Metrics
Precision prices a false alarm, recall prices a miss, and one threshold moves both. Every number you report is a claim about which mistake you'd rather make.
- Core ML
Confusion Matrix
Four counts, one grid. Positive means the model said yes, true or false means whether it was right, and every other classification number divides these cells.
- Core ML
Fairness Metrics
Math cannot define what is fair, but it can measure exactly how unequal your model's mistakes are across different demographic groups.
- Core ML
Precision-Recall Curve
Neither axis touches the true-negative count, so a huge pile of obvious negatives can't flatter you. The baseline is the positive class rate, not one half.
- Core ML
Probability Calibration
A model can rank perfectly and still lie about the odds. Calibration asks a narrower thing: of everything you scored 0.70, did about 70% actually happen?
- Core ML
Regression Metrics
Each metric is a claim about which mistakes cost most: MAE says a miss is a miss, RMSE says the worst day owns the score, R-squared says nothing on its own.
- Core ML
ROC and AUC
AUC measures one thing: the chance a random positive scores above a random negative. That's ranking quality, and it can't see calibration or class imbalance.
- Core ML
Threshold Selection
0.5 is a default that nobody chose. Put a price on each kind of mistake and the cut that minimises expected cost falls out as a ratio of those two prices.
Evaluation Science
- Core ML
Model Explainability (XAI)
When a model makes a high-stakes decision, humans demand to know why. Explainability is the suite of mathematical techniques used to open the 'black box' and reveal which features drove the prediction.
- Core ML
Bayesian Neural Networks (BNNs)
What if, instead of assigning an exact, rigid number to every parameter, a neural network assigned a flexible probability distribution to every parameter? It could explicitly tell you when it was guessing.
- Core ML
Integrated Gradients
Basic saliency maps fail when a network is extremely confident (gradient saturation). Integrated Gradients fixes this by calculating gradients along a linear path from a blank image to the actual image.
- Core ML
Aleatoric vs Epistemic Uncertainty
When a model is unsure, it is for one of two reasons: the data is noisy (Aleatoric) or the model is ignorant because it hasn't seen this before (Epistemic).
- Core ML
Calibration
A highly accurate model can still be a dangerous liar if it claims to be 99% confident when it's actually guessing. Calibration measures whether a model's stated confidence matches its true accuracy.
- Core ML
Conformal Prediction
Instead of forcing a model to make a single, overconfident guess, conformal prediction wraps around the model and forces it to output a set of possibilities with a mathematical guarantee.
- Core ML
Counterfactual Explanations
Instead of explaining why a model rejected you, a counterfactual explanation tells you the exact minimum changes you need to make to your profile to get approved.
- Core ML
Deep Ensembles
A single model will confidently lie to you if it doesn't know the answer. But if you ask five different models, they will all lie in completely different ways, exposing the fact that they are guessing.
- Core ML
Error Analysis
The process of manually reading model failures, grouping them into causes, and counting the groups to convert an opaque error rate into a ranked list of fixable problems.
- Core ML
Evaluation Harness Design
A systematic infrastructure that isolates models from datasets and metrics, ensuring that a score change is caused by the model, not the testing pipeline.
- Core ML
Statistical Significance in Evaluation
Because models are trained using random seeds and evaluated on limited data samples, performance metrics are distributions, not single numbers. You must report intervals to prove an improvement is real.
- Core ML
Grad-CAM
Basic saliency maps are too noisy because pixel-level gradients are chaotic. Grad-CAM fixes this by calculating gradients at the final convolutional layer, producing smooth, highly semantic heatmaps.
- Core ML
Human Evaluation Protocols
Because humans are subjective, inconsistent, and get tired, you cannot just ask a human to 'rate this output.' You must use rigorous protocols involving shared rubrics, inter-rater reliability, and conflict adjudication.
- Core ML
LIME
You cannot explain a complex model globally, but if you zoom in close enough on a single prediction, the boundary looks like a straight line. LIME builds a simple model to explain that local area.
- Core ML
Monte Carlo Dropout (MC Dropout)
What if you could turn one model into a massive ensemble of thousands of models for free? Just leave Dropout turned on while you make predictions.
- Core ML
Out-of-Distribution (OOD) Detection
A model only knows the world it was trained on. OOD detection acts as a bouncer at the door, blocking completely foreign data from ever reaching the model.
- Core ML
Partial Dependence and ICE
Once you know a feature is important, you need to know exactly how it changes the prediction. PDP shows the average effect across all users, while ICE shows the unique effect on every single individual.
- Core ML
Permutation Feature Importance
To figure out how much a model relies on a specific feature, intentionally destroy that feature by shuffling its data. The bigger the drop in model accuracy, the more important the feature was.
- Core ML
Saliency Maps
Instead of telling you which column in a spreadsheet is important, a saliency map highlights the specific pixels in an image that caused a neural network to make its prediction.
- Core ML
Selective Prediction
In high-stakes environments, a model shouldn't guess. Selective prediction adds a safety threshold that allows the AI to explicitly abstain from making a decision, routing the problem to a human.
- Core ML
SHAP Values
Using cooperative game theory to distribute the exact 'credit' for a model's prediction among the features, ensuring a mathematically fair, perfectly additive explanation.
- Core ML
Slice-Based Evaluation
A single aggregate metric hides critical failures; slice-based evaluation reveals them by partitioning the dataset into distinct cohorts and scoring each independently.
- Core ML
Task-Specific Evaluation Design
Public benchmarks tell you if a model is generally smart, but they cannot tell you if it will succeed at your specific business use case. You must build your own private evaluation.
- Core ML
Temperature Scaling
Deep neural networks are naturally overconfident. Temperature scaling mathematically softens their probabilities, making a '99% confident' model actually right 99% of the time.
- Core ML
Attention is Not Explanation
Just because a Transformer's attention mechanism highlights a word does not mean that word caused the final prediction. Attention measures data routing, not mathematical contribution.
- Core ML
Benchmark Contamination
When the answers to your evaluation test accidentally leak into your training dataset, giving the model a falsely inflated score because it memorized rather than learned.
- Core ML
Benchmark Saturation
When a model approaches the absolute upper limit of a test's score, the test loses its ability to measure progress or distinguish between competing models.
Security
- Core ML
Model Extraction
By repeatedly querying a commercial API and recording its answers, an attacker can train a cheap, local 'knock-off' model that perfectly mimics the expensive proprietary model.
- Core ML
Model Inversion
Model inversion is a reverse-engineering attack where an adversary reconstructs the actual raw training data (like faces or text) by analyzing how the model behaves.
- Core ML
Adversarial Examples
By adding invisible, mathematical noise to an input, attackers can trick a highly accurate neural network into making bizarre, confidently incorrect predictions.
- Core ML
Adversarial Training
To make a model immune to adversarial attacks, developers generate attacks during the training process itself and force the model to learn the correct labels for the poisoned data.
- Core ML
Bias and Fairness
Machine learning models mathematically encode the human biases present in their training data. Mitigating this requires algorithmic interventions to ensure the model's predictions do not unfairly discriminate against protected groups.
- Core ML
Content Moderation Systems
A content moderation system is an automated pipeline that uses fast classification models to filter out toxic, illegal, or harmful content before a human ever has to see it.
- Core ML
Data Poisoning and Backdoors
Instead of attacking a finished model, data poisoning attacks the training data itself, inserting hidden backdoors that the model learns to obey.
- Core ML
Differential Privacy
Differential privacy adds a calculated amount of mathematical noise during training, guaranteeing that a model's outputs will look essentially identical whether any specific individual's data was included in the training set or not.
- Core ML
Federated Learning
Instead of bringing private user data to a central server to train a model, federated learning brings the model to the user's device, trains it locally, and only sends the mathematical updates back to the server.
- Core ML
Machine Unlearning
Machine unlearning is the mathematical process of forcing a model to 'forget' a specific piece of training data without having to retrain the entire model from scratch.
- Core ML
Membership Inference
By analyzing how confidently a model predicts an output, an attacker can mathematically determine whether a specific person's data was used in the model's training set.
- Core ML
PII Detection and Redaction
Before processing sensitive data or sending it to an LLM, a system must automatically identify and mask personally identifiable information to prevent privacy leaks.
Compression
- Core ML
Model Pruning
Most of a trained network's weights are small enough to zero out with almost no accuracy loss, so pruning finds and removes exactly those, leaving it sparse.
- Core ML
Knowledge Distillation
A large trained model's full probability spread over every class, not just its top pick, becomes the training target a much smaller model learns to match.
- Core ML
Quantization-Aware Training
Round weights to low precision during training itself, not after, so gradients push the network toward values that survive rounding instead of being surprised.
Training
Reinforcement Learning
- Core ML
Reinforcement Learning Foundations
There's no answer key, just a number that lands after you act. The framework is a Markov decision process, and the reward never names a correct action.
- Core ML
Policy Gradient Methods
Push up the probability of actions that paid off. The estimate is unbiased and wildly noisy, so every refinement after REINFORCE is about shrinking variance.
- Core ML
Proximal Policy Optimization
Reuse the same batch of experience for several updates, but clip the objective so no single update can push the policy too far from where it started.
- Core ML
Q-Learning and DQN
Learn what each action is finally worth, then take the biggest number. Two tricks made that work with a network: sampled replay, and a target that holds still.
Classical Nlp Embeddings And Tasks
- Core ML
Sentence Embeddings
An embedding technique that maps entire sentences or paragraphs into a single dense vector, allowing algorithms to compare the semantic meaning of large blocks of text.
- Core ML
FastText
An extension of Word2Vec that represents words as bags of character sub-words, allowing it to understand morphology and generate vectors for words it has never seen before.
- Core ML
Named Entity Recognition
The task of scanning unstructured text to locate and classify proper nouns, such as people, organizations, dates, and locations.
- Core ML
Part-of-Speech Tagging
The grammatical foundation of NLP, where every word in a sentence is classified by its syntactic role (noun, verb, adjective, etc.).
- Core ML
Text Classification
The foundational NLP task of assigning a predefined label or category to a given block of text.
Validation
- Core ML
Statistical Model Comparison
Two accuracies from one test set aren't independent draws. McNemar's test uses only the disagreements, more powerful than treating the two scores as separate.
- Core ML
Bayesian Optimization
A surrogate model predicts both a score and its uncertainty everywhere, and an acquisition function picks the next expensive trial from that belief map.
- Core ML
Hyperparameter Tuning
Grid search spends its budget evenly across knobs whether or not they matter. Random search spends that same budget where the score actually responds.
- Core ML
Learning Curves
Plot error against training set size, not epochs. If both curves have flattened and the gap has closed, more data buys you nothing. That's the whole diagnosis.
- Core ML
Nested Cross-Validation
Tuning against the same fold that reports the final score lets a search overfit the split. Two loops, one inside the other, keep the reported number honest.
- Core ML
Time Series Cross-Validation
Shuffling a time-ordered fold lets a model peek at the future to score the past. Expanding and sliding windows keep training strictly earlier than validation.
- Core ML
Validation Curves
Learning curves sweep data volume. Validation curves sweep one hyperparameter instead, and the U-shape it traces diagnoses over- and underfitting directly.
Business Decisions And Tabular
- Core ML
Tabular Foundation Models
What if you could train a Transformer on millions of completely random Excel spreadsheets, so that when you give it a brand new spreadsheet, it can predict the target column in 1 second without any training?
- Core ML
Tabular Deep Learning
Deep learning dominates images, text, and audio. But if you throw a standard neural network at an Excel spreadsheet, it will usually lose to a 10-year-old tree-based model. Tabular Deep Learning tries to fix this.
- Core ML
Entity Resolution
Is 'Apple Inc.', 'Apple Computer', and 'Apple' the same company? Entity resolution is the process of looking at messy, real-world data and figuring out which records actually point to the exact same real-world object.
- Core ML
Graph Representation Learning
Machine learning models only understand matrices of numbers. Graph representation learning is the math used to compress a messy web of relationships into a clean, flat matrix of vectors, without losing the network structure.
- Core ML
Knowledge Graphs
Instead of storing data in rigid SQL tables or unstructured text, store it as a web of concepts. 'Steve Jobs' (Node) -> 'Founded' (Edge) -> 'Apple' (Node).
- Core ML
Optimisation for Decisions
Machine Learning predicts the future. Optimization tells you exactly what to do about it. When you combine them, you transition from Descriptive Analytics to Prescriptive Analytics.
- Core ML
Propensity and LTV
Instead of treating every customer equally, you calculate two numbers: how likely are they to do something (Propensity), and how much money will they generate over their entire relationship with you (Lifetime Value)?
- Core ML
Survival Analysis
Standard classification answers 'Will this user churn?'. Survival Analysis answers a much more useful question: 'How long until this user churns?'
Classical Nlp Close Out
- Core ML
Topic Modeling
An unsupervised machine learning technique that scans a massive collection of documents and automatically discovers the hidden themes or "topics" running through them.
- Core ML
Dependency Parsing
The process of analyzing the grammatical structure of a sentence to determine how words relate to each other, establishing a tree of dependencies.
- Core ML
Machine Translation
The task of automatically translating text from one language to another while preserving the original semantic meaning and grammatical structure.
- Core ML
Text Summarization
The task of condensing a long document into a shorter version while preserving the core informational content and overall meaning.
Causal Inference And Experimentation
- Core ML
Uplift Modeling
Don't waste marketing budget on people who were going to buy anyway, and definitely don't spend it on people who will get annoyed and churn. Uplift modeling predicts who will change their behavior *because* of your intervention.
- Core ML
A/B Testing for ML
How to evaluate a model's business impact using statistical experiments.
- Core ML
Causal Graphs
A causal graph is a map of how the world works. It tells you which variables to control for, and more importantly, which variables you must ignore to find the true effect.
- Core ML
Confounding & Colliders
Controlling for a confounder removes bias. Controlling for a collider creates bias. The hardest part of causal inference is knowing which one you are looking at.
- Core ML
Correlation vs Causation
When two things move together, one might cause the other, or a hidden third factor might be driving both. ML optimizes correlation; intervention requires causation.
- Core ML
Difference-in-Differences (DiD)
If two cities have always grown at the exact same rate, and then one gets a new ML feature while the other doesn't, any sudden change in their growth rate must be caused by the new feature.
- Core ML
Double Machine Learning (DML)
Standard ML models are obsessed with correlation. If you try to extract a causal effect from a Random Forest, it will give you a biased answer. Double ML uses two models to strip away all the correlation, leaving only the causal truth.
- Core ML
Heterogeneous Treatment Effects (HTE)
An average effect of +$5 hides the fact that the feature made +$15 for young users and lost -$5 for older users. HTE moves from asking 'Did it work on average?' to 'Who exactly did it work for?'
- Core ML
Instrumental Variables (IV)
When a confounder completely obscures the causal effect, find a random variable (an instrument) that only affects the treatment, not the outcome. It acts as a natural randomized experiment hidden inside your observational data.
- Core ML
Potential Outcomes
For every decision, there are two alternate universes: one where you took the action, and one where you didn't. Causal inference is the math of comparing them when you can only ever observe one.
- Core ML
Propensity Score Methods
When you can't run an A/B test, you have to mimic one. Propensity scores compress all of a user's confounding traits into a single probability, allowing you to match treated users with identical control users.
- Core ML
Randomized Experiments
When you flip a coin to decide who gets a treatment, you guarantee that nothing else caused the decision. Randomization severs the link from all confounders, known and unknown.
- Core ML
Sequential Testing
Standard A/B testing forbids you from peeking at the results before the test ends. Sequential testing changes the math so you can look at the data every single day and stop early without ruining the validity of the test.
Generative
- Core ML
Vector-Quantized Models
Snap every encoder output to the nearest entry in a fixed, learned codebook, turning a continuous latent space into a finite set of discrete symbols.
- Core ML
Autoencoders
Force data through a layer too narrow to hold it, then ask for the original back. Whatever survives the squeeze is the structure the data actually has.
- Core ML
Generative Adversarial Networks
Nobody writes the loss. A second network learns to spot fakes, and the generator's only job is to fool it, which makes it a game rather than a minimisation.
- Core ML
Variational Autoencoders
A plain autoencoder's latent space has holes, so sampling it gives nothing. Make the encoder emit a distribution instead of a point and the gaps fill in.
Networks
- Core ML
Activation Functions
Six functions, each fixing the last one's defect. Sigmoid's gradient dies with depth, ReLU's never shrinks, and the output layer's choice is the task's call.
- Core ML
Batch Normalization
Standardise each unit's values across the batch, then let the layer learn a scale and shift back. Training uses this batch; inference uses a running average.
- Core ML
Exploding Gradients
The same layer-by-layer multiplication behind vanishing gradients runs the other way when local derivatives are consistently above one, and the gradient grows exponentially with depth instead of shrinking toward zero.
- Core ML
Gradient Clipping
Cap the gradient's size before the optimizer step, either by shrinking each entry to a fixed range or by rescaling the whole vector, so one bad batch can't throw the weights somewhere training never recovers from.
- Core ML
Loss Functions
Every common loss is the negative log-likelihood of some noise model, so picking one is a claim about how your data got its errors. Choose the claim first.
- Core ML
Vanishing Gradients
Backpropagation multiplies local derivatives layer by layer, and when those derivatives are consistently below one, the product shrinks toward zero the deeper it travels.
- Core ML
Backpropagation
One forward pass stores what each layer computed. One backward pass hands every parameter its gradient, because the gradients share almost all their work.
- Core ML
Debugging Training Runs
Run one ordered protocol instead of guessing: overfit a single batch first, check the loss at initialization against chance level, then shapes and masks, then the learning rate, then precision — each step's result decides the next.
- Core ML
Dropout
Zero out a random slice of a layer on every training pass, and one set of weights quietly trains a huge family of thinned networks that vote at inference.
- Core ML
Group Normalization
Split a layer's channels into fixed-size groups and normalise each group within one example, never across the batch, so a detector running two images per GPU normalises exactly as reliably as one running two hundred.
- Core ML
Multi-Layer Perceptron
A hidden layer invents new coordinates, and in those coordinates one straight line is enough. Depth is a change of representation, not extra parameter room.
- Core ML
Perceptron
One neuron learning a boundary from its own mistakes and nothing else. It halts the instant nothing is wrong, and loops forever when no straight line exists.
- Core ML
Universal Approximation
A wide-enough single hidden layer can get within any accuracy you name of any continuous function on a bounded domain — but the theorem only proves such weights exist, never that gradient descent can find them or that the required width is affordable.
- Core ML
Weight Decay
Adding a squared-size penalty to the loss is the same as multiplying every weight by a shade under one each step. That equality breaks under Adam, so AdamW.
- Core ML
Weight Initialization
Random weights break symmetry between units, but the wrong scale shrinks every layer's output toward zero or blows it up — the right scale depends on the layer's width.
Optimization
- Core ML
Batch Size Effects
The batch size sets how noisy each gradient estimate is, trading off against how many steps fit in a compute budget and how well the model ends up generalizing.
- Core ML
Gradient Accumulation
Run several small forward-backward passes and sum their gradients before one optimizer step, so limited GPU memory can still train as if it saw one large batch.
- Core ML
Gradient Descent
Work out which way the loss falls, step a little that way, repeat. That single loop trains almost everything, and the step size decides whether it works.
- Core ML
Second-Order Optimization
First-order methods only feel the slope underfoot. Second-order methods use curvature too, cutting steps needed sharply, at a memory cost that rules them out at scale.
- Core ML
Stochastic Gradient Descent
One row per step instead of all of them looks like a shortcut. The wobble it adds is doing real work: it walks the model out of shallow dips and saddles.
- Core ML
Adam and AdamW
Adam keeps two running averages of the gradient, so every parameter gets its own step size. AdamW moves the weight decay after that division, and took over.
- Core ML
Adaptive Optimizer Landscape
Adam's per-parameter scaling is one point on a family of optimizers, each trading memory or compute for a different way to tame wildly different gradient sizes.
- Core ML
Learning Rate Scheduling
Instead of one fixed learning rate for the whole run, schedule it to shrink over training so early steps move fast and later steps settle in carefully.
- Core ML
Learning Rate Warmup
Ramp the learning rate up from near zero over the first few hundred steps, because the earliest updates run on the least trustworthy gradient estimates.
- Core ML
Mixed Precision Training
Run the forward and backward pass in a lower-precision format for speed and memory savings, while keeping a full-precision master copy of the weights for accuracy.
- Core ML
Momentum
Plain descent zig-zags across a narrow valley. Average the last few gradients instead: the sideways pushes cancel and the downhill push keeps stacking up.
Convolutional
- Core ML
CNN Architecture Lineage
One lineage, one fix at a time — from the first working digit classifier to networks a thousand layers deep, then engineered back down to fit a phone.
- Core ML
Convolutional Neural Networks
Stack convolution, activation, and pooling, then read out with a classifier — the architecture that made raw pixels a workable neural-network input.
- Core ML
Convolution Operation
One small grid of weights slides across the whole input, multiplying and summing at every position — the same small set of weights, reused everywhere.
- Core ML
Depthwise Separable Convolutions
Split one expensive convolution into a cheap per-channel pass and a cheap channel-mixing pass, cutting the parameter count by nearly an order of magnitude.
- Core ML
Pooling Layers
Shrink a feature map by keeping only the strongest signal in each small patch, trading exact position for tolerance to roughly where a pattern sat.
- Core ML
Receptive Fields
Each unit only ever looks at a small patch directly, but stacking layers lets that patch grow until a deep unit has effectively seen the whole image.
- Core ML
Residual Connections
Let a layer learn only the correction and pass the original input straight through untouched — the identity path that made 100-plus-layer networks trainable.
Ensembles
- Core ML
Gradient-Boosted Tree Libraries
Production boosting libraries add three things to the algorithm: features bucketed into bins, the greediest leaf grown first, and categories encoded in order.
- Core ML
Gradient Boosting
Ordinary gradient descent nudges parameters. Gradient boosting nudges the prediction itself, one small tree per step, fitted to the gradient of your loss.
- Core ML
AdaBoost
Fit a weak learner, see which rows it got wrong, make those rows count for more, refit. The vote weights and the row weights are both one line of algebra.
- Core ML
Bagging
Fit the same twitchy model to many resamples of your own data, then average. Variance falls fast, then stalls, because the resamples overlap so heavily.
- Core ML
Boosting
Bagging fits models in parallel and averages the variance away. Boosting fits them in sequence, each cleaning up after the last, and takes bias down instead.
- Core ML
Feature Importance
The importance your tree library prints for free answers a narrower question than it looks like. Permuting a column on held-out data answers the one you meant.
- Core ML
Random Forests
Bagged trees stall when one column dominates every split. Hide most of the columns at each split and the trees stop agreeing, which is when averaging pays.
- Core ML
Stacking
Averaging models with fixed weights can't know which one to trust when. Train a meta-model on their predictions instead, and it learns the weighting for you.
Representation Learning
- Core ML
Graph Neural Networks
Each node collects information from its neighbors, combines it with its own, and repeats — so a network can learn from relationships, not just isolated rows.
- Core ML
Continual Learning
Train on a new task and the old one collapses, because nothing in the new gradient knows the old loss exists. Every fix is a form of don't move too far.
- Core ML
Contrastive Learning
Two crops of one photo should land close together, crops of other photos far apart. Whatever you augment away is exactly what the model learns to ignore.
- Core ML
Domain Adaptation
A model trained on one data distribution gets deployed on a shifted one, and domain adaptation closes that gap without collecting new labels.
- Core ML
Multi-Task Learning
Train one shared network on several related tasks at once, so what one task teaches the shared layers can help the others, not just itself.
- Core ML
Self-Supervised Learning
Hide part of your own data and train the model to predict what you hid. The answer was already sitting there, so you get supervision that nobody paid for.
- Core ML
Transfer Learning
A pretrained model already knows edges, textures and syntax. So freeze all of that, train a new head, and unfreeze deeper only when your labels support it.
Automl
Time Series Foundations
- Core ML
ARIMA
The classic time series algorithm that forecasts the future by combining the momentum of past values (AutoRegressive) with the momentum of past errors (Moving Average), after stabilizing the data (Integrated).
- Core ML
Autocorrelation
A statistical metric that measures how strongly a time series is correlated with its own past values, revealing hidden momentum and seasonal patterns.
- Core ML
Exponential Smoothing
A time series forecasting method that predicts the future by taking a weighted average of past observations, where recent observations are given exponentially more weight than older ones.
- Core ML
Stationarity
The property of a time series whose statistical properties (like mean and variance) do not change over time. A strict requirement for almost all classical forecasting algorithms.
- Core ML
Time Series Decomposition
The mathematical process of breaking a time series down into three distinct components: the overall trend, the repeating seasonal pattern, and the random noise.
Time Series Evaluation
- Core ML
Backtesting (Time Series)
Evaluating a forecasting model by simulating how it would have performed in the past, strictly using expanding or rolling windows to ensure the model never 'peeks' into the future.
- Core ML
Change Point Detection
Finding the exact moment in time when the underlying statistical properties (mean or variance) of a dataset permanently shifted, creating a 'new normal'.
- Core ML
Forecast Evaluation
The mathematical metrics used to grade how 'wrong' a forecasting model is. Different metrics punish different types of errors, such as massive outliers vs small consistent misses.
- Core ML
Time Series Anomaly Detection
Finding specific data points in a timeline that severely violate expected patterns, usually indicating a system failure, a fraudulent transaction, or a sensor malfunction.
Classical Nlp Representations
- Core ML
Bag of Words
A simple text representation that counts how many times each word appears, ignoring grammar, word order, and context.
- Core ML
GloVe
An embedding technique that generates dense word vectors by training on global word-word co-occurrence statistics across an entire corpus.
- Core ML
Text Preprocessing
Before a model can read text, the text must be cleaned, normalized, and split into tokens so mathematical operations can be applied to it.
- Core ML
TF-IDF
A statistical measure that evaluates how relevant a word is to a document in a collection, rewarding frequent words but penalizing words that appear everywhere.
- Core ML
Word2Vec
A neural network technique that converts words into dense vectors, placing words with similar meanings close together in a mathematical space.
Framing
- Core ML
Bias–Variance Tradeoff
Error splits into three parts: a wrong-shaped model, a jumpy one, and noise you can never fix. Only the first two are yours to trade against each other.
- Core ML
Double Descent
Push capacity past the point where a model fits every training row exactly and test error spikes, then falls again — often below the U-curve's first minimum.
- Core ML
Early Stopping
Watch validation loss every epoch and stop once it hasn't improved for a set number of checks, then restore the weights from whichever epoch actually scored best — not the one training happened to end on.
- Core ML
The ML Workflow
Problem, data, model, evaluation, shipping, and monitoring run in a loop, not a line — what monitoring finds reframes the problem, not just the model.
- Core ML
No Free Lunch
Averaged across every possible problem, no learning algorithm beats another — an edge on the problems you care about costs ground somewhere else.
- Core ML
Overfitting and Underfitting
Underfitting is both curves high and flat. Overfitting is training loss still falling while validation loss turns up. You read it off curves, not off vibes.
- Core ML
Statistical Learning Theory
A generalization bound caps how far true error can exceed training error — and the honest catch is that cap is usually far too loose to use as a real number.
- Core ML
What Is Machine Learning?
Instead of writing the rules yourself, you hand over examples and let a procedure find the function. What arrives alongside those examples splits the field.
Recommender Systems Ranking
- Core ML
The Cold Start Problem
Machine learning recommenders rely on historical data. If a brand new user signs up, or a brand new product is added to the store, there is zero historical data. The algorithm completely freezes.
- Core ML
Learning to Rank (LTR)
Instead of training a model to predict an absolute score (like '4.2 stars'), LTR trains a model to understand relative ordering ('Item A is better than Item B').
- Core ML
Sequential Recommendation
Predicting what a user wants next by analyzing the strict chronological order of their past actions, recognizing that buying a phone case usually happens *after* buying a phone, not before.
- Core ML
Two-Tower Retrieval
A neural network that passes the User through one 'tower' and the Item through another. The two towers never touch until the very end, allowing for blazing fast candidate retrieval.
Recommender Systems Architecture
- Core ML
Collaborative Filtering
Recommending items to a user based on what similar users liked. It entirely ignores what the item actually is, focusing purely on the overlapping behavior of the crowd.
- Core ML
Implicit Feedback
Deducing what a user likes by observing their silent actions (clicking, watching, scrolling) rather than asking them to explicitly rate an item with 5 stars.
- Core ML
Matrix Factorization
Squeezing a massive, mostly-empty grid of users and items into two tiny matrices. By doing this, the algorithm is forced to invent 'hidden topics' (Latent Factors) that explain why a user likes a specific item.
- Core ML
Recommender System Architecture
The multi-stage funnel that takes a massive catalog of millions of items and efficiently filters it down to the top 10 most relevant items for a specific user.
Recommender Systems Evaluation
- Core ML
Contextual Bandits
Instead of running a 2-week A/B test and throwing away half your traffic on a losing idea, a Contextual Bandit learns *during* the test, smoothly shifting traffic away from the loser and toward the winner.
- Core ML
Off-Policy Evaluation
Guessing how much money a new algorithm will make, using only the historical data generated by an old algorithm, without actually deploying the new algorithm to real users.
- Core ML
Recommendation Evaluation
You can't use standard ML metrics like Accuracy or RMSE to judge a search engine. We need metrics that care about the exact position of the correct answer, and metrics that prove we aren't just recommending the same 10 popular things to everyone.
Trees
Serving
Regularization
- Core ML
Elastic Net
Mix the two penalties and correlated features stop fighting over one slot. Lasso keeps one of a cluster and drops the rest; a little L2 moves them together.
- Core ML
Lasso Regression
Charge the absolute size of each coefficient instead of its square and the penalty stops easing off near zero, so coefficients land on it. Selection, for free.
- Core ML
Ridge Regression
Add a penalty on coefficient size and the bias-variance tradeoff becomes a dial you turn: accept a little bias on purpose, buy a much bigger drop in variance.
Architecture
- Core ML
Expert Routing and Load Balancing
The router in a mixture-of-experts model scores each token against all experts and picks the top-k — but without a load-balancing loss, all tokens pile onto the same few popular experts, wasting most of the expert capacity.
- Core ML
Mixture of Experts
Replace the single feed-forward layer in a transformer with a bank of N expert networks and a router that picks only k of them per token — total parameter count scales with N, but the compute per token stays constant because most experts are idle.
Label Efficiency
- Core ML
Few-Shot Learning
Classify a new category correctly after seeing only a handful of labeled examples of it, instead of the thousands an ordinary classifier needs per class.
- Core ML
Meta-Learning
Train a starting point across many tasks, not a solution to one, so adapting to any new related task takes only a handful of gradient steps.
- Core ML
Semi-Supervised Learning
Train on a handful of labeled examples, let that model guess labels for the rest, and keep only the guesses it's confident about. Confidence is the whole method.
Classification
- Core ML
K-Nearest Neighbors
No training at all: store every row, then answer each query by finding the k closest rows and taking a vote. The cost moved to prediction, it did not vanish.
- Core ML
The Kernel Trick
The algorithm never needs a point's coordinates in the huge new space, only dot products between pairs, so a kernel computes those and skips the projection.
- Core ML
Logistic Regression
It classifies, despite the name. A linear score goes in, a sigmoid squeezes it down into a probability, and the log-odds come out straight in the features.
- Core ML
Multi-Label Classification
Softmax forces every class to compete for one probability budget. Real tagging often needs several labels at once, which is a different architecture entirely.
- Core ML
Naive Bayes
It treats every word as independent evidence, which is plainly false. The probabilities come out wrong, and the class that wins usually still wins correctly.
- Core ML
Softmax Regression
One score per class, exponentiated then divided by their total. The classes now compete for a fixed budget of one, so only the gaps between them matter.
- Core ML
Support Vector Machines
Of all the lines that separate two classes, pick the one furthest from the nearest point on either side. Convex problem, one answer, and no probability at all.
Mechanistic Interpretability
- Core ML
Activation Patching
To prove a specific layer handles a specific task, you run a corrupted input that fails the task, then overwrite just that layer's activations with the clean ones. If the model suddenly gets the right answer, you found the circuit.
- Core ML
Sparse Autoencoders for Features
Neurons inside an LLM are polysemantic—they fire for many unrelated things at once. A sparse autoencoder takes those messy neuron activations and disentangles them into a massive dictionary where each entry represents exactly one clear, human-readable concept.