Data Concepts
Data engineering, pipelines, and data-centric AI
Foundations
- Data Concepts
Exploratory Data Analysis
Looking at your data before you model it, in a fixed order, where every answer you get turns into a written decision about a column, a row, or a split.
- Data Concepts
Feature Engineering
A model never sees your data. It only ever sees the table of numbers you hand it, and feature engineering is every decision that turns one into the other.
- Data Concepts
Numerical and Categorical Data
What kind of arithmetic a column actually allows. Get that wrong and a model will happily average country codes, or decide midnight is nowhere near 11pm.
- Data Concepts
Structured, Semi-Structured and Unstructured Data
Three kinds of data, split by one question: does the shape of a record exist before the data, travel inside each record, or not exist in writing at all?
Representation
- Data Concepts
Vector Embeddings
One-hot says every category is exactly as unrelated to every other one. An embedding is a short list of numbers whose distances actually mean something.
- Data Concepts
Latent Spaces and Manifold Learning
Real data fills a tiny curved sliver of the space it is stored in. A latent space is the coordinate system on that sliver, and it is where generation works.
- Data Concepts
Multimodal Data Fusion and Alignment
Fusion is the modelling question and alignment is the data question. Getting the image and the words that truly describe it paired up is where the work is.
Curation
- Data Concepts
Weak Supervision
Nobody will hand-label a million rows. Write a dozen noisy rules instead, then model how often they are wrong and combine them into probabilistic labels.
- Data Concepts
Active Learning
Labelling budget is finite, so spend it on the examples the model finds confusing. Random sampling spends most of it confirming what the model already knows.
- Data Concepts
Data Quality Filtering
At web scale most text is not worth training on. Filtering is what turns a scrape into a corpus, and the filter you choose becomes part of the model itself.
- Data Concepts
Dataset Decontamination
If a benchmark's questions are in your training data, its score measures memory. At web scale they are, and checking is a search problem rather than a choice.
- Data Concepts
Dataset Documentation
Six months on, nobody remembers where the rows came from or what the label meant. A data card is the artefact that answers that without the person who built it.
- Data Concepts
Labelling and Annotation
Your labels are the ceiling on your model. If two annotators disagree on a fifth of the examples, nothing you train can be more accurate than they are.
- Data Concepts
Preference Data Collection
Asking which of two answers is better is far more reliable than asking someone to score one. Every aligned model is built on that switch from rating to ranking.
Modalities
- Data Concepts
Audio Data Preparation
One second of speech at 16kHz is 16,000 numbers. Most audio models want a time-frequency picture instead, and every parameter in that chain is a decision.
- Data Concepts
Spatial and Geospatial Data
Latitude and longitude are not two ordinary numbers. Treat them as such and your distances are wrong, and your random split hands the model its neighbours.
- Data Concepts
Graph and Network Data
Here the data is the relationships, so the one-row-per-thing assumption breaks. How you store the edges decides what you can compute and how you must split.
- Data Concepts
Image and Video Data Preparation
An image is a tensor of numbers, and every choice about how those numbers get produced is part of the model. Get one wrong and nothing throws an error.
- Data Concepts
Text Data Preparation
Classical text cleaning and a pretrained tokeniser want opposite things. Strip the casing, punctuation and stopwords for one, and you damage the other.
- Data Concepts
Time-Series and Temporal Data
Ordering is information, and every default in your toolkit assumes it isn't. Shuffle these rows and you have destroyed the exact thing you were modelling.
Feature Engineering
- Data Concepts
Automated Feature Engineering
A generator is good at enumeration and bad at judgement. It will invent a thousand columns, and without a cutoff time most of them quietly know the future.
- Data Concepts
Dimensionality Reduction
One question sorts every method here: does it hand you a transform you can apply to a new row? PCA does. t-SNE never will, which is why it stays a plot.
- Data Concepts
Feature Construction
This is where you add what the table doesn't already know. A ratio, a lag, a count in a window — and every one of them computed as of the prediction time.
- Data Concepts
Feature Selection
Three families of method, and which family you pick matters much less than whether the choosing happened inside the fold or on the whole dataset first.
- Data Concepts
SQL for Feature Extraction
A GROUP BY hands you one row per key. A window function hands you one row per row, and correct feature extraction almost always needs that second shape.
- Data Concepts
Text Feature Representation
Counting words gets you a strong baseline in seconds. TF-IDF weights each count by how rare the token is, and that weighting is most of the whole trick.
Transformation
- Data Concepts
Categorical Encoding
Count the distinct values first. Two, a dozen, or fifty thousand — that number decides how the column becomes numbers, and taste has nothing to do with it.
- Data Concepts
Discretisation and Binning
Binning throws away resolution on purpose. Sometimes that buys you a step a linear model could never fit; hand it to a tree and you've only lost detail.
- Data Concepts
Feature Scaling
Scaling only matters when the algorithm compares columns to each other. If it only compares a column against a threshold, the numbers can stay as they are.
- Data Concepts
Non-Linear Transformations
Scaling slides and stretches a distribution. A log changes its shape, and shape is what turns equal ratios into equal distances a linear model can add up.
- Data Concepts
Target Encoding and Leakage
Replace a category with the average outcome for that category. Do it on the row you're encoding and you've handed the model the answer it was meant to predict.
Lifecycle
- Data Concepts
Columnar Formats
Row formats make you read every column to get one. Columnar formats store each column together, which is why a training loop stops waiting on the disk.
- Data Concepts
Data Augmentation
Augmentation isn't extra data. Every transform tells the model which changes shouldn't change the label, and a transform that can change it is only noise.
- Data Concepts
Data Drift and Distribution Shift
Three different problems get called drift. The inputs moved, the class balance moved, or the right answer moved, and only the last one needs a new model.
- Data Concepts
Data Validation and Contracts
The good failure breaks the pipeline, so you find out immediately. The expensive one keeps everything running and quietly changes what the numbers mean.
- Data Concepts
Data Versioning
Pinning your code does not reproduce your run. If you cannot say which exact rows trained the model, then the comparison you just ran was not a comparison.
- Data Concepts
Online / Offline Skew
The model is fine and the data is fine. The same input still produces two different feature vectors, depending on which of your two code paths computed it.
- Data Concepts
Sequence Packing
Padding short samples to the longest one wastes most of your compute on nothing. Packing concatenates them instead, and the attention mask is the catch.
- Data Concepts
Streaming and Real-Time Data
A batch job sees the whole day before it answers. A stream has to answer now, so the hard part isn't throughput, it's deciding when a window is finished.
- Data Concepts
Synthetic Data Generation
A generator only samples from a distribution it learned, so synthetic data adds coverage and never adds information. Train on it, then test on real data.
- Data Concepts
Training Data Pipelines
Your GPU is probably idle. The bottleneck in most training runs is the code that fetches and decodes the next batch, not the arithmetic that learns from it.
Validation
- Data Concepts
Cross-Validation Schemes
Cross-validation gives you a mean and a spread from the same rows. The spread is the half people ignore, and it's what tells you a 0.3-point win is noise.
- Data Concepts
Data Leakage
Leakage is the model knowing at training time something it can't know when it predicts. It's the one bug that makes your validation score better, not worse.
- Data Concepts
Class Imbalance
At 0.3% fraud, a model that always says no scores 99.7% accurate and is worth nothing. Most imbalance problems are metric problems, so fix the metric first.
- Data Concepts
Train, Validation and Test Splits
Three sets, because you need somewhere to make decisions and somewhere that has never informed one. Look at the test set twice and you no longer have one.
Data Quality
- Data Concepts
Data Cleaning
The same mess arrives again next week. So write every fix down as a rule the data has to pass, and your cleaning turns into something you can trust twice.
- Data Concepts
Data Imputation
Every filled gap is a value you made up. The model can't tell it from a measurement, and your variance and correlations both shift the moment you do it.
- Data Concepts
Deduplication at Scale
Exact duplicates are a hash lookup. Near-duplicates are the expensive ones, and the ones that straddle your split turn memorised text into a benchmark win.
- Data Concepts
Missing Data
A blank cell is a fact about how your data was collected, not an absence of information. Why the value is gone decides what you're allowed to do about it.
- Data Concepts
Noise, Outliers and Anomaly Detection
The same statistics, the opposite action. A height of 3.2 metres is a typing mistake you delete; an odd transaction is the fraud you were hired to catch.
- Data Concepts
Sparse Data and High Dimensionality
Two problems that show up together. Sparsity is about storage and statistics; dimensionality is about geometry, and it makes every distance look the same.