Skip to content
AI360Xpert
Paper Breakdowns
Paper breakdown

Mixtral 8x7B

The 2024 paper from Mistral AI that brought Sparse Mixture-of-Experts (MoE) to the open-source community, enabling a 47B parameter model to run at the speed of a 14B model.

Paper: Mixtral of Experts

Authors: Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed · 2024

Read the paper
In an MoE model, a router network looks at each token and sends it to only the top 2 'expert' neural networks, leaving the other 6 experts inactive to save compute.
In an MoE model, a router network looks at each token and sends it to only the top 2 'expert' neural networks, leaving the other 6 experts inactive to save compute.

The Problem

Scaling up LLMs traditionally means making every layer wider and deeper (a "dense" model). However, this means every single parameter must be loaded into memory and multiplied for every single token generated. A 70B parameter model is slow and expensive to run because it does 70 billion calculations per token. Researchers knew about Sparse Mixture of Experts (MoE)—a way to make a model huge in parameter count but cheap in compute by only using parts of the network at a time—but no high-quality, open-weights MoE existed.

The Idea

Mistral AI released Mixtral 8x7B. Instead of a single massive Feed-Forward Network (FFN) in each transformer block, Mixtral has 8 distinct FFNs (the "experts"). For every single token passing through the layer, a small "router" network decides which 2 experts are best suited to process that specific word. The token is only sent to those 2 experts, and the other 6 are completely ignored. This allows the model to have 47 Billion total parameters (vast knowledge capacity) but only use 13 Billion active parameters during inference (fast generation speed).

How It Works

Mixtral modifies the standard Transformer block:

  1. Shared Attention: The self-attention mechanism is shared across all tokens normally. There are no "experts" in the attention layer.
  2. The Router: After attention, the token hits a routing gate. This is a simple linear layer that outputs a probability distribution over the 8 experts.
  3. Top-2 Routing: The router selects the 2 experts with the highest probabilities.
  4. Expert Execution: The token is passed through only those 2 FFN experts. The outputs of the two experts are multiplied by their router probabilities and summed together.

Because different tokens activate different experts (e.g., a math token might activate Expert 3 and 7, while a French token activates Expert 1 and 2), the model can compartmentalize its knowledge.

Why It Mattered

Mixtral 8x7B shocked the industry by matching or beating GPT-3.5 and Llama 2 70B on most benchmarks, while requiring significantly less active compute than the Llama model. It proved that MoE architectures could be trained stably and served efficiently on consumer hardware (if you had enough VRAM to store the idle experts).

What Came After

Mixtral forced the entire open-source ecosystem to pivot toward MoE architectures. Within months, models like DBRX, Qwen1.5 MoE, and Snowflake Arctic adopted the sparse MoE paradigm to maximize their parameter-to-compute ratios. It also publicly confirmed the worst-kept secret in AI: that GPT-4 itself was likely a massive MoE model.

What to Read Next