Microsoft Trained a 2.8 Trillion Parameter Model. The Interesting Part Is How.

Most AI announcements follow the same pattern: a new model, a benchmark chart, a price. On October 15, Microsoft AI published something different. Not a product, but a lab note: Frontier Pre-training at Microsoft AI, a detailed look at how it trained its newest foundation model, MAI-Base-2.

It's a technical read, full of scaling laws and GPU kernels, and Microsoft also provides the full lab note as a PDF. But under the math are a few ideas every developer can use. So here is my plain-language version.

As always, a note first: I work at Microsoft. The numbers in this post are Microsoft's own, from the lab note. This post explains those reported results; it is not independent validation of them.

First: what a base model is

Training a large model happens in two big phases.

Pre-training is school. The model reads an enormous amount of text, code and images, and learns to predict what comes next. The result is a base model: very knowledgeable, but not yet good at following instructions or having a conversation.

Post-training is job training. The base model is taught to follow instructions, reason step by step, use tools and behave safely. That's what turns it into something you can actually use.

In June, Microsoft announced MAI-Thinking-1, which was post-trained on top of an earlier base model, MAI-Base-1. MAI-Base-2 is the next foundation. Pre-training is finished and post-training is happening now. This lab note is not a product launch and gives no public availability date. This post is about how the foundation was built.

The numbers

MAI-Base-2 is a mixture-of-experts model, or MoE: it has 2.8 trillion parameters in total, but only 80 billion are active for each piece of text. Think of a restaurant kitchen: a huge kitchen, but only the chefs you need for each dish.

Microsoft trained it on 16,000 NVIDIA GB200 GPUs over 38 days in August and September 2026. On its own held-out evaluations, meaning sets reserved for evaluation and checked for overlap with the training data, Microsoft reports lower bits per byte than every open-weight model it tested. That's a measure of how well a model predicts unseen text, not a chatbot or agent benchmark. The four evaluation areas were code, Q&A, math and STEM, and the comparison used checkpoints available before October 2026.

The number I find more interesting is about efficiency. Using fitted scaling laws, Microsoft estimates its recipe is 4.2 to 12.3 times more pre-training FLOPs-efficient than DeepSeek-V4-Pro-Base, which it identifies as the strongest open-weight base model in its comparison. In plain words: Microsoft estimates its recipe could match that reference model's scores on these evaluations with a fraction of the total training computation. That is not a measured reduction in elapsed time or money. Microsoft used the efficiency gains to train a bigger model, and MAI-Base-2 actually used more total training compute than the DeepSeek baseline.

It also compares the new recipe with MAI-Base-1, from June. The estimated improvement is 6.3 to 12.7 times in pre-training FLOPs efficiency on those evaluations. That's a substantial recipe improvement in four months, even before we see what the post-trained model can do.

Why should a developer care about training efficiency? My take is that a recipe that does more with less compute could improve the economics of future models. But training efficiency alone does not establish how fast a deployed model will run or what its API will cost. The lab note gives neither inference measurements nor pricing.

Think big, start small

Here is the part that made me smile. You can't afford to run a 2.8 trillion parameter experiment a hundred times to see what works. So how do you make good decisions for a model that size?

Microsoft uses what it calls a scaling ladder: a family of small models built with the same recipe, each one a step bigger than the last. You test a change on the small rungs, watch how the results change as the models grow, and predict how it will behave at full size. The lab note shows the final model landing close to where the ladder predicted.

There's a clever detail. If a short experiment uses the same pool of data as the full run, it repeats that data less often. That can make small experiments lie: a setting that looks great small can fall apart at scale. So Microsoft artificially limits the data on the small rungs, so each source gets repeated just as often as in the big run. The small test is made to feel like the real thing.

My motto for community building has always been think big, start small, grow fast. This is the same idea at 2.8 trillion parameters, with one lesson for anyone who runs experiments, in code, product or community: a small test only helps if it reflects the real conditions. Otherwise it tells you a story that isn't true.

The model that found a bug

My favorite detail in the whole lab note is small.

Training a huge model can become unstable when a few strange examples produce extreme updates. Microsoft added a technique called activation gradient clipping to limit the effect of those outliers inside the model, rather than relying only on scaling down the whole batch. It retained global weight-gradient clipping too. Microsoft reports that activation gradient clipping made optimization more stable, and its efficiency benefit grew as the models got bigger.

But it also turned out to be a detector. When the system clipped an unusual example, the team could trace it back to the training data. In one case it pointed to a pull request in the code data where a license header was cut off in the middle of a sentence. That led them to a bug in the data processing pipeline, which they fixed.

I love that. The stability mechanism became a monitoring tool. It's a good reminder for anyone running production systems, AI or not: the signals that keep your system stable often tell you where your data is broken.

Where the data comes from

This is the part enterprises will read most carefully.

Microsoft says MAI-Base-2's pre-training uses organic data, with no distillation from other models. Here, distillation means training a model on the outputs of another model. Microsoft reports that it processed every training token in-house, that its web crawler respects robots.txt and related page-level controls, and that it excludes sources that violate its Responsible AI policies or appear on the US government's Notorious Markets list, which names marketplaces associated with piracy and counterfeiting.

A few other figures Microsoft reports: the unique training data grew by about 36% compared with MAI-Base-1, the knowledge cutoff is March 2026, and unlike its predecessor, MAI-Base-2 learned from images from the start. Its vision-language corpus contains 4.5 trillion unique tokens, spanning tasks from chart reading and computer use to sheet music. With repetition, vision-language data accounted for 18% of the 50-trillion-token training mixture: about 9 trillion token presentations, not 4.5 trillion.

Why does provenance matter? Because more and more companies ask where a model came from before they build on it. Knowing what went in is becoming as important as knowing what comes out.

What Microsoft says about the weights

One more line from the lab note is worth knowing. Microsoft writes that its strong data provenance will allow it to "license our model weights to trusted partners through Foundry", its platform for building AI apps. The lab note doesn't go into when or how, so for the details, keep an eye on Microsoft's own announcements.

What to keep in mind

Three things before anyone draws big conclusions.

These are Microsoft's own evaluations, not independent validation. The code evaluation uses Microsoft's own codebase; the STEM, math and Q&A sets were commissioned from outside vendors to reduce contamination risk. Microsoft also says it checked all evaluation sets against its training data and removed overlaps. Those precautions matter, but commissioning test data externally does not make the model evaluation independent.

Some comparisons are against models that were already post-trained, because only post-trained checkpoints were available for those models in the comparison. Microsoft flags this itself: post-training can change those scores.

And a base model is not a product. What matters to developers is the model that comes out of post-training, how it behaves in your own work, and what it costs.

Why I like this kind of publication

What stands out to me is how much Microsoft shared, including operational details that are easy to leave out. It reports average model FLOPs utilization of 21%, measured against the GB200's theoretical BF16 throughput and excluding recomputation and optimizer FLOPs. That is a specific training-compute metric, not a claim that the GPUs were idle 79% of the time. It also reports 89% goodput, the fraction of time training progressed. The note does not break down where the remaining time went. Those details make the publication more useful to me.

The lab note ends with three principles: intelligence should be learned from first principles, simple methods scale best, and scientific rigor over shortcuts. You don't have to train trillion-parameter models to use those. They work for building software, and for building communities too.

Preparations for MAI-Base-3 are already underway, so this won't be the last lab note. I'll keep translating them.

Read the original

For Microsoft's full explanation, charts and methodology, read Frontier Pre-training at Microsoft AI or download the complete lab note (PDF).