How We Taught an Open-Weight Model Deel’s Knowledge of 150+ Countries

Deel AI team, Ömer Gözüaçık, Anthony Li, Abhijit Mehta

This 4-part series introduces how we taught and benchmarked Pangaea K1, our new 9B model.

Pangaea, Part 1: Teaching a model what Deel knows

Almost every AI product in the industry today, ours included, is built on rented intelligence: a frontier model from a large lab, called over an API, with the company's knowledge and tools arranged around it. This approach, retrieval-augmented generation (RAG), is the industry default for good reasons. But it is slow, and every answer starts from zero.

Renting has a structural limit. A rented model knows nothing about your world until you show it, on every single request. Everything Deel understands about employment, payroll and compliance across 150+ countries has to be fetched, read and re-explained to the model, question after question, forever. For general intelligence, renting is the right trade. For the domain that is our business, we think the expertise should live inside the model: trained in, versioned, and improved the way we improve our product code.

Nobody at Deel knows everything. But for almost any question, somebody does, and over the years they have written it down: our internal knowledge base now holds tens of thousands of pages of policies, procedures and how-to guides. Experienced colleagues carry a working picture of most of it in their heads, and know where to look for the rest.

Most AI assistants work like a new joiner on their first day: they search for everything. They query an index, read the pages that come back, and answer from those. Pangaea is our attempt to give an assistant what the experienced colleague has. It is a language model that has studied Deel's knowledge base until it can answer most questions from memory. The goal is not a model that never looks anything up. It is one that knows Deel's general facts by heart and uses a light retrieval step for what memory does badly: checking specifics, catching recent changes and citing sources.

Memory is fast

Retrieval is hard to beat on accuracy, and published head-to-head studies confirm it. What it costs is time. A retrieval pipeline searches, reads, often searches again, and only then writes. In our latest evaluation run, eight models with search tools over our knowledge base took a median of 13 to 28 seconds per question. Pangaea, answering from memory, took under 8 seconds. Since nothing has to be searched before the first word, answers in the product begin appearing within 1 second of asking. Throughout this series, the times we quote are medians for the complete answer in our evaluation harness; time-to-first-word in production is a different, much smaller number.

Memory alone has limits: it can be confidently wrong, it cannot show its sources, and it goes stale. Those are exactly the things retrieval is good at, which is why we see the two working together. First, though, we had to find out how much of Deel's knowledge a model could actually hold.

A lot to know, and it keeps changing

The knowledge base holds about 43,500 pages, around 69 million tokens, close to double its size a year ago. And it moves: since August 2026, about 15% of pages change every month.

So we needed a model big enough to hold all of it, and a training pipeline cheap enough to rerun often.

Exposure matters more than size

We started from a 9-billion-parameter open-weight model. Research on how much a model can store per parameter suggests that is plenty for a corpus of our size. The real constraint is exposure: a model learns a fact by seeing it many times, in many different forms, not by seeing it once.

Our first training corpus missed this. It gave every chunk of every page the same small bundle of rewrites, so long chunks, which held 81% of the text, were barely rewritten at all. On average the model saw each fact only four or five times. A 2023 study from Meta and MBZUAI (Physics of Language Models, Part 3.1) explains why that fails: without varied restatements, a model can memorise text and still score 0% on questions about it.

One fact, many views

So we rebuilt the corpus around one idea: every fact should reach the model many times, from many angles.

We call each restatement a view. Every chunk now gets around a dozen views of different types: written for different readers, focused on numbers, stated in reverse, summarised by topic, and linked to related pages. Each view is generated separately by a frontier LLM and becomes its own training example, shuffled across the whole corpus.

We also mix question-and-answer pairs into this stage. Recent research shows models absorb new documents better when they also see how that knowledge will be asked about.

Generating views at this scale taught us one lesson above all: read the output. Generator models differed far more than we expected, and every real problem we found showed up in samples, not in dashboards.

The study pack

The final corpus is about 1.2 billion tokens, roughly 18 times the size of the knowledge base itself.

Most of it is views. The original pages are included word for word, plus the question-and-answer pairs and a small share of general text, which we come back to below. When pages change, we regenerate views only for the chunks that changed; our last rebuild reused about two-thirds of the existing views.

Not forgetting what it already knew

Training hard on one narrow corpus risks catastrophic forgetting: as the weights adjust to the new material, the model can lose skills it had before, such as following instructions or writing clearly about anything outside our knowledge base. Updating every weight, as we do, raises that risk. The standard remedy is replay, mixing a share of general-purpose data back into training so the model keeps practising what it already knew. Research on continued pretraining finds that even 1% measurably reduces forgetting.

Replay makes up about 7.5% of our corpus, mostly educational web text plus some textbook-style writing. Every extra point spends training time on text with no Deel facts in it, so we keep the share small. It is a safeguard rather than a guarantee, so we treat general skills as something to measure, not assume. Fine-tuning gets the same treatment, which Part 2 covers.

Training: more views, not more passes

This stage is called continued pretraining (CPT): the model keeps predicting the next token, on our corpus instead of the open web. We update every weight (full fine-tuning) rather than using a lightweight adapter such as LoRA, which research shows learns substantially less from pretraining-scale data.

Loss flattened after two passes over the corpus, and continuing into a third added nothing. One detail matters here: the usual practice of validating on unseen documents works against you, because memorising these documents is the goal. We validate on unseen views of the documents the model is studying.

Where this left us

After this stage alone, Pangaea's score on our evaluation set rose from about 19 for the base model to about 54. The model now knew a large share of what our knowledge base says.

However, knowing isn't the same as answering well. A model that has only studied writes like a textbook. Part 2 covers how we taught it to answer like a colleague, which took the score to 70.

Pangaea is the work of Deel's AI team, with support from everyone across Deel who writes and maintains the knowledge it learned from.

Pangaea, Part 2: From knowing to answering

In Part 1, we taught a 9B model what Deel's knowledge base says. After that stage it scored about 54 on our evaluation set, up from about 20. But a model that has only studied writes like a textbook: it drifts, it stops mid-thought, and it does not shape its answer to the question.

This part is about teaching it to answer like a colleague. That step took the score to 70.

Learning from a teacher

For supervised fine-tuning (SFT), a stronger teacher model writes the answer a good colleague would give. It sees the relevant Deel knowledge, plus prompts and example answers curated by specialists. Pangaea is then trained to give the same answer without seeing any of that.

That one asymmetry drives most of the design. The teacher reads; Pangaea has to recall. So every training answer has to be something Pangaea could plausibly give from memory: grounded in what the page says, no longer than the page can support, and phrased as knowledge rather than as “the passage says…”.

Not every question–answer pair goes to fine-tuning. The Deel-specific part of the SFT set holds only these teacher-written answers. The remaining pairs go into the study material from Part 1 instead, mixed in among the views, so the model sees how its knowledge will be asked about while it is still learning it. That keeps the SFT stage a thin layer: it teaches behaviour, not facts.

Facts in study, behaviour in practice

Keeping new facts out of fine-tuning is deliberate. A 2024 study from Google found that when fine-tuning examples contain facts a model does not already know, the model learns them slowly, and once it does, it becomes more likely to hallucinate. Their conclusion matches ours: models get knowledge from pretraining, and fine-tuning teaches them how to use it.

So the split is simple. Continued pretraining puts the facts in. Fine-tuning teaches the model what a good answer looks like.

Answers the right length

The first fine-tuning data produced answers that were far too short. Our evaluation questions need about three facts on average, but a typical training answer was about two dozen words long.

We introduced length bands: short answers of 25–50 words, medium of 60–110 and long of 130–220. Each answer's band is capped by how much its source page can actually support, so a one-line policy never turns into a paragraph of filler.

The typical training answer went from 24 words to 78. We also learned that “more human” meant more specific and better organised, not warmer. Friendly filler was never the problem.

We erred on the long side on purpose. Covering every necessary fact mattered more than brevity at this stage, and we come back to trimming in Part 4.

The examples teach the format

When an answer is a set of steps or a list of options, readers want a list. Our instructions to the teacher already said so, yet only about a third of list-shaped answers in the training set were formatted as lists.

The cause was the examples. Every example answer in the teacher's prompt was written in prose, and the teacher copied the examples rather than the instructions. Once we added list-shaped examples, the share of list-formatted answers in the training set rose sharply on pages that call for one.

It is a small fix with a general lesson: when you prompt a model to generate training data, its examples are your real specification.

Keeping answers clean

Longer answers brought one side effect: the teacher began including more bare URLs and contact details. Pangaea learns whatever appears in its answers, so we added a rule that answers must be grounded in the page's content and never invent links or addresses. In the training set, both dropped to zero.

Staying a general assistant

Fine-tuning on a narrow set of answers carries the same forgetting risk as continued pretraining (Part 1), so it gets the same remedy: replay. Alongside the teacher-written Deel answers, the SFT set includes a small, curated sample of everyday tasks that have nothing to do with Deel: summarising, classifying, brainstorming, answering open questions. It is kept small on purpose. Its job is to remind the model that it is still a general assistant, not to teach it anything new.

The result

After fine-tuning, Pangaea's score rose from about 54 to 70. More importantly, its answers read like answers: shaped to the question, organised, and complete.

Part 3 explains what that 70 means, how we measure it, and how it compares with models that can search our knowledge base.

Next: reasoning and tool calls

Today, the fine-tuning data teaches plain answers. Pangaea does not yet reason before it answers, and it does not call tools. Both are in the pipeline, and both follow the same distillation recipe: a teacher shows the behaviour, and Pangaea learns to reproduce it. For reasoning, one rule carries over from everything above: a trace should sound like recall, not reading. A trace that says “according to the document” would teach Pangaea to imagine a document it will never have. Tool calls are what will connect Pangaea to the light retrieval step in Part 4.

Pangaea, Part 3: How do you know it knows?

In Part 2, fine-tuning taught Pangaea to answer like a colleague, and its score rose to 70. This part explains what that number means and how we measure it.

The challenge is that a fluent answer and a correct answer look the same. A model that has studied a knowledge base can produce confident, well-organised text that is subtly wrong. We needed a measure that checks every fact, not the overall impression.

Scoring facts, not impressions

Our evaluation set has 601 questions, written to match the kinds of questions our teams ask every day. Each comes with a reference answer, which we break into individual facts, about three per question on average. Each fact is marked as necessary (the answer is wrong without it) or supporting (useful, but optional).

A judge model then reads Pangaea's answer and checks each fact in turn: is it there, and is it stated correctly? A question's score is the share of its facts that the answer gets right.

This takes more work than asking a judge to grade an answer out of ten, but it pays off. A low score points at the exact facts that were missed, which tells us where to look in the training data.

The headline numbers

Each stage of training moved the score substantially. The base model, before any training on Deel's knowledge, scored 19. After continued pretraining (Part 1) it reached about 54, and after fine-tuning (Part 2) 70.

For comparison, we ran the same 601 questions through eight other models: Claude Opus 5.5 and Sonnet 5.5 and Sonnet 5, GPT-6 Sol and Luna, Gemini 3.8 Flash, DeepSeek V4.1 Flash and GLM 5.3 Flash. Each had search tools over our knowledge base that it could call while answering. Their scores ranged from 52 to 75. Pangaea, answering from memory alone, scored 70: ahead of Gemini 3.8 Flash, Claude Sonnet 5 and both GPT-6 models, within five points of the best, and faster than all of them. Its median answer took 7.7 seconds, against 14.8 for the best-scoring model, DeepSeek V4.1 Flash, and 28.4 for Claude Opus 5.5.

The base 9B model makes the same point from the other side. Given the same search tools, it reached 45 on the questions it was scored on. Training took that model to 70 without searching at all.

Table 1 · All systems on the same 601 questions, sorted by score. “Necessary facts only” scores just the facts an answer is wrong without. Bold: the best value in each column, and Pangaea.

An earlier run of the same evaluation compared Pangaea question by question with one frontier model using search. The two were closer than their averages suggest: they tied on about half the questions, and Pangaea won outright on about one in five.

In that run, Pangaea answered 48% of questions perfectly, against 52% for the model with search. It scored zero on 17%, against 14%.

We read these results as validation of the approach, not as a finish line. Answering from memory gets most of the way there, quickly. Retrieval still adds the last few points, which is why our goal remains less search, not no search.

Facts that move, facts that stay

Not every fact ages at the same speed. A holiday calendar is published once and then stays put; a fee schedule may be edited every few weeks. A model that is retrained periodically can hold the first kind in memory, but not reliably the second.

So we tag every fact in the evaluation as stable or volatile, based on what the page's edit history shows rather than on what the fact looks like. For each page the evaluation draws on, we collect the visible-text edits from the last 90 days. A judge model then checks, fact by fact, whether any edit changed that fact's value: a rate, a date, a provider, a yes or no. Rewording, moving or reformatting a fact does not count. Pages that keep revising facts faster than our retraining cycle are flagged as a whole.

This gives us two things. We can score volatile facts separately, so a stale answer is never mistaken for a learning gap. And it draws the line for the end goal in Part 4: stable facts belong in memory, and volatile ones are what a light retrieval step should check.

From a score to a fix

Fact-level scores make problems traceable. For example, questions about pages that shared text with other pages scored noticeably lower. Following the misses back to the training data showed why: the shared text had been kept for only one of the pages, so the others were missing part of their content. The next corpus build fixes this.

Checking the checker

The judge is a model too, and it can be wrong. When we compared judge models on the same answers, they sometimes ranked candidates very differently, and one was rewarding answers that sounded plausible without checking the facts. We now read samples of the judge's decisions, not only its averages.

We also keep an eye on what the score rewards. A reference answer can include facts the question never asked for, and a long answer is more likely to hit them by accident. That is one reason we score necessary facts separately, and one input into the work in Part 4.

Who checks the facts?

Every score above rests on two things: the judge that reads each answer, and the reference facts it checks that answer against. Reading samples of the judge's decisions tells us when the judge gets an answer wrong. It cannot tell us when a reference fact is wrong, missing or out of date. The judge will score every answer against that fact, consistently, and the error will look like signal.

The reference facts are written by a model, which keeps the evaluation fast to build and cheap to refresh. It also leaves a gap a model is poorly placed to fill. Every fact describes something a correct answer should say, and none describes something it must not say, so the evaluation never tests whether a model states a plausible falsehood. Writing a convincing one is hard for a model and easy for a specialist.

So Emet’s in-country specialists, who answer these questions every day, are now checking the evaluation itself. They verify each fact against the law and Deel practice, with a source, and correct it where it is wrong. They add the facts that are missing, along with false statements that a correct answer must reject. And on a sample of answers they make the same calls as the judge, so we can see how far to trust it. Studies on general chat questions find that strong model judges agree with people about as often as people agree with each other. We are evaluating whether that holds for employment rules across more than 150 countries as well.

In an earlier evaluation round, in which specialists graded Pangaea's answers directly, we found specialists’ time is best spent explaining their reasoning: what was wrong and, often, what the right answer is.

Pangaea, Part 4: Memory first, search when it matters

Part 4 of 4

In Part 3, we saw that Pangaea answering from memory comes within five points of the best model with search tools, and answers faster than any of them. This final part is about closing that gap, and about where Pangaea is headed.

The end goal

From the start, our aim has been less search, not no search. A colleague who knows Deel well answers most questions from memory, and checks the details that matter. We want the same: a model that knows Deel's general facts by heart, and uses a light retrieval step to confirm specifics, pick up recent changes and cite its sources.

We are not alone in this direction. Harvey recently reported that, on a benchmark built to stand in for a law firm's knowledge base, a model trained to hold that knowledge made more targeted searches and needed less than half the tokens to complete its tasks, compared with searching from scratch each time.

Two directions we are exploring

We are exploring two ways to add that light retrieval step. Nothing is decided yet.

• Look things up when needed. The model decides for itself when a question needs a search, and answers the rest from memory.

• Answer, then verify. The model answers from memory, and the answer is checked against search before it reaches the user, with sources attached.

Both depend on one skill Pangaea does not have yet: knowing when it does not know. Today it always answers. Teaching it to recognise uncertainty, and to search or say so instead of guessing, is central to everything that follows.

Reinforcement learning: two tracks, one path

We have recently started applying reinforcement learning (RL) to Pangaea. RL does not add knowledge; the facts still come from continued pretraining. Instead, it shapes how the model uses what it already knows. The work is early, and we have no results to share yet. It runs on two tracks.

Track 1: the most necessary content in the least length. In Part 2, we deliberately made answers longer so they would cover every necessary fact. Pangaea now tends to say a little too much. This track rewards answers that contain the necessary facts with less repetition and fewer words. The reward counts only the necessary facts, so the model is never encouraged to drop one to save space.

Track 2: fewer tool calls, better citations. Once tool calling is distilled in (Part 2), this track teaches the model when a search is worth making, and to cite the pages its answer relies on. A citation earns credit only if the cited page actually states the fact, so the model cannot score by producing a plausible-looking link.

Over time, we plan to merge the two tracks into one path: answer from memory when confident, search when not, and cite in either case. Knowing when it does not know is what joins them.

Keeping up with change

About 15% of the knowledge base changes every month, so a model that answers from memory has to keep learning. The incremental pipeline from Part 1, which regenerates study material only for pages that changed, makes frequent refreshes practical. A light retrieval step then covers what memory cannot keep up with: the volatile facts from Part 3, whose pages revise them faster than we retrain.

One model today, a family by design

Pangaea K1 is the first Pangaea model. Pangaea P1, for payroll, launches alongside it, and more are planned. Everything this series described (the view generation, the study pack, the teacher distillation, the fact-level evaluation) is a pipeline, not a one-off. Point it at a different body of Deel expertise and it produces a different expert. Knowledge came first because it was the clearest test: a large, written, well-maintained corpus and a question type our teams face thousands of times a day. The domains we operate in go deeper than what is written down( payroll runs, compliance workflows, the patterns inside our platform's data ) and that is the direction the family grows. Each future model will carry its own letter, be measured the same fact-level way, and earn its place the way K1 did: against the best available alternative, in the open.

And the same principle holds for all of them: frontier models stay in the flow for what they do best. Pangaea models take over the work where owning the expertise wins.

What we would tell a team starting out

• Exposure beats size. A model learns a fact from many varied restatements, not from a bigger parameter count or more passes over the same text.

• Put facts in pretraining, behaviour in fine-tuning. Each stage does one job well.

• Measure facts, not impressions. Fact-level scoring turns a vague “it seems worse” into a specific fix.

• Read the samples. Almost every real problem we found was visible in raw output long before it showed up in a metric.

Pangaea started as a question: can a model hold what an organisation knows? For Deel's knowledge base, the answer so far is yes.

Pangaea is the work of Deel's AI team, with support from everyone across Deel who writes and maintains the knowledge it learned from, and from the Deel specialists who check its evaluation through Emet.