Distilling hamo-score-0.6b: A Plateau, Three Bugs, and Why Data Beat Model Size

AI for Inner Explorers.

These are the methodology notes behind hamo-score-0.6b, our first open-weights release. A 0.6B student learned to score psychological state from a cloud API's judgments through a three-stage teaching chain, got stuck on a plateau that more data would not shift, and broke through only when we stopped adding data and started auditing it. A model three times larger gained one point. A data audit plus a one-line config fix gained three.

When we published the weights, we said the point was that a component like this can be small, local, and auditable. Auditable means showing the work — including the parts where the work went badly. These are those notes.

The constraint that shaped everything

Before any architecture decision, there was a privacy rule: real client conversations never enter the training set.

That is more restrictive than it sounds, because the obvious way to distil a scorer is to take the messages your production system already scored and train on them directly. We couldn't. So real production data was allowed exactly two jobs:

It was the exam. 1,198 desensitised real turns, each carrying the score the production system had historically assigned — split into a 440-turn calibration set and a 758-turn final exam that nothing was ever trained on.

It was the distribution reference. Not the content — the shape. How long real messages actually are, how fragmentary, how much they code-switch. This job sounds minor and isn't. Synthetic training data written without a distribution reference drifts toward articulate, well-formed, novel-length paragraphs. Nobody in distress types like that. A model trained on tidy prose learns to score tidy prose.

Everything the student actually learned from was synthetic.

The teaching chain

Three stages, each with a gate.

The answer key. Two years of production scoring had already accumulated in a database. It cost nothing further to use and represented the actual judgments the live system had been making.

The teacher had to pass an exam first. An open model — DeepSeek-chat — was given the production scoring rubric verbatim and made to sit a qualification test on the 440-turn calibration set. It reached 89% agreement at the dimension level — and 97.5% on the final decision those dimensions feed into. Only then was it allowed to label anything.

Why route through an intermediate teacher at all, instead of learning directly from the incumbent scorer's outputs? Two reasons. Most frontier terms of service prohibit using outputs to train a competing model. And we wanted a lineage reproducible end to end — someone with no commercial API key should be able to run this and get a comparable result. A distillation chain nobody can rerun is a claim, not a method.

The student. Qwen3-0.6B, fine-tuned with LoRA on teacher-graded synthetic dialogue windows — roughly 20,000 admitted across five data generations, about 15,000 of them in the released version. Training ran on a MacBook in 40-minute rounds. The scoring rubric — about 1,200 tokens of it — went from being pasted into every request to being internalised in the weights. That is the entire economic argument for distillation in one sentence: you stop paying to re-read the instructions on every call.

The curriculum was not written freely

Synthetic data generated without constraints produces a corpus that is fluent, plausible, and useless. Ours was generated against a grid of 40+ scenario cells with explicit quotas, and each cell carried a target:

• Neutral small talk must score zero across all five dimensions.
• Crisis phrasing must max Withdrawal.
• Resistance to the AI itself appears at five graded intensities.
• A two-word reply like "whatever" must only be interpretable in context.

Then every generated example passed through the same admission gate the live system uses. If the teacher's score on a generated sample fell outside that cell's design target, the sample was discarded, not patched. Patching a mislabelled example teaches the boundary in the wrong place; throwing it away only costs you generation budget, which is the cheap resource.

The plateau

The version table on the model card tells the story flatly: the first distillation reached 81% dimension agreement. The generation after it — a deliberate rebalance-and-defect-repair pass — also reached 81%. Real work, real cost, no movement. And each generation costs a synthesis run plus a training run.

The instinct at that point is to reach for a bigger student. We did try that, and we'll get to it.

What we did first was audit the model's outputs rather than its scores. The diagnosis, once seen, was hard to unsee:

The student had collapsed into a grid classifier. It was emitting only 59 distinct score combinations. Real conversations, scored by the reference system, produce 233. Whatever came in, the model snapped it to the nearest combination it had memorised. It looked like a scoring model and behaved like a lookup table — and every accuracy metric we were tracking hid this completely, because snapping to a nearby grid point is usually within ±0.5 of the truth.

And the mid-band was evaporating down the chain. We measured how often each link in the teaching chain assigned a middling score, one of the 1.0–1.5 values that mean "somewhat, but not markedly":

Each teacher passed on fewer mid-band judgments than it had received. By the time it reached the student, ambivalence had almost entirely drained out — and one particular value, Boundary at 1.0, had never once been emitted. The student had learned that the world is confident, when most of a therapy conversation is not. (The released version recovered to 7.4%, which the model card lists as a known limitation rather than a solved problem — still well under the reference band.)

The fix was mostly, but not entirely, on the data side

The generation that broke the plateau kept the same base model and the same size. It changed the curriculum:

• ~240 mislabelled examples repaired.
• 35 ambiguous samples with safety implications pulled rather than relabelled — where the correct score is genuinely arguable and the topic is risk, the honest move is to remove the example, not to force a decision the humans disagreed on.
• 5,350 counter-grid examples added, deliberately targeting exactly what the model had learned to avoid: very short messages, mid-band intensities, mixed emotions, and — the hardest category — heavy signals stated calmly.

Results: dimension agreement broke the plateau at 84.0%. Crisis-phrasing misses fell from 11 to 5. And Boundary 1.0 was emitted for the first time.

One honest confound. That same release also switched on prompt-masked loss — the configuration bug described below — so the three points from 81% to 84.0% are not attributable to the curriculum alone. Measured in isolation, the config fix was worth about 1.4 points, which leaves roughly 1.6 for the data work. We'd rather report a muddied attribution than a clean one we can't defend.

The bigger-model experiment. We also trained a student roughly three times the size. It gained about one point. The data audit and a one-line config change together gained three, plus the safety improvement, plus the diversity recovery. For anyone distilling a small task-specific model: the bottleneck is almost always the curriculum and the recipe, not the student's capacity. Buying parameters is the expensive way to not fix your data.

Three bugs, three disciplines

Each of these cost real time. Each produced a rule we now apply without exception.

The 72% tuition problem. For several successive generations, mask_prompt was never enabled in the training config. The student was spending roughly 72% of its learning capacity memorising how to reproduce the client's message verbatim, and only 28% on the thing it was actually being graded on. Turning it on halved training time and gained 1.4 points on agreement.

Discipline one: audit the training recipe before adding data. The cheapest improvement available is usually one line of configuration.

The silent memory blowout. Doubling the context length without halving the batch size overran 16GB of memory. Loss went from 0.1 to 22 and three hours of training produced garbage. The retrain added a loss sentinel: check every minute, alarm immediately on anomaly.

Discipline two: when max sequence length doubles, batch size halves. Any training run you care about gets a watchdog — never "find out when it finishes."

The silent thread deaths. During the shadow-mode rollout, a function signature changed and one thread's parameters weren't updated with it. Every scoring thread crashed on startup. The exception was swallowed by a guardian thread. The system reported healthy, the dashboards were green, and the comparison table received zero rows. An entire test session's data was lost silently.

Discipline three: a side-channel system that isn't erroring is not the same as a system that's working. Every deployment gets verified all the way to "the rows actually landed in the table."

That third one is the most generalisable, and the most uncomfortable. A shadow-mode evaluation pipeline is exactly the kind of system nobody watches, because by design it affects nothing. Which is why it can fail for a full test cycle without a single alert firing.

Shipping it: four steps, four exit conditions

The report card supported a switch. We didn't switch all at once. Each step had a predefined exit condition, agreed before the step began — so that "it looks fine" could never be the reason to advance:

One thing here is easy to misread, so it needs saying plainly: step 3's thresholds are measured on the 500+ live turns of the dev rotation, which is not the same paper as the report card earlier in this post — that comes from the 758-turn final exam (84.0% dimension-level, 5 crisis-phrasing misses). The same model reads differently on two different sets, which is normal, but reading them as one produces the false conclusion that we set a bar at 85% and shipped at 84%.

Step 2 is the one worth arguing for. A fallback path is unglamorous and it is the reason the other three steps are allowed to exist. Without it, every subsequent step is a bet.

And step 3's crisis condition is not negotiable and not a threshold — zero, meaning zero live misses during the rotation. Crisis handling runs upstream of scoring and independent of it, so the scorer swap should not have been able to affect it at all. We verified that it didn't rather than assuming it. (The 5 crisis-phrasing misses on the final exam are a different thing: that is the weights' own recall, one layer of defence in depth, not the upstream gate.)

What generalises

Most of this transfers to any small task-specific distillation, whether or not it has anything to do with psychology:

• Check output diversity, not just agreement. A grid classifier scores well and is broken. Count distinct outputs against what the reference system produces in the wild.
• Expect the middle to evaporate. Each link in a teacher chain sharpens toward confidence. If your task has meaningful mid-band values, measure their survival rate at every hop.
• Discard, don't patch, ambiguous training examples. Especially where the topic carries risk.
• Audit the recipe before scaling the data. A configuration flag was worth more than several generations of data collection.
• Buy data before you buy parameters. Three times the model, one point. One good data audit, three points.

The whole project — every generation, the failed bigger-model experiment, all of it — cost about seven dollars of API budget and a laptop. That number is the real argument for this shape of system. A component that reads psychological state does not require a frontier lab's resources to build, which means it does not have to be owned by one.

“The most useful thing in this whole write-up is the plateau — generation after generation where we did the obvious thing, added more data, and got nothing. The answer wasn't a bigger model or a better algorithm. It was that we had never looked at what the model was actually emitting. We publish that because everyone distilling a small model will hit the same wall, and the cost of finding out this way was two weeks we'd rather other people didn't spend.”
— Chris Cheng, Founder and CEO of Hamo AI

---

Model and evaluation method: chrischengzh/hamo-score-0.6b · Scoring basis: 758 desensitised real turns held out from all training

Grounded in code, not slideware.

Hamo AI — making minds aware, and awake.

---