Distilling hamo-score-0.6b: A Plateau, Three Bugs, and Why Data Beat Model Size

AI for Psychology, Psychology for AI.

These are the methodology notes behind hamo-score-0.6b, our first open-weights release. A 0.6B student learned to score psychological state from a cloud API's judgments through a three-stage teaching chain, got stuck on a plateau that more data would not shift, and broke through only when we stopped adding data blindly and started auditing it. A model three times larger gained one point. A data audit plus a recipe fix gained three.

When we published the weights, we said the point was that a component like this can be small, local, and auditable. Auditable means showing the work — including the parts where the work went badly. These are those notes.

The constraint that shaped everything

Before any architecture decision, there was a privacy rule: real client conversations never enter the training set.

That is more restrictive than it sounds, because the obvious way to distil a scorer is to take the messages your production system already scored and train on them directly. We couldn't. So real production data was allowed exactly two jobs:

It was the exam. 1,198 pseudonymised real turns, each carrying the score the production system had historically assigned — split into a 440-turn calibration set and a 758-turn final exam that nothing was ever trained on.

It was the distribution reference. Not the content — the shape. How long real messages actually are, how fragmentary, how much they code-switch. This job sounds minor and isn't. Synthetic training data written without a distribution reference drifts toward articulate, well-formed, novel-length paragraphs. Nobody in distress types like that. A model trained on tidy prose learns to score tidy prose.

Everything the student actually learned from was synthetic. [Update, 2026-08: true of the v4 release these notes describe. Since v6.1 there is one disclosed exception — 440 consented internal-staff turns, ~8% of the corpus. External client conversations still never enter training.]

The teaching chain

Three stages, each with a gate.

The answer key. Production scores had already accumulated in a database. They cost nothing further to use and represented the actual judgments the live system had been making.

The teacher had to pass an exam first. An open model — DeepSeek-chat — was given the production scoring rubric and made to sit a qualification test on a several-hundred-item calibration paper. On the rubric alone it averaged about 87% across the five dimensions, with the Boundary dimension at 72%. We added a calibration note on Boundary that production never had, and on the same paper it reached 88.7% agreement at the dimension level, just under our 90% line, and passed conditionally. Only then was it allowed to label anything. Worked out a day later, its agreement on the final decision those dimensions feed into was 97.5%.

Why route through an intermediate teacher at all, instead of learning directly from the incumbent scorer's outputs? Two reasons. Most frontier terms of service prohibit using outputs to train a competing model. And we wanted a lineage reproducible end to end — someone with no commercial API key should be able to run this and get a comparable result. A distillation chain nobody can rerun is a claim, not a method.

The student. Qwen3-0.6B, fine-tuned with LoRA on teacher-graded synthetic dialogue windows — roughly 20,000 admitted across six data generations, about 15,000 of them in the v4 release these notes describe. Training ran on a MacBook, about an hour a round at first and about two hours by v4. The scoring rubric — about 1,200 tokens of it — went from being pasted into every request to being internalised in the weights. That is the entire economic argument for distillation in one sentence: you stop paying to re-read the instructions on every call.

[Update, 2026-11: in v10h, the default on Hugging Face since 2026-10-31, human counselors engaged by Hamo reviewed and re-scored the synthetic training labels. Details on the model card.]

The curriculum was not written freely

Synthetic data generated without constraints produces a corpus that is fluent, plausible, and useless. Ours was generated against a grid of 40+ scenario cells with explicit quotas, and each cell carried a target:

• Neutral small talk must score zero across all five dimensions.
• Giving up and avoidance must score high on Withdrawal.
• Resistance to the AI itself appears at five graded intensities.
• A one-word reply like "whatever" must only be interpretable in context.

Then every generated example passed through an admission check. If the teacher's score on a generated sample fell outside that cell's design target, the sample was discarded, not patched. Patching a mislabelled example teaches the boundary in the wrong place; throwing it away only costs you generation budget, which is the cheap resource.

The plateau

The version table on the v4 model card tells the story flatly: the first distillation reached 81% dimension agreement. The generation after it — a deliberate rebalance-and-defect-repair pass — also reached 81%. Real work, real cost, no movement. And each generation costs a synthesis run plus a training run.

The instinct at that point is to reach for a bigger student. We did try that, and we'll get to it.

What we did first was audit the model's outputs rather than its scores. The diagnosis, once seen, was hard to unsee:

The student had collapsed into a grid classifier. It was emitting only 59 distinct score combinations. Real conversations, scored by the reference system, produce 233. Whatever came in, the model snapped it to the nearest combination it had memorised. It looked like a scoring model and behaved like a lookup table — and every accuracy metric we were tracking hid this completely, because snapping to a nearby grid point is usually within ±0.5 of the truth.

And the mid-band was evaporating down the chain. We measured how often each link in the teaching chain assigned a middling score, one of the 1.0–1.5 values that mean "somewhat, but not markedly":

Each teacher passed on fewer mid-band judgments than it had received. By the time it reached the student, ambivalence had almost entirely drained out — and one particular value, Boundary at 1.0, had never once been emitted. The student had learned that the world is confident, when most of a therapy conversation is not. (The v4 release recovered to 7.4%, which the model card then listed as a known limitation rather than a solved problem — still well under the reference band.)

The fix was mostly, but not entirely, on the data side

The generation that broke the plateau kept the same base model and the same size. It changed the curriculum:

• ~240 mislabelled examples repaired.
• 35 ambiguous samples with safety implications pulled rather than relabelled — where the correct score is genuinely arguable and the topic is risk, the honest move is to remove the example, not to force a decision the humans disagreed on.
• 5,350 counter-grid examples added, deliberately targeting exactly what the model had learned to avoid: very short messages, mid-band intensities, mixed emotions, and — the hardest category — heavy signals stated calmly.

Results: dimension agreement broke the plateau at 84.0%. And Boundary 1.0 was emitted for the first time.

One honest confound. That same release also changed the training recipe, including switching on prompt-masked loss — the configuration bug described below — so the three points from 81% to 84.0% are not attributable to the curriculum alone. We never measured a clean split; that most of the gain came from the data is our reading, not a measurement. The closest evidence is an earlier run on the old data that changed only the recipe: on the calibration set, dimension agreement went from 81% to 82% and decision agreement from 96.1% to 97.5%, both inside the two-point margin our plan treated as noise. That run never sat the final exam. We'd rather report a muddied attribution than a clean one we can't defend.

The bigger-model experiment. We also trained a student roughly three times the size. It gained about one point. The data audit and the recipe changes together gained three, plus the diversity recovery. For anyone distilling a small task-specific model: the bottleneck is almost always the curriculum and the recipe, not the student's capacity. Buying parameters is the expensive way to not fix your data.

Three bugs, three disciplines

Each of these cost real time. Each produced a rule we now apply without exception.

The 72% tuition problem. For several successive generations, mask_prompt was never enabled in the training config. The student was spending roughly 72% of its learning capacity memorising how to reproduce the client's message verbatim, and only 28% on the thing it was actually being graded on. The first run with mask_prompt on gained about 1.4 points of decision agreement on the calibration set, but that run also changed the learning-rate schedule and cut the steps in half, so we can't credit the flag alone. The step cut, not the flag, is what halved training time.

Discipline one: audit the training recipe before adding data. The cheapest improvement available is usually one line of configuration.

The silent memory blowout. Doubling the context length without halving the batch size overran 16GB of memory. Loss went from 0.1 to 22 and three hours of training produced garbage. The retrain added a loss sentinel: check every minute, alarm immediately on anomaly.

Discipline two: when max sequence length doubles, batch size halves. Any training run you care about gets a watchdog — never "find out when it finishes."

The silent thread deaths. During the shadow-mode rollout, a function signature changed and one thread's parameters weren't updated with it. Every scoring thread crashed on startup. The exception was swallowed by a guardian thread. The system reported healthy, the dashboards were green, and the comparison table received zero rows. An entire test session's data was lost silently.

Discipline three: a side-channel system that isn't erroring is not the same as a system that's working. Every deployment gets verified all the way to "the rows actually landed in the table."

That third one is the most generalisable, and the most uncomfortable. A shadow-mode evaluation pipeline is exactly the kind of system nobody watches, because by design it affects nothing. Which is why it can fail for a full test cycle without a single alert firing.

Shipping it: four steps, four exit conditions

The report card supported a switch. The plan was not to switch all at once: each step had a predefined exit condition, agreed before the step began, so that "it looks fine" could not be the reason to advance. [Update, 2026-11: this table is the plan as of this post, not a record of what was done.]

One thing here is easy to misread, so it needs saying plainly: step 3's thresholds are measured on the 500+ live turns of the dev rotation, which is not the same paper as the report card earlier in this post — that comes from the 758-turn final exam (84.0% dimension-level). The same model reads differently on two different sets, which is normal, but reading them as one produces the false conclusion that we set a bar at 85% and shipped at 84%.

Step 2 is the one worth arguing for. A fallback path is unglamorous and it is the reason the other three steps are allowed to exist. Without it, every subsequent step is a bet.

Crisis handling runs upstream of scoring and independent of it, in the spine, the layer above the model, so the scorer swap should not be able to affect it at all. It is not this model's job: none of the model's scores, Withdrawal included, is a crisis signal. What the spine's keyword pre-screen misses falls to its other layers and to the responsible clinician.

[Update, 2026-11: scoring can run entirely on your own machine when you run the open model yourself. Hamo's own live service is not there yet: reply generation and the semantic crisis screen run in the cloud, and the cloud scorer this model replaced still runs on every turn for comparison, so messages still leave our machines.]

What generalises

Most of this transfers to any small task-specific distillation, whether or not it has anything to do with psychology:

• Check output diversity, not just agreement. A grid classifier scores well and is broken. Count distinct outputs against what the reference system produces in the wild.
• Expect the middle to evaporate. Each link in a teacher chain sharpens toward confidence. If your task has meaningful mid-band values, measure their survival rate at every hop.
• Discard, don't patch, ambiguous training examples. Especially where the topic carries risk.
• Audit the recipe before scaling the data. We did six full training runs with one loss setting left off before we noticed.
• Buy data before you buy parameters. Three times the model, one point. One good data audit and a recipe fix, three points.

Getting to v4, including the failed bigger-model experiment, cost about six and a half dollars of API budget and a laptop (later relabelling and audits cost more). That number is the real argument for this shape of system. A component that reads psychological state does not require a frontier lab's resources to build, which means it does not have to be owned by one.

“The most useful thing in this whole write-up is the plateau — generation after generation where we did the obvious thing, added more data, and got nothing. The answer wasn't a bigger model or a better algorithm. It was that we had never looked at what the model was actually emitting. We publish that because everyone distilling a small model will hit the same wall, and the cost of finding out this way was two weeks we'd rather other people didn't spend.”
— Chris Cheng, Founder and CEO of Hamo AI

---

Model and evaluation method: HamoAI/hamo-score-0.6b · Scoring basis: 758 pseudonymised real turns held out from all training

Grounded in code, not slideware.

Hamo AI — making minds aware, and awake.

---

[Corrected 2026-11: the length of the production scoring record, how the teacher qualified, how generated examples were screened, the fallback time limit, how long training rounds took, the API cost and what the config fix did.]