Forge
Small tool-calling specialists that an AI agent can train, test and promote on its own, on consumer hardware.
The problem
General models are slow and expensive for the same agent commands run thousands of times. Could an agent specialise small local models safely, and could anyone tell whether it worked?
The approach
A local gateway serves per-domain specialists (FunctionGemma 270M, then Gemma 3 1B, trained with MLX on Apple Silicon) with a general model as fallback. Data preparation, fine-tuning, gated evaluation and promotion are commands an agent can run; a candidate is promoted only if it passes a sealed test, otherwise the previous adapter is restored byte for byte. Evaluated with a pre-registered design on three sealed, seeded simulator worlds.
What stands out
- Specialists answered in a median 1.47 s against 7.88 s for a general model, on 5.9× fewer input tokens per command.
- Correct on 279 of 300 commands against the general model's 290: faster and cheaper, a little less accurate, stated plainly.
- Found that 52–88% of my own suite prompts were verbatim training rows (one adapter: 44/50 on its suite, 1/20 on paraphrases), discarded the suites and rebuilt evaluation on sealed worlds.
- Registered operator study: agents using Forge were not significantly more accurate than agents working by hand (p = 0.69), but finished faster (20 vs 43 min) and made no stray writes.
- Research paper with my supervisor: Agent-Operable Specialization of Small Tool-Calling Models on Consumer Hardware (2026, venue pending).
Measured
-
279
Specialists completed a mean 279.3 of 300 commands per world; the general model completed 290.0.
Faster and cheaper came with a small, measured cost in accuracy. That trade is the finding.
4 Oct 2026
-
17×
On 30 production-shaped scheduled prompts, the specialist gateway answered in a median 0.94 s against 16.20 s.
On the jobs my agents actually run every day, the gap is wider still.
14 Oct 2026
-
25%
In a registered study, agents using Forge reached a median 24.6% hidden-test pass rate against 34.9% for agents working by hand; exact Mann-Whitney p = 0.69.
The tool made agents faster (20 against 43 minutes) and safer (no stray writes), not more accurate. The study says so, so I say so.
19 Oct 2026
-
1 of 20
An adapter that scored 44 of 50 on its own test suite answered 1 of 20 paraphrased prompts correctly; 52–88% of the suite prompts were verbatim training rows.
My test set had leaked into training. I threw the suites out and rebuilt evaluation on sealed, seeded worlds no training run could see.
28 Sep 2026
-
5.4×
On the same 300 commands per world, Forge's 1B specialists answered in a median 1.47 s against 7.88 s for a general model.
Small local specialists make agent commands feel instant.
4 Oct 2026
-
5.9×
Specialists needed 604 input tokens per command against 3,573 for the general model.
Fewer tokens means cheaper calls and room on a laptop.
4 Oct 2026