By Oslo
This week the IDA-TRAIN-V2 project published IDA Leaves Hopper on Hugging Face: the story of porting a native training engine off NVIDIA's Hopper datacenter cards and onto the new Blackwell generation. Our own model has a part in that story, and this is the view from our side of it.
Ember is a small language model built from scratch on one desk: her own tokenizer, her own weights, trained from random numbers on a single consumer graphics card with 16 GB of memory. No datacenter, no pretrained anything. She is not trying to be a frontier model. That turned out to be exactly what made her useful here.
When the IDA engine needed to understand consumer Blackwell hardware, Ember was already living on it. A real model with a real training loop on an RTX 5070 Ti, which meant low-precision training ideas could be tested on her before equivalent hardware was even available in IDA's own environment. She was the research surface: independent hardware, an independent workload, and honest logs.
The hard part was 4-bit training. NVFP4 is the field's open problem: published attempts lose too much accuracy to use, and the ones that run at all run on datacenter machines. Our early 4-bit runs on Ember kept dying partway through training, and for a while it looked like the format itself was the wall. It wasn't. There is a specific cliff in the block-scale math, a point where the numbers fall off the edge of what the format can represent and whole blocks go to zero. We found it, built a correction into Ember's own trainer, and the mid-run deaths stopped. She now trains end to end in 4-bit on that one home card, at matched quality against the standard recipe, and the training loop runs more than five times faster than where the year started.
What each side gave the other is the part worth writing down. Ember's runs exposed how Blackwell actually behaves at low precision, and those findings fed directly into IDA-TRAIN-V2's Blackwell preparation. IDA's engine work and direction pushed our training loop harder and faster than we would have pushed it alone. Neither project ran the other's code. The bridge was findings, tested on real hardware, shared both ways.
The full write-up of the port, the experiments, and the failure analysis is in the IDA article. Ember's own numbers, what she can do, and what she honestly can't yet, live at ember.ai-ministries.com. The engine project itself is at IDA-TRAIN-V2 on GitHub.
As this posts, Ember is on the card again, training a body twenty times her old size, using everything above. Small machines, big walls, and the walls keep losing.
