The next age of LLMs? Dev gets a small LLM running at 10 tokens a second locally on a $10 microcontroller

Date:
Fri, 07 Aug 2026 19:35:00 +0000

Description:
A 28.9M-parameter model runs on a microcontroller costing less than $10 at 9.88 tokens a second because 25M of those parameters never leave flash storage

FULL STORY ======================================================================Copy link Facebook X Whatsapp Reddit Pinterest Flipboard Threads Email Share this article 0 Join the conversation Follow us Add us as a preferred source on Google Newsletter Subscribe to our newsletter A developer has a 28.9-million-parameter model generating TinyStories-style text at 9.88 tokens per second, fully offline, on a $10 microcontroller It was achieved by
fitting a language model on a chip with 512KB of RAM by leaving most of it in flash storage The project is available on GitHub under an MIT license A developer going by slvDev has a 28.9-million-parameter language model generating text on an ESP32-S3, a microcontroller built for sensor nodes and smart plugs, at 9.88 tokens per second, with nothing leaving the chip.

The project, esp32-ai, went up on GitHub under an MIT license in late July 2026, was showcased on the Better Stack YouTube channel, and has since collected over 3,600 stars and has more than 470 forks of the underlying
code. Given the underlying hardware's limitations, the attempt meant the model, which was trained on the TinyStories dataset by Microsoft Research,
had to be scaled down. Latest Videos From TechRadar Watch full video here: Still a capable LLM despite the loss in accuracy The arithmetic involved here is brutally unforgiving; weights typically need one to four bytes per parameter just to sit in memory, so 28.9 million parameters at 16-bit precision still require nearly 60 MB of RAM. An ESP32-S3 offers 512 KB of
SRAM and 8 MB of PSRAM that is woefully inadequate at first glance.

Quantization closes the first part of the gap: dropping the weights to four-bit precision trades a little accuracy for a 75% reduction, bringing
60MB down to 14.9MB. You may like Chinese DoorDash rival smashes open source record with 1.6-trillion-parameter LLM Startup backed by the world's largest battery maker just launched a supercheap mini PC that competes with Nvidia's $5000 AI DGX Spark PC Tiny company steals AMD's thunder and challenges Nvidia with old-tech PCIe AI accelerator

The second step is where the project earns most of its attention: most of a language model's parameters sit in embedding tables that the model reads from rather than computes against, and something you only read can live somewhere slow.

Borrowing Per-Layer Embeddings from Google 's Gemma 3n, slvDev moved about 25 million parameters (roughly 12 MB at four-bit) into the board's 16 MB of flash, pulling around 6 rows, about 450 bytes, per token. Are you a pro? Subscribe to our newsletter Sign up to the TechRadar Pro newsletter to get
all the top news, opinion, features and guidance your business needs to succeed! Contact me with news and offers from other Future brands Receive email from us on behalf of our trusted partners or sponsors By submitting
your information you agree to the Terms & Conditions and Privacy Policy and are aged 16 or over.

What remains in working memory is closer to 2 MB: the dense core and output head in PSRAM, and activations and norm weights in SRAM. Each tier stores whatever is read at its own frequency, and the chip's memory hierarchy serves as the architecture that enables the LLM.

Offloading weights to storage is not new; it is how people coax
frontier-class models onto hardware that has no business running them , but
it usually destroys throughput, turning tokens per second into seconds or minutes per token.

The PLE approach avoids that because the offloaded weights are accessed sparingly, so flash bandwidth never becomes the bottleneck. The resulting performance, 9.88 tokens per second, is faster than most people read.

The LLM has its limitations, however: the model writes short, mostly coherent stories and will not answer questions, follow instructions, write code, or know facts. The developer says the limit stems from the small part of the model that performs the reasoning (~4M parameters) and that the memory trick does not change its capabilities. His repository also ships a second model, Barista, which expressly answers espresso questions only (pun intended) .

In short, the approach doesn't make small models smarter, but it does allow them to offload data within the board onto chips they could not previously
run on, making it an interesting choice, though one with limited practical uses in its current iteration. Follow TechRadar on Google News and add us as
a preferred source to get our expert news, reviews, and opinion in your feeds.



======================================================================
Link to news story: https://www.techradar.com/pro/the-next-age-of-llms-dev-gets-a-small-llm-runnin g-at-10-tokens-a-second-locally-on-a-usd10-microcontroller


--- Mystic BBS v1.12 A49 (Linux/64)
* Origin: tqwNet Technology News (1337:1/100)