Local AI Is Finally Real. It Is Also Weird, Fragile, and Slightly on Fire.

There is a version of the local AI argument that goes something like this: unless you have a pair of RTX 3090s, a small modular reactor (SMR) power supply, and enough airflow to endanger low-flying aircraft, you are not really doing local AI. You are merely downloading disappointment.

That view is not entirely wrong. If your definition of “local AI” is running the largest open models at high precision, high context, high throughput, with multiple concurrent users and no compromise, then yes: welcome to the land of dual 32GB GPUs, used workstation motherboards, riser cables, suspicious BIOS settings, and forum posts that begin with “after two weeks of debugging PCIe lanes…”

But it is no longer the whole story.

The more interesting story is that local AI has started to become practical on normal-ish enthusiast hardware. Not magical. Not consumer-friendly. Not “download one app and replace your engineering department by Tuesday.” But practical enough that a developer with an entry-level GPU, enough system RAM, and a tolerance for configuration spelunking can now use a local model as a real part of their coding workflow.

That is new.

The core shift is sparse models, especially Mixture of Experts. Traditional dense models are brutally simple in one important sense: if it is a 27B model, you are dragging 27 billion parameters around whether the current token needs all of them or not. MoE models are different. They may contain tens of billions of parameters in total, but only activate a smaller slice per token. Qwen’s recent A3B models are the obvious examples: large total parameter counts, but only around 3 billion active parameters during inference.

My own journey

A colleague knew that I, a seasoned software developer of 30 years, had just discovered Claude Code (having been previously wowed by CoPilot auto-complete in VSCode) and was evangelizing to anyone that would stay still enough to be talked at. So he knew he had the right audience when he asked if I had heard about the new Gemma 4 models for running local AI models.

I had not. As far as I was concerned, local AI was a world of very knowledgeable experts using scary terms like ‘quant’, ‘sparse models’ and ‘tensor layers’ and I was not sure how eager I was to disappear down that rabbit hole. But then, as so often happens, Claude Code came to my rescue and blew away my week’s tokens with 3 days left in the billing week and suddenly I had the motivation.

Necessity, as they say, is the mother of invention. I dove down the rabbit hole. After spelunking in the /r/localLLM subreddit, I learned that rookies use LM Studio and the more confident users switch to llama.cpp. I installed both and started with LM Studio which is a GUI that makes downloading models trivial and running them even easier.

LM Studio and llama.cpp (or Ollama if that’s your jam) serve as the inference engine which makes the model accessible to tools which can query the model. If you want to start coding with it, you still have to put a terminal and tooling in front of it. Otherwise, you just setup your own chat bot and if that’s what you’re after I recommend just finding an enterprise website and making friends with their chatbot and keeping life simple. To start coding, you effective need a tool that

• knows how to send and receive messages from the inference engine
• has a bunch of tools that a chat bot could invoke to do things (such as read your existing files, modify them, run build tools etc)
• has a system prompt that is sent to the model, before anything, that explains to it how to use the suite of tools (syntax, payloads etc)

In a future article I’ll talk about my favourites Pi and Forge.

Free lunch? No token limits? You SOB I’m in!

This is not free lunch. It is more like finding a Greggs pasty behind a firewall rule. You still need the model weights somewhere. You still need memory bandwidth. You still need a runtime that knows what to place on the GPU, what to leave in system RAM, and how much pain you are willing to tolerate per token. But it changes the shape of the trade-off.

The old question was: “Can the whole model fit in VRAM?”

The new question is messier, but more useful: “Can enough of the right work happen on the GPU, while the rest limps along through RAM, without making the experience unbearable?”

That is why a humble 8GB GPU is no longer automatically disqualified. I am not claiming my RTX 5060 8GB has secretly become an H100 after reading a self-help book. It has not. It remains an 8GB card, with all the emotional range of an 8GB card. But with quantised GGUF models, llama.cpp, careful context choices, and MoE-aware offloading, it can do work that would have sounded ridiculous not very long ago.

The community evidence is all over the place, in the best possible way. On one side, LocalLLM and LocalLLaMA are full of people building machines around RTX 5090s and RTX 6000s. That makes sense: 32GB of VRAM is still the great consumer AI cheat code. Multiple cards give you access to larger models, assuming your case, motherboard, PSU, thermals, driver stack, and relationship with cable management all survive the experience.

At the extreme end, people are discussing four-card 3090 builds, Threadrippers, EPYCs, hundreds of gigabytes of RAM, and custom workstation-class rigs. This is the “frontier outpost made of Noctua fans” school of local AI. I respect it. I also do not want to explain it to my electricity supplier or partner.

But alongside that, there is a more interesting pattern emerging: people running Qwen MoE models on 8GB and 12GB GPUs with a pile of RAM, a lot of tuning, and very specific expectations. Not always chatty. Not always fast. Not always stable. But useful.

That last word matters.

The goal is not to beat Claude, GPT-5, or whatever cloudy deity is currently being sold by the million tokens. The goal is to have a local coding assistant that can read context, perform structured tasks, generate refactors, propose tests, explain unfamiliar code, and operate without sending your repository to someone else’s inference cluster.

That is a different value proposition.

For my workflow, the local model is not an autonomous genius. It is not a junior developer. It is not even an intern. It is more like a caffeinated search-and-transform engine with flashes of competence and occasional episodes of staring directly into the sun. I use it where I can inspect the output. I give it bounded tasks. I ask for diffs, explanations, candidate implementations, and test scaffolds. I do not let it roam the repo unattended while I make tea and hope the robot overlord has internalised SOLID principles.

This is the caveat most AI boosterism skips. Local AI is real, but it is not yet “vibe coding without overwatch”. If you cannot evaluate the output, you are not accelerating your development; you are outsourcing confusion to a matrix multiplication appliance.

The best current use case is a capable developer using local AI as leverage. You still need taste. You still need judgement. You still need to know when the answer is subtly wrong, insecure, overcomplicated, or confidently hallucinating an API that only exists in the model’s tiny synthetic dream world.

The other caveat is that the tooling remains enthusiast-grade. llama.cpp is powerful, but it is still very much a place where flags matter. Context length matters. KV cache settings matter. Batch and ubatch values matter. Whether thinking mode is on matters. Whether the model spends its entire output budget thinking instead of producing code matters. Ask me how I know.

There is no single magic command. There is a process: choose a model, choose a quant, test realistic tasks, watch token speed, watch VRAM, adjust context, try different cache settings, change offload strategy, and eventually convince yourself that this is “productive” rather than “a technically elaborate form of procrastination.”

I’ll share my latest command line at the end, so you can see what my current flags are to squeeze the most out my humble hardware.

For the right person, it is productive.

The point is not that everyone should do this. Most people should not. Most teams should keep using hosted tools, enjoy their polished UX, and spend their weekends doing something less deranged than benchmarking quantisations.

But for frontier explorers — developers who care about privacy, cost control, offline capability, repeatable workflows, and understanding the machinery they are using — local AI has crossed an important threshold. It is no longer only a toy. It is not yet an appliance. It is a workshop tool: sharp, useful, badly labelled, and quite capable of removing a finger if you stop paying attention.

That is the interesting moment we are in.

The future of local AI probably does not arrive as one giant model that needs two £1,400 GPUs and a shrine to CUDA. It arrives sideways: sparse activation, better quantisation, smarter runtimes, CPU-GPU hybrid inference, long-context models, and coding agents that can be boxed into workflows where their mistakes are visible before they are expensive.

So yes, local AI is real now.

But like most real things in computing, it comes with configuration files, caveats, and at least one Reddit thread that tells you the answer was obvious if only you had known to set the flag that was renamed three releases ago.

The robot overlords may be coming.

For now, they still need help fitting into 8GB of VRAM.

My hardware

• i5-14400f
• 64GB DDR5 RAM
• RTX 5060 8GB

My command line

  C:\apps\current-llama\llama-server.exe `
    --model "Qwen3.6-35B-A3B-UD-Q4_K_S.gguf" `
    --alias Qwen3.6-35B-A3B-UD-Q4_K_S `
    --host $HostName `
    --port $Port `
    -ngl auto `
    -fa on `
    --chat-template-file $Template `
    --jinja `
    -ctk q4_0 `
    -ctv q4_0 `
    -c 64000 `
    -t 8 `
    --mlock `
    -b 1024 `
    --ubatch-size 512 `
    --temperature 0.6 `
    --spec-type draft-mtp --spec-draft-n-max 3 `
    --metrics