Local AI on 8GB of VRAM: this is how I do

The local model is not the magic part

The most important thing about my local AI setup is that it is not one thing. It is a small stack of tools, scripts, compromises, and command-line arguments balanced on top of a graphics card that was clearly not asked whether it wanted this life.

The basic shape is simple enough:

LM Studio      -> find and download models
llama.cpp      -> run them as a local server
Pi terminal    -> use them quickly from the command line
frontier models -> plan the work too large for the local model
task files      -> feed the local model small, executable jobs

That is the tidy diagram. The lived experience is more like maintaining a classic car, except the carburettor has a chat template and sometimes confidently deletes the wrong import.

I use local AI because it is useful. Not theoretically useful. Not “one day when the agents can do the whole sprint while I sit in the garden” useful. Useful today, in the modest, slightly grubby engineering sense: explain this function, write this test, adjust this script, refactor this small thing, find the obvious mistake I have been staring at for ten minutes.

It is also cheap at the margin, private enough for many working notes, fast enough when tuned properly, and pleasingly under my control. I can swap models, change quantizations, tune context, run offline, and poke the engine while it is running. This is exactly the sort of freedom software people claim to want right up until they receive it and discover it comes with documentation.

LM Studio is the parts counter

I use LM Studio mostly for convenience. It is very good at the bit of local AI that should be boring: finding models, downloading them, keeping them somewhere sensible, and sparing me from doing file management archaeology on Hugging Face.

I do not generally use LM Studio as the runtime for my actual working setup. Once it has done the useful work of fetching the model, I turn it off. This feels a bit unfair to LM Studio, like inviting someone round to carry a piano and then asking them to leave before the music starts, but it is the workflow that suits me.

The model files live locally. The serving layer is mine. That matters because I want repeatable startup commands, predictable flags, and the ability to test exactly what changed between one setup and the next. If the model got faster, I want to know whether it was the quant, the context setting, the cache type, the GPU offload, flash attention, the runtime fork, or just me misreading the numbers because hope is a hell of a profiler.

So LM Studio is where the model enters the system. It is not where the system ends.

llama.cpp is where optimism meets physics

The centre of the setup is llama.cpp, usually running as a server that exposes an OpenAI-compatible local endpoint. That endpoint is the trick that lets other tools treat the model like a normal chat or coding backend without caring that the whole thing is being held together by PowerShell, GGUF files, and stubbornness.

In practice I have scripts for the models and runtime modes I use often. The exact flags change because local AI is still at the stage where the answer to “what is best?” is “what did you compile last Thursday?” But the categories are stable:

• model path
• context length
• GPU offload
• cache quantization
• flash attention
• chat template
• server host and port
• metrics
• sampling defaults

This is the layer where I spend the most time tuning. I have an 8GB VRAM GPU and 64GB of system RAM, which means I can do more than a sensible person would expect and less than the part of my brain reading benchmark screenshots wants. Local AI is mostly a negotiation between those two facts.

Eight gigabytes of VRAM is not a lot in modern AI terms. It is enough to be dangerous, which is a very specific category of enough. The GPU can carry part of the work. The rest spills into system RAM and CPU. Quantization becomes important. Context length becomes expensive. A command-line flag that looks minor can be the difference between “this is perfectly usable” and “I have invented a very slow way to heat the room.”

The goal is not to win a synthetic benchmark. The goal is to make the system pleasant enough that I reach for it during normal development instead of only when I feel like conducting a lab experiment.

Qwen MoE is the current sweet spot

The model I tend to come back to is Qwen 3.6 35B A3B.

Qwen 3.6 and MoE- dancing in the rain

The model is 22GB for the lowest quant, so how can that work with an 8GB GPU? ‘Mixture of experts’, that’s how.

Models come in 2 variants

• Dense
• Sparse

Dense models require the inference engine to read every parameter in order to produce each token. If some of the parameters are in RAM, rather than VRAM, either the CPU has to process them OR they have to be shuffled from RAM to VRAM. CPUs are far slower at generating tokens but shuffling parameters back and forth across the system bus is very slow too.

Sparse models, on the other hand, are built into packages of parameters that can be intelligently shuffled in an out of VRAM, depending on the task at hand. The appropriate ‘expert’ can be moved into VRAM and the irrelevant ones can live in RAM. If you’re project is a Java backend, Postgres DB and a React SPA, you can just have those experts in VRAM and the .NET, Python, Rust etc etc experts can go fish in RAM.

This is an over-simplification, I’ll admit, but it serves a purpose, and it’s about as detailed an explanation as I am qualified to give.

The result? On my machine, this class of model can feel much smarter than the small dense models while still being fast enough to use interactively. That is the prize.

For my normal workflow I often see roughly:

generation: around 30 tokens per second
prefill:    around 200 tokens per second
TTFT:       usually under 10 seconds in Pi terminal

These numbers are not laws of nature. They depend on context, prompt size, runtime, cache settings, quant, what else Windows has decided to do in the background, and whether the machine has entered one of those spiritual states where every process claims to be idle while the fans suggest otherwise. Also I discovered today - leaving Battlefield 6 running in the background makes my room warm and my tokens infrequent.

But they are good enough. That is the important phrase in local AI. Not perfect. Not frontier. Good enough to sit inside the development loop without making me resent it.

Turboquant is the other lever

I switch between standard llama.cpp and the turboquant fork depending on what I need. If I am doing normal small tasks, plain llama.cpp is often the comfortable choice. If I need a larger context, or I want to experiment with squeezing more conversation or source into the session, turboquant becomes attractive.

This is less elegant than it sounds. It is not a grand architecture. It is more like having two spanners because one of them reaches the awkward bolt behind the sink.

The trade-off is always the same:

• How much context do I need?
• How much quality can I keep?
• How quickly does it start responding?
• How much generation speed do I lose?
• How much RAM and VRAM am I consuming?
• Can I reproduce this setup tomorrow?

The last question matters more than it first appears. Local AI tuning is full of accidental discoveries. You change three things, performance improves, and for a few glorious minutes you feel like a systems engineer. Then you realise you did not write the command down. At that point you are not doing research. You are doing folklore.

That is why the scripts matter.

Pi terminal is fast enough to stay invisible

The client I use most is Pi terminal. I started with Forge but I struggled to make it work the way I wanted. At one point I found myself installing WSL in Windows, so that I could get a Linux shell so that I could install the dependencies and it felt like a kluge. Meanwhile, I kept hearing about Pi terminal and gave it a shot. The difference was pretty stark - none of the bells and whistles… but then I wasn’t interested in whistling or ringing bells. It did just enough and, the big win, the TTFT was much shorter. I got responses back so much quicker.

So I like it because it is quick and simple. It gets out of the way. Its system prompt is small, hence the short TTFT. Under ten seconds is the rough line where the tool still feels conversational rather than like submitting a batch job to a departmental mainframe.

This is an underrated part of AI tooling. Model quality matters, obviously, but latency changes behaviour. If the assistant responds quickly, you ask smaller, more frequent questions. If it takes too long, you batch things up, over-prompt, and make every interaction heavier than it needs to be.

For local AI, the prompt wrapper can be the difference between a nimble little tool and a machine that spends most of its life re-reading a constitution it wrote for itself.

Pi terminal works well for the way I use local models: focused, practical, incremental. I am not asking it to reinvent the architecture of a business system while blindfolded and armed with a stale dependency graph. I am asking it to do jobs small enough that I can check them.

This is important, because I have seen AI make catastrophic mistakes. Not funny mistakes, like inventing an API that sounds plausible. Proper mistakes: confidently wrong assumptions, incorrect edits, subtle regressions, changes that look coherent until you realise the model solved the problem in a nearby universe.

And that means…

So yes, I use AI constantly. I am cautiously optimistic. I am also cautious because I have been developing software for 30 years, and one thing 30 years teaches you is that confidence is not the same as correctness. Sometimes it is just a better class of bug report.

Small tasks are where local AI shines

The best use of my local setup is small coding work. Not glamorous work. Useful work.

Examples:

• write a narrow unit test
• explain a small module
• convert a script from one style to another
• fix a failing assertion
• add a simple option to an existing command
• summarise a log
• draft a commit message
• check a refactor plan for obvious holes
• generate a boring bit of boilerplate

This is where the economics and ergonomics line up. The local model is fast enough. The context is small enough. The risk is contained enough. I can inspect the result quickly. If it gets something wrong, the blast radius is not “Tuesday is cancelled.”

The trick is to make the task fit the tool. A local model on constrained hardware is not a substitute for a frontier model with a huge context window and a horrifying electricity bill. It is a capable assistant with limited working memory. Treating it that way makes it much more useful.

Big tasks get broken down elsewhere

For larger work, I use a different pattern.

First, I ask a frontier model to think through the whole job. The point is not that frontier models are magic either. They are just better suited to wide context, architectural reasoning, and holding more of the problem in view at once. If I am changing a feature across several files, or planning a migration, or trying to understand the shape of a system, that broader pass is valuable.

Then I ask the frontier model to break the plan into small, explicit tasks that a local model can execute. Each task goes into its own file. The file includes the goal, the relevant context, the constraints, and the expected checks.

Then I work through those files with the local model.

That division of labour matters. The frontier model does the expensive thinking. The local model does the bounded execution. I review the output. The workflow keeps the local model in the zone where it is useful instead of asking it to pretend it can see the whole battlefield from behind a filing cabinet.

It also gives me a trail. Each task file is a little contract. What was it asked to do? What files mattered? What does done mean? That is much easier to manage than a single enormous chat where the assistant slowly accumulates the personality of a confused build server.

The stack is the product

The conclusion I keep coming back to is that local AI is not really about the model. The model matters, but the stack is the product.

LM Studio makes acquiring models easy. llama.cpp makes serving them flexible. Scripts make the setup repeatable. Pi terminal makes the interaction fast enough to use. Qwen MoE gives me a useful capability-to-speed trade-off. Turboquant gives me another lever when context becomes the problem. The task file workflow keeps the work small enough to trust, inspect, and recover from.

None of this looks like the glossy agent demos. There is no dramatic music. No autonomous software engineer quietly shipping a product while I sip coffee and practice saying “leverage” on a podcast.

What I have instead is better: a toolchain I understand, running on hardware I own, doing real but bounded work, with enough speed that I actually use it.

Local AI is still awkward. It still requires taste, tuning, suspicion, and the occasional evening spent discovering that a flag did exactly what the documentation said it did, which is rude. But it is also genuinely useful. For an 8GB GPU and a pile of scripts, that is not bad.

The robot overlords may arrive eventually. For now, mine is a local server with a small context window, a surprisingly useful coding habit, and a fan curve that suggests it has opinions.

My configuration

Hardware

Component	Spec
CPU	i5-14400f
RAM	64 GB DDR5
GPU	RTX 5060 8GB

Software

Item	App	Version
Model management	LM Studio	0.4.16
Inference engine	llama.cpp	9245
Client	Pi terminal	0.75.2

Llama.cpp script

$LlamaServer = "C:\apps\turboquant-plus-tqp-v0.1.1-windows-x64-cuda12.4\llama-server.exe"
$HostName = "127.0.0.1"
$Port = 8080
$Template = "C:/Users/X/qwen3.6_tool_template.jinja"

. "$PSScriptRoot\common.ps1"

& $LlamaServer `
  --model "D:\models\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-Q8_0.gguf" `
  --alias Qwen3.6-35B-A3B-Q8_0 `
  --host $HostName `
  --port $Port `
  -fa on `
  --chat-template-file $Template `
  --jinja `
  -ctk turbo4 `
  -ctv turbo3 `
  -c 64000 `
  -t 10 `
  --mlock `
  --no-mmap `
  --ubatch-size 512 `
  --batch-size 1024 `
  --top-p 0.95 `
  --top-k 20 `
  --parallel 1 `
  --temperature 0.6 `
  --metrics