Local AI Explained in 12 MinutesBeginner's Guide

The file is the model (April 2, 2026, Gemma 4 E4B, 6.6 GB)

April 2, 2026. Google released Gemma 4. 5 billion effective parameters. 6 gigabytes.

That file is the model. On your disk. Not a demo. The weights themselves.

A cloud chat never gives you that file. The weights stay on their servers. Your prompt goes out. Their computer does the math.

What "local" means

You get text back. Local is the reverse. You download the weights once. Inference runs on your CPU, or on your GPU.

No API call for the answer. Three pieces. The weights are a file. Ollama keeps them in your home folder.

On a Mac, that path is the dot ollama folder, then models. Inference is the work. Your prompt goes in. The model writes the next token.

Then the next. That work is on your chip. And there is no API in the loop. Ollama's FAQ says they do not see your prompts or your answers when you run locally.

The server on your machine is local host, port one, one, four, three, four. One catch, from that same FAQ. Ollama also has cloud models. The name ends in colon cloud.

Those prompts leave the machine. Local only is a switch. In the dot ollama folder in your home directory, open server dot json, and set disable ollama cloud to true. Or set the variable ollama, no, cloud, to 1.

Why people run it locally

Then restart. Four reasons. All of them are limits of the cloud. Privacy.

A local run stays on the computer. That is the FAQ line above. Turn cloud off if you want it to stay true. Cost.

You do not pay per token. You already own the machine. The model is a download. LM Studio's download page lists the app at a price of zero.

Offline. LM Studio's docs say the app can run entirely offline, once the model files are already on disk. The first download still needs the network. No company meter.

A cloud account can cap you, queue you, or swap the model. A file does not. Your limit is memory.

The one number: memory

That is next. The one number is memory. A model is a list of numbers. Those numbers are the parameters.

1 8B means about 8 billion of them. Meta's card says 8B. Released July 23, 2024. Each parameter is stored at a precision.

That is bits per weight. 16 bit is two bytes. 8 billion times 2 is 16 gigabytes. 8 bit is one byte.

8 gigabytes. 4 bit is half a byte. 4 gigabytes. Parameters, times bytes per weight.

That is the whole formula. Real files run a little larger than the raw math. Quantization packs each weight into blocks, and each block keeps a scale. 5 bits per weight, not a flat 4.

89 bits per weight. Their table, same model. These figures are gibi bytes. G, I, B.

16 bit, called F16. 96. 8 bit, called Q 8, underscore, zero. 95.

4 bit, called Q4_K_M. 58. 9 gigabytes. 541.

Same model. Three sizes. You pick the one that fits. Smaller is not free.

cpp says quantization can cost some accuracy. It can also speed generation up. The speed numbers come with the limits. Where does that memory sit?

A Windows PC with a graphics card has two pools. RAM for the CPU. VRAM for the GPU. Apple said this plainly in a 2020 developer talk.

A discrete GPU has its own memory. Moving data across takes a bus.

RAM, VRAM, and what fits

Apple silicon is one chip. CPU and GPU share one pool. Apple calls it a unified memory architecture. A Mac with 16 gigabytes does not split that into 8 plus 8.

The model can use the same memory the GPU uses. Ollama turns that on through Metal. Ollama will tell you which pool it used. The command is ollama ps.

100 percent GPU means the model loaded on the GPU. 100 percent CPU means system memory. A split, like 48 percent CPU and 52 percent GPU, means both. cpp can also split layers.

Some on the GPU. The rest on the CPU. That is for a model bigger than VRAM. The weights have to be loaded.

cpp's quantize notes say that, for these files, memory and disk match. 9 gigabyte model needs about that much free memory. Then more, for the context. Context is extra.

Ollama's default context is 4096 tokens. The model card can say 128 thousand. That is the max the model was built for. It is not what Ollama loads unless you change it.

The setting is num, underscore, C T X. The FAQ says a longer context uses more memory. So. What fits.

These are published file sizes. Leave room for the system, and for those 4096 tokens. 8 gigabytes. LM Studio says an 8 gigabyte Mac can run, if you stick to smaller models and a modest context.

3 gigabytes. 0 gigabytes. 9 gigabytes is weights alone. That is tight on eight gigabytes of memory.

Sixteen gigabytes. LM Studio recommends sixteen or more. 9 gigabytes fits, with room. 5 gigabytes of weights, is possible.

Not roomy. On Windows, LM Studio also asks for at least 4 gigabytes of dedicated VRAM. 32 gigabytes. The 16 bit file for the 8B is about 16 gigabytes, so it fits.

The 70B at 4 bit does not. 1 70B at 43 gigabytes. 1. Next rung.

Sixty four gigabytes and up. 43 gigabytes of 70B weights can fit, with room left over. The 405B does not. Ollama lists it at 243 gigabytes.

1. That is a server. Three tools. All three run the model on your machine.

Ollama is an app, and a terminal command. Mac or Linux. com. Windows.

A PowerShell installer from the same site. Then the ollama command.

Ollama, LM Studio, llama.cpp

The API stays on localhost, port one, one, four, three, four. LM Studio is a desktop app. Download it from L M studio dot A I, slash download. Mac, Windows, and Linux.

cpp. On Apple silicon it can also run MLX. Discover is where you download a model. Chat is where you talk.

It can serve a local API too. cpp is the engine under a lot of this. Written in C, and in C plus plus. The license is MIT.

app. Then a real run line from that same page. It starts with llama cli, dash H F. The model is Qwen, 3 point 5, 0 point 8 B.

The repo is G G M L dash org. That pulls a small model from Hugging Face and runs it. Ollama, if you want one command. LM Studio, if you want buttons.

cpp, if you want the engine. Three steps. First model. 2.

0 gigabytes. Step one. Set Ollama up.

Run your first model

The exact line is on screen, and in the description. com. Windows, it is the PowerShell installer on that same site. Windows docs say the app itself needs about 4 gigabytes.

Models are extra. Tens of gigabytes. Sometimes hundreds. Step two.

Run this. ollama run llama three point two That tag is the 3B. 0 gigabytes. The library page lists a 128 thousand token context on the model.

Ollama still loads 4096 unless you change it. First run downloads the weights. Then you get a prompt. Step three.

Ask something small. What is 15 percent of 80. The answer is computed on your machine. Then check the pool.

ollama ps. You should see the model, a size, and a processor. GPU, CPU, or a split. Done for now.

Stop it with ollama stop, then llama three point two. By default, Ollama keeps a model in memory for 5 minutes. Then it frees it. The limits.

Honest limits

The laptop file is not the frontier model. 1 also comes in 70B and 405B. 5, Sonnet. That quant is about 243 gigabytes.

The file that fits in 16 gigabytes is the 8B. Or the 3B. Smaller model. Different job.

Speed is tokens per second, and it belongs to your hardware. cpp is one machine, and they do not name it. 9 tokens a second at 4 bit, on 128 tokens, for this 8B. 9 at 8 bit.

2 at 16 bit. A CPU only load is the 100 percent CPU line in ollama ps. It will feel slower than a full GPU load. Your number is yours.

Time it yourself. Context is a budget. 128 thousand on the card is not 128 thousand free. The default is 4096.

Longer costs memory. Quality moves when you quantize. 4 bit is the common download because it fits. It is not the 16 bit file.

cpp says the process may cost some accuracy. And local does not mean private if the tag says cloud. Check the name. Check ollama ps.

If the weights are not on your disk, it is not local.

Next: best local coding model

That is the loop. Weights on disk. Math on your chip. Memory picks the size.

Next video. The best local coding model for your GPU. Same ladder. 8, 16, 32, 64.

A model picked for code. If this helped, subscribe. Build with AI. For real.