1. How This Experiment Started

After burning through my 5-hour Claude Code session limit in barely an hour, I had four unexpected hours on my hands.

Instead of doomscrolling, waiting for the timer to reset, I decided to push Apple Silicon:

How far can Apple Silicon really go with large language models?

No cloud GPUs. No rented infrastructure. No Kubernetes cluster hiding in the background.

Just a MacBook M3 Max with 36 GB of unified memory, llama.cpp, a collection of heavily quantized models and willingness to push the hardware well beyond what most people would consider reasonable, with a lot of patience...

What started as a quick experiment turned into a surprisingly deep exploration of memory management, quantization strategies, MoE architectures, and the limits of local inference.

The result was something I didn't expect: with the right combination of CPU-only execution, --cpu-moe, --no-mmap, aggressive quantization, and carefully managed context sizes, even massive Mixture-of-Experts models that seem completely impractical on paper can become genuinely usable on a consumer laptop.

I got a 117B-parameter model generating tokens on a 36GB MacBook!

💡

It hardly takes 15-30 mins to Download and Run LLM locally

You can also run LLM locally (your own CHATGPT!), how big a model can run depends almost entirely on your available RAM (or VRAM on your GPU).

The biggest bottleneck on Macs is usually memory capacity first, memory bandwidth second, not raw compute. Unified memory is fantastic, but once you run out, swap destroys performance.

Recommended Models to run locally on MacBook:

2. The Breakthrough

The best local model isn’t the biggest one you can load — it’s the largest one that stays responsive. For a 36 GB M3 Max that’s the Qwen3-30B-A3B MoE, with the 80B, and 120B as proof of how far the machine goes once you stop offloading to Metal and start managing memory pressure by hand.

3. Why It Works

3.1 Why Apple Silicon Changes the Game for Local AI

Apple Silicon's Unified Memory Architecture (UMA) is the key reason large AI models can run locally on consumer hardware. Unlike traditional PCs, where CPU and GPU have separate memory pools, Apple combines the CPU, GPU, and Neural Engine into a single chip that shares one high-bandwidth memory pool.

This delivers three major advantages:

• Larger Models Fit in Memory – Up to ~75% of system memory can be used by the GPU, allowing significantly larger models to load than on typical consumer GPUs with 8–16 GB of VRAM.

• Zero-Copy Access – The CPU, GPU, and Neural Engine access the same memory directly, eliminating the costly RAM-to-VRAM transfers common on traditional systems.

• Massive Memory Bandwidth – M3 Max and larger Apple Silicon chips provide 400-800 GB/s of memory bandwidth, keeping AI workloads fed with data during inference.

Apple Metal: Unlocking the GPU:

Apple Metal is Apple's proprietary, low-level hardware acceleration API. It is the software layer that allows developers to speak directly to the Mac's GPU.

• Direct Hardware Control: Metal bypasses standard operating system layers, giving software near-instant access to the Mac's graphics cores and Matrix Coprocessors.

• Unified Memory Integration: Because of Metal, the GPU doesn't have to wait for instructions to copy data. Metal tells the GPU exactly where the model weights are sitting in the Unified Memory pool, executing calculations instantly in place.

• The Metal Shading Language (MSL): llama.cpp includes custom MSL code written specifically for Apple chips. When you load a model, llama.cpp uses Metal to compile these specialized kernels on the fly, turning your Mac's graphics cores into high-performance AI processors, allowing the GPU to accelerate matrix operations and dramatically improve model loading and token generation performance.

The result is simple: models that would normally require expensive workstation-class GPUs can often run directly on a MacBook.

3.2 What is LLAMA.CPP?

Created by Georgi Gerganov, llama.cpp made it possible to run large language models on everyday hardware without complex server infrastructure.

Its key advantages are:

• Lightweight and Fast – Written in C/C++ with minimal dependencies, avoiding the overhead of large AI frameworks.

• GGUF Quantization – Compresses models from FP16 to lower-precision formats such as INT4 and INT8, dramatically reducing memory requirements with minimal impact on quality.

• Flexible Execution – Supports both CPU-only inference and GPU acceleration, including optimized support for Apple Silicon GPUs through Metal.

These features have made llama.cpp the go-to runtime for running large open-source models locally.

3.3. Understanding Mixture-of-Experts (MoE) models

Mixture-of-Experts (MoE) models increase an AI model's total knowledge without requiring all of that knowledge to be used for every token generated.

Traditional LLMs are dense models: if a model has 70 billion parameters, all 70 billion participate in every prediction. MoE models are sparse. They divide the network into specialized "experts" and activate only the most relevant experts for a given task.

Think of it like a manager routing a problem to the right specialists rather than asking the entire company to work on every request.

GPT-OSS-120B is a 120-billion-parameter Mixture-of-Experts (MoE) model, it is ‘sparse’, only about 5.1 billion parameters are active for each token generated.

In practice, GPT-OSS-120B is commonly deployed on GPUs with roughly 80GB of VRAM (NVIDIA H100, A100), depending on quantization and runtime configuration.

4. Apple MacBook (M3 Max)

💡

MoE cuts both ways. The full weight footprint is what blocks loading on 36 GB; the tiny active set is what makes these models fast on the CPU once loaded. The whole game is getting past the loading wall — then letting the small active set carry the speed. 

5. Setup Guide-Install Prerequisites

1. Install Homebrew packages:

brew update 

brew install git cmake ninja python huggingface-cli 

1. Install llama.cpp

brew install llama.cpp 

git clone https://github.com/ggml-org/llama.cpp.git 

cd llama.cpp 

cmake -B build -G Ninja -DGGML_METAL=ON 

cmake --build build –j 

Then your binaries are here:

./build/bin/llama-server 

./build/bin/llama-cli 

./build/bin/llama-bench 

1. Download model and Run Server

Download GPT-OSS-120B(Experiement)

mkdir -p ~/models/gpt-oss-120b 

cd ~/models/gpt-oss-120b 

hf download ggml-org/gpt-oss-120b-GGUF \ 

  --local-dir ~/models/gpt-oss-120b

Download Qwen3-Next-80B-A3B-Instruct-GGUF

hf download unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF \
  --include "*Q4_K_M*" \
  --local-dir ~/models/qwen3-next-80b-a3b-q4

*Execute llma-server command to Load Model and Run Server

1. Memory Recommendation

Under this default, macOS automatically reserves roughly 25% to 30% of total unified memory for the CPU, operating system, and background apps, capping the GPU at about 70–75% of total RAM.

The command overrides the default VRAM cap on Apple Silicon Macs, allocating ~30 GB of your Mac's unified memory exclusively to the GPU.

sudo sysctl iogpu.wired_limit_mb=30000 

Verify that the command applied correctly by running:

sysctl iogpu.wired_limit_mb 

Command to Reset it back to default:

sudo sysctl iogpu.wired_limit_mb=0

1. Host Endpoints

Once server is running, Open:

http://127.0.0.1:8080 

Open AI Compatible Endpoints:

http://127.0.0.1:8080/v1/chat/completions

6. Experiments: Loading LLM Model and Running a Server

1) GPT-OSS-120B

For each model, the parameters that failed and the parameters that succeeded are shown side by side — on a memory-constrained Mac, the parameters are the experiment.

Failed  — conservative CPU-only attempt (Q3_K_M):

~/llama.cpp/build/bin/llama-server \ 
  -m ~/models/gpt-oss-120b/Q3_K_M/gpt-oss-120b-Q3_K_M-00001-of-00002.gguf \ 
  -ngl 0 -c 512 -np 1 -b 8 -ub 4 --cache-ram 0 \ 
  --no-warmup --jinja --host 127.0.0.1 --port 8080 

Result: zsh: killed during load under memory pressure.

Failed  — CPU-only Q3_K_S, still no --no-mmap:

~/llama.cpp/build/bin/llama-server \ 
  -m ~/models/gpt-oss-120b/Q3_K_S/gpt-oss-120b-Q3_K_S-00001-of-00002.gguf \ 
  -ngl 0 -c 512 -np 1 -b 8 -ub 4 --cache-ram 0 \ 
  --no-warmup --jinja --host 127.0.0.1 --port 8080

Result: zsh: killed again.

Failed  — hybrid --cpu-moe, still no --no-mmap:

~/llama.cpp/build/bin/llama-server \ 
  -m ~/models/gpt-oss-120b/Q3_K_S/gpt-oss-120b-Q3_K_S-00001-of-00002.gguf \ 
  -ngl 4 --cpu-moe -np 1 -c 512 -b 8 -ub 4 --cache-ram 0 \ 
  --no-warmup --jinja --host 127.0.0.1 --port 8080 

Result: zsh: killed — but the log suggested --no-mmap for CPU tensor overrides. That was the clue.

Success  — the breakthrough command:

~/llama.cpp/build/bin/llama-server \ 
  -m ~/models/gpt-oss-120b/Q3_K_S/gpt-oss-120b-Q3_K_S-00001-of-00002.gguf \ 
  -ngl 4 \ 
  --cpu-moe \ 
  --no-mmap \ 
  -np 1 \ 
  -c 256 \ 
  -b 4 \ 
  -ub 2 \ 
  --cache-ram 0 \ 
  --no-warmup \ 
  --jinja \ 
  --host 127.0.0.1 \ 
  --port 8080 

💡

Loaded with n_slots = 1, n_ctx = 256. Observed speeds: 2.03, 3.12, and 2.45 tok/sec. A 117B model generating tokens on a 36 GB laptop — impressive, even if it is a benchmark rather than a workhorse. 

What it costs the machine is the real story. Memory pressure sat in the amber state with ~33.9 GB used, 19.2 GB compressed, and ~29 GB of swap in use:

And because --no-mmap forces a full load while the machine pages hard, the SSD took a sustained beating — hundreds of GB read and written at roughly half a gigabyte per second:

2)  Qwen3-Next-80B-A3B-Instruct (80B / 3B active) — the best large model

This model draws the line between fighting Metal and working with the CPU. Aggressive GPU offload hits Metal OOM; going CPU-only turns it into the best large-reasoning model in the set.

Failed  — GPU offload (Metal OOM):

~/llama.cpp/build/bin/llama-server \ 
  -m ~/models/qwen3-next-80b-a3b-q3/Qwen3-Next-80B-A3B-Instruct-UD-IQ3_XXS.gguf \ 
  -ngl 16 -c 4096 -b 64 -ub 32 -fa auto \ 
  --no-warmup --jinja --host 127.0.0.1 --port 8080 

This run is worth studying, because the failure mode is more subtle than “it ran out of memory.”

Success  — CPU-only:

~/llama.cpp/build/bin/llama-server \ 
  -m ~/models/qwen3-next-80b-a3b-q3/Qwen3-Next-80B-A3B-Instruct-Q3_K_S.gguf \ 
  -ngl 0 \ 
  -c 1024 \ 
  -b 32 \ 
  -ub 16 \ 
  --no-warmup \ 
  --jinja \ 
  --host 127.0.0.1 \ 
  --port 8080 

~19.83 tok/sec

3)  Qwen3-30B-A3B (30.5B / 3.3B active) — the daily driver

The family that quietly won the whole exercise. Unlike the giants, it offloads happily to Metal and runs fast.

Success  — Q5_0, first run (partial offload, small context):

~/llama.cpp/build/bin/llama-server \ 

  -m ~/models/qwen3-30b-a3b-q5/Qwen3-30B-A3B-Q5_0.gguf \ 

  -ngl 12 -c 2048 -b 32 -ub 16 \ 

  --no-warmup --jinja --host 127.0.0.1 --port 8080 

~37–43 tok/sec.

Success  — Q5_0, optimized (full offload, 16K context):

~/llama.cpp/build/bin/llama-server \ 

  -m ~/models/qwen3-30b-a3b-q5/Qwen3-30B-A3B-Q5_0.gguf \ 

  -ngl 999 -c 16384 -b 32 -ub 16 \ 

  --no-warmup --jinja --host 127.0.0.1 --port 8080 

~69.25 tok/sec — the fastest model in the entire test set.

Success  — Q6_K, the best balance (daily driver):

~/llama.cpp/build/bin/llama-server \ 

  -m ~/models/qwen3-30b-a3b-q6/Qwen3-30B-A3B-Q6_K.gguf \ 

  -ngl 999 -c 16384 -b 32 -ub 16 \ 

  --no-warmup --jinja --host 127.0.0.1 --port 8080 

~63.97 tok/sec — only moderately slower than Q5_0, but better quality. My standing daily driver: fast, stable at 16K, high enough fidelity for code review and strategy work.

Failed  — Q8_0, GPU offload (Metal OOM):

~/llama.cpp/build/bin/llama-server \ 

  -m ~/models/qwen3-30b-a3b-q8/Qwen3-30B-A3B-Q8_0.gguf \ 

  -ngl 8 -c 16384 -b 32 -ub 16 \ 

  --no-warmup --jinja --host 127.0.0.1 --port 8080 

Result: kIOGPUCommandBufferCallbackErrorOutOfMemory. The Q8 weights plus a 16K KV cache exceeded the ~27 GB Metal working set. The CPU-only fallback (-ngl 0) loaded but ran at only ~5–10 tok/sec. Bottom line: Q8 is not worth it on 36 GB for this model.

4)  Qwen3.6-27B (Dense, 27B) — the dense baseline

A dense 27B is a useful baseline for the dense-vs-MoE comparison.

Loads cleanly — no OOM, no swap drama — but with thinking mode enabled it landed at ~9.19 tok/sec (live tg decaying from ~12 to ~9). The verdict:

~/llama.cpp/build/bin/llama-server \ 
  -m ~/models/qwen3.6-27b-q5/Qwen_Qwen3.6-27B-Q5_K_M.gguf \ 
  -ngl 999 -c 16384 -b 32 -ub 16 \ 
  --no-warmup --jinja --host 127.0.0.1 --port 8080 

💡

A dense 27B is now clearly beaten by the 30B-A3B MoE, which runs roughly 7× faster for comparable-or-better output because only ~3.3B parameters are active per token. This is the dense-vs-MoE thesis made concrete. 

The two opposite playbooks:

• Small-enough MoE (30B): push -ngl 999, give it 16K context, let Metal do the work. Fast.

• Too-big MoE (80B/120B): pull work off Metal (-ngl 0 or --cpu-moe), shrink context to 256–1024, set -np 1, and for the 120B add --no-mmap --cache-ram 0. Slow but alive.

7. The llama.cpp Parameters That Mattered

8. VS Code Integration

The llama.cpp server exposes an OpenAI-compatible endpoint at http://127.0.0.1:8080/v1, which works with Continue.dev, Cline, Roo Code, Aider, and Open WebUI.

The default model is now the Qwen3-30B-A3B Q6 MoE; maxTokens is raised to 16384 so longer strategy scripts don’t truncate mid-generation.

Install Continue in VS Code.

name: Local Config 
version: 1.0.0 
schema: v1 
models: 
  - name: Qwen3-30B-A3B Q6 Local 
    provider: Qwen 
    model: qwen3-30b-a3b-q6 
    apiBase: http://127.0.0.1:8080/v1 
    apiKey: dummy 
    roles: [chat, edit, apply] 
    defaultCompletionOptions: 
      temperature: 0.2 
      topP: 0.9 
      maxTokens: 16384 

Temperature 0.2 / topP 0.9 keeps code generation conservative — low enough to avoid invented syntax, not so low that regenerations get stuck repeating a wrong answer.

9. Key Lessons Learned

• MoE beats dense on this machine, decisively. Qwen3-30B-A3B-Q6 runs ~7× faster than a dense 27B because only ~3.3B parameters fire per token.

• The big models are runnable — the trick is to stop fighting Metal. The 80B runs CPU-only at ~20 tok/sec; the 117B GPT-OSS loads and generates with the right memory flags.

• Active parameters ≠ required memory. The full weight footprint must fit; the tiny active set is what keeps CPU inference fast once it does.

• Q8 is not worth it on 36 GB for this model family — Metal OOM with GPU, too slow CPU-only. Use Q6_K.

10. The Final Verdict

The largest model you can technically memory-map is not the best local model. But with --cpu-moe, --no-mmap, and CPU-only execution, the practical ceiling on a 36 GB Apple MacBook is far higher than the dense models alone would suggest.

My Standing setup:

• Default: Qwen3-30B-A3B-Q6_K (-ngl 999, 16K) for daily coding and review.

• Bigger brain: Qwen3-Next-80B-A3B-Q3_K_S (CPU-only, small context) for deeper reasoning.

• For fun / benchmarking**:** GPT-OSS-120B-Q3_K_S with the breakthrough command — a ~2–3 tok/sec novelty that proves how far the machine can be pushed.

• Anything larger or denser**:** cloud / API.

#Local LLM #ChatGPT #GPT-OSS-120B #Qwen3 #MoE #llama.cpp #VSCode