The open-source llama-modchip project runs DeepSeek-V4-Flash with 284 billion parameters on a computer with 64 GB of RAM, without using a graphics card. Imagine Vasyl, a pig engineer who wants to keep a language model on his own workstation: his first constraint here won’t be the number of parameters itself, but the size of the weights that have to fit in memory. The project’s authors compress them to an average of 1,72 bits per weight, at the cost of slower performance and a noticeable loss of quality compared with the reference version.

In release 2.5.3, published on November 13, the authors rebuilt the weights for 284B directly from the original DeepSeek checkpoint and changed the unsafe handling of temporary files. Release 2.5.4 came out the same day, with fixes for the benchmark, chat, and Windows builds. The project is distributed under the Apache 2.0 license and consists of patches for llama.cpp, but the quantized files created for it can only be read by its own engine.

Why 284 billion parameters doesn’t mean 284 billion are active

DeepSeek-V4-Flash is an MoE model, or mixture of experts. For each token in each layer, six of the 256 routed experts and one shared expert are engaged. So roughly 13 billion parameters are active, but all 284 billion weights still need to be available in memory. This is where compression comes in: a smaller set of weights means less data for the processor to read during generation.

To do this, llama-modchip adds two quantized block formats, each with 64 weights. Fractal-64 stores weights at 3,75 bits per weight. LUT-VQ3-64 uses 3 bits: it encodes groups of three weights as an index into a table of 128 points. The table fits into six 512-bit AVX-512 registers, and its values are decoded by the VPERMI2B instruction without cache accesses.

But these compact formats aren’t applied uniformly to every part of the model. Fractal-64 and LUT-VQ3-64 are used for the MLA attention mechanism, the indexer, and some of the shared experts. The bulk of the model—the routed experts—is compressed even more aggressively, bringing the average to 1,72 bits per weight. MaxQ-Fractal also has 174 dense tensors in formats Q6_K and Q8_0, 317 attention tensors in Fractal-64, and 129 expert tensors quantized with an importance matrix.

As a result, the MaxQ-Fractal file itself takes up 56,93 GiB, while a process with a 32-thousand-token context uses 57,5–58,1 GB of RAM. Vasyl would have to account for more than just the file size: on a machine with 64 GB, about 62,5 GB is available to the system, so there isn’t much headroom. A pig-config with a large context quickly turns into a question of available memory, not simply whether the file loaded.

Speed: suitable for background tasks, not instant autocomplete

The authors’ main test was run on a 16-core Ryzen 9 9955HX with dual-channel DDR5-5600, whose read bandwidth is around 75 GB/s. With all six experts per layer engaged, generation reached 8,66 tokens per second for a single request without speculative decoding. Prompt processing from a cold start was 36,1–37,6 tokens per second, and 44 with a warmed-up prefix.

These figures have different practical implications. Generation at around nine tokens per second may work for a background agent that doesn’t need to respond instantly. But processing a long input takes a while: the authors estimate that 10 thousand prompt tokens at this speed take at least four minutes. For agents, they cache the system-prompt prefix in /dev/shm so it can be restored in 10–30 ms. That cuts down on repeating work on the prefix; it doesn’t make just any arbitrary large prompt faster.

The mode with four experts per layer, except for the first three layers, plus an additional SwiGLU optimization raises speed on the Fractal quantization to 10,1–10,7 tokens per second. There’s a trade-off for that gain: according to the authors, the deviation from the original distribution reaches a KL divergence of 0,07 nats per token. The mode must be enabled manually and resets when the server restarts.

Reports from other users show that results depend on the processor and configuration. On a Ryzen 7 9800X3D with 64 GB on Windows 11, users reported 6,35 tokens per second with all experts and 8,04 with four. On a Ryzen 9 9950X3D with 128 GB, generation was 8,11 tokens per second. Apple M5 Max posted higher figures, but this wasn’t a CPU-only test: the model ran on the GPU and CPU together, with a substantial portion using Metal.

The drafter speeds up predictable text, but not every answer

llama-modchip has a speculative decoding mechanism that doesn’t use a separate small draft model. Its Stockfish-PVS keeps tables of likely continuations in the style of chess engines, updating them during the conversation. This approach works best where text is repetitive or predictable: when editing code, calling tools, or generating JSON.

For DeepSeek-V4-Flash, the authors’ average test result with the drafter is 8,80 tokens per second, with 78,7% of draft tokens accepted. On some short code snippets and tool calls, speed temporarily rose to 11–21 tokens per second, and to 25,3 for the VQ3 variant. These are peak figures on predictable text, not a steady generation speed for just any answer.

Compactness comes at a cost in quality, and compatibility is limited

On the wiki-alt corpus test, MaxQ-Fractal’s perplexity is 7,30 versus 4,54 for the reference MXFP4 quantization. That’s 60,8% higher. At the same time, the reference MXFP4 takes up 150,8 GiB and won’t fit on a system with 64 GB of memory. The authors warn that they’re comparing their own CPU results with AtomicChat’s published GPU estimates, and that the methodologies don’t fully match.

In the developers’ own functional test, the model scored 70 out of 80: full marks for tool calling, finding a fact in a long context, and math, but 20 out of 30 for code generation. This is the authors’ own test; the source gives no independent assessment of code quality on standard benchmarks. So Vasyl shouldn’t take the figure 70 as independent confirmation that the compressed model is equivalent to the full-size one.

There’s also a hard limit on software compatibility. Fractal-64 is assigned tensor type number 42, which is already used for a different format, Q2_0, in the main llama.cpp branch; LUT-VQ3-64 adds type 43. So llama-modchip GGUF files won’t open in regular llama.cpp, Ollama, vLLM, or LM Studio. The advantage of local inference doesn’t mean that every familiar client will automatically pick up these weights.

What you need to run it, and where to be careful

For the recommended context of 32 thousand tokens, the authors recommend enabling zram and closing the browser to free up memory. They recommend a dedicated server with zram, or a system with at least 96 GB, for a context of 256 thousand tokens. Although the model supports up to a million tokens, one user reported running it with that context on a Ryzen 7 9800X3D with 64 GB; that’s a single report, not a guarantee of stable operation on any computer.

The ready-made Linux binaries are intended for x86_64 with glibc 2.30 or later and require libgomp. Zen 4 and Zen 5, as well as Intel processors from Ice Lake onward, require an AVX-512 build with the avx512vbmi instruction. For older x86-64-v3-class processors, an AVX2 version is available; on even older ones, the ready-made binaries won’t run. For Apple Silicon, build scripts with Metal are provided; on Windows 10 and 11, the engine needs to be built from source using MinGW-w64.

Before running it, it’s worth considering the history with temporary files. In versions 2.5.1 and 2.5.2, routing settings were written to /tmp/dsv4_moe_cfg.bin with permissions 0666, meaning any user on the machine could modify the file. The authors warned against installing those versions on multi-user hosts. In 2.5.3, shared paths were removed, and settings and shared memory became separate for each user; files are created with permissions 0600.

So llama-modchip makes it possible to run a very large MoE model locally without a separate graphics card, but only in exchange for dense quantization, demanding memory management, and its own engine format. For Vasyl, it could be a workable setup for unhurried background tasks, if he accepts lower quality and long waits on large prompts. For interactive autocomplete, though, the key limitations are still slow input processing and limited generation speed.