Nemotron Labs Dory 1.0 4B on an 8 GB GPU: What Fits and What Does Not

TL;DR: Nemotron Labs Dory 1.0 4B has about 4 billion parameters and a 256K-token context. Its BF16 weights need about 8.0 GB, the full capacity of an 8 GB GPU. The GGUF Q5_K_M file is 3.67 GB, leaving room for a KV cache. Speed on an RTX 5050 is unverified: NVIDIA’s tested hardware excludes that card.

Illustrative scenario: A developer has a desktop with an 8 GB GPU and 16 GB of system RAM. They want a small model for code review that runs offline. The model card lists a 256K context and a Q5_K_M file that is under 4 GB, but the same card lists H100, B200, B300 and DGX Spark as tested hardware. The question is whether the small file is enough.

Thesis: On an 8 GB card, the quantization format decides whether Dory 1.0 4B fits. BF16 does not fit with any working room. Q5_K_M leaves about 4 GB for the KV cache and runtime buffers. Speed on consumer hardware is the open question, and this article does not answer it with a measurement.

Units in this article: GB means 10^9 bytes, decimal only. Weight sizes are estimates from the parameter count unless marked as a file size. Parameter count is nominal (“4B”), as stated on the model card.

Why an 8 GB card is the real test for a 4B model

A 4B model is small by current standards, but an 8 GB card is also small. The memory a GPU must hold is the sum of three parts: the weights, the KV cache that grows with context, and runtime buffers. The weights are fixed once the format is chosen. The KV cache depends on context length and on how the model stores attention state. Runtime buffers depend on the software.

The NVIDIA specification page for the GeForce RTX 5050 desktop card lists 2,560 CUDA cores, 8 GB of GDDR6 memory on a 128-bit interface, and a 130 W total graphics power. The page does not list memory bandwidth, so this article does not compare bandwidth between cards. The 8 GB figure is the constraint that matters for model fit.

Dory 1.0 is a looped model. The model card describes eight input layers, twenty recurrent layers that run more than once per token, and eight output layers. This matters for compute, and it matters for the KV cache, as the next section explains.

Nemotron Labs Dory 1.0 4B weights first: the memory budget by format

The first step is the weight footprint. Table 1 uses the parameter count for the estimates and the measured file size for Q5_K_M.

The arithmetic is simple, and that is the point. Each format stores a fixed number of bytes per parameter, so the weight size scales linearly with the model. For a 4B model, the gap between BF16 and a 4-bit format is roughly a factor of four, and that gap decides whether the model fits on the card at all.

The weight file is the part of the budget you can know before you download anything. Everything else depends on the runtime and the context you choose.

Source: NVIDIA model card for the BF16 repository, and the GGUF repository on Hugging Face (total file size 3,673,288,864 bytes, read November 1, 2026); estimates from our calculation with calcs/footprint.py. Takeaway: BF16 has no headroom on an 8 GB card, so the format choice is the first decision.

The BF16 row is a lower bound. It counts only the parameters at 2 bytes each. Real BF16 checkpoints add tensors that the formula does not model, so the true weight size is at least 8 GB. On an 8 GB card with a desktop that also uses video memory, BF16 is not a realistic option.

The Q5_K_M file is the practical choice for this card. Its measured size leaves about 4.3 GB. The NVFP4 and MXFP8 rows are only estimates. NVIDIA publishes NVFP4 and MXFP8 repositories, but the card does not state their file sizes, so their real sizes are not verified here.

The file size also deserves a note. At 4B nominal parameters, 3.67 GB works out to about 7.35 bits per parameter, higher than a typical Q5_K_M model. This suggests that the quantized file keeps some tensors at higher precision, or that the real parameter count is higher than the nominal figure. The model card does not explain this, so this article reports the file size and does not infer the bit width from it.

Looped layers change compute, not the weight file

Dory’s looped design reruns the recurrent block one to four times per token. The model card says the loop count trades compute for accuracy. Two loops is the suggested default, one loop uses the least compute, and four loops gives the most accuracy.

The weight file does not change with the loop count, because the same block runs again. The memory cost of looping sits in the KV cache. The card says per-loop caching is the default in its examples. Per-loop caching stores attention state for each loop, so the cache grows with the number of loops. The card also describes last-loop caching, which keeps the cache at the size of one loop.

This article cannot give a KV-cache size for Dory. The card does not state the hidden size, the number of attention heads, or the head dimension, and those values are needed to size the cache. Without them, any KV-cache number would be invented, so this article gives none. The practical rule is to measure the cache with the runtime at the context length you plan to use. For how cache size scales on a larger model, see KV cache memory for Llama 3 on H200. The 4.33 GB headroom from Q5_K_M is the budget for that cache and for runtime buffers.

What the benchmarks show, and against which models

The benchmark table is the only quality evidence the model card provides, and it matters for the choice of loop count.

The table below is read the same way as the memory budget: each number is a claim from the model card, and the comparison is only as fair as the harness NVIDIA used.

NVIDIA reports benchmark scores that it measured under its own harness. The model card also includes some reported scores, marked separately. The comparison set is Qwen3.5-4B, Qwen3.5-9B, Gemma 4 26B A4B and Qwen3.6-35B-A3B. Table 3 shows the 1-loop, 2-loop and 4-loop results for Dory against Qwen3.5-4B, which is the closest size.

Source: NVIDIA model card for Nemotron Labs Dory 1.0 4B (BF16 repository, read November 1, 2026). Scores measured by NVIDIA under its harness, BF16 precision. Takeaway: Dory gains most from the second loop on coding and instruction-following tests, while Qwen3.5-4B stays ahead on GPQA Diamond.

Two points stand out. First, the gains from looping are real in NVIDIA’s table, and most of the gain comes between one and two loops. Second, the comparison is mixed. Dory leads on IFBench and AA-LCR at two loops, but Qwen3.5-4B scores higher on GPQA Diamond at every loop count.

The benchmark figures are not independently verified. Some outlets covering the release note the same caution. Treat Table 3 as NVIDIA’s claim, to be checked on your own tasks before you rely on it.

Running it on an RTX 5050: what is verified and what is not

The model card lists tested hardware as H100, B200, B300 and DGX Spark. The RTX 5050 is not on that list. The card lists Hopper and Blackwell as the supported microarchitectures. The RTX 5050 is a Blackwell-generation desktop card, but support on a specific card is not the same as a tested configuration.

The software path is also incomplete at the time of writing. The card says vLLM support is under review in a pull request and is not in a release, with installation from that branch. For llama.cpp, the card pins a specific revision of a pull request and uses the Q5_K_M file. A reader who installs a release build may not have Dory support. Check the release notes of the runtime before you plan around it.

The card’s examples use Linux as the preferred system. Apple Silicon has a separate llama.cpp build path. Windows is not discussed.

If you run the Q5_K_M file on a card with 8 GB of VRAM and 16 GB of system RAM, any layers that do not fit in VRAM must run from system RAM. The card does not give speed numbers for that split on consumer hardware, so this article gives none. The DGX Spark speedups in the card, 1.69× to 2.44× over BF16 for the MXFP8 and NVFP4 formats, are measured on a different system and should not be read as RTX 5050 results.

Source: NVIDIA model card, Hugging Face repository metadata, NVIDIA RTX 5050 specification page, read November 1, 2026. Takeaway: the memory fit is verified on paper; speed and software support are not.

Worked example: a 2-loop session on Q5_K_M

Take the Q5_K_M file as the weights. Start with 8.00 GB of VRAM, reserve the driver and desktop allowance (not measured here, so this step is left open), and subtract the 3.67 GB file. That leaves 4.33 GB before the desktop reserve.

Next, the KV cache. The cache has to hold attention state for the context length you plan to use, multiplied by the loop count if per-loop caching is in use. Because NVIDIA does not state the hidden size or head count, the cache size cannot be calculated from this card. Measure it with your runtime at your target context.

Last, check the context you need. A 4K-token session needs far less cache than a 256K-token session. If the measured cache plus the driver reserve stays under 4.33 GB, the session fits on the card. If it does not, reduce the context, use one loop, or move layers to system RAM at a speed cost you must measure.

The counterargument: a small model does not need a big GPU

The strongest objection is that a 4B model is small enough to ignore the GPU and run on the CPU. For short prompts and low throughput, that is often true, and a 16 GB system can hold the Q5_K_M file. The GPU matters when throughput matters: longer generations, several users, or agent loops that make many calls.

The response is that the card does not decide the question. The question is the throughput your task needs. NVIDIA reports loop-dependent accuracy gains, and those gains cost compute. A CPU-only setup may not run two loops at a usable speed, and this article does not have a measured number to settle it. What the card does establish is that the model fits an 8 GB GPU in Q5_K_M, and that fit is the reason to consider the GPU at all.

What this analysis cannot tell you

The primary limitation is speed. No tokens-per-second figure is measured on an RTX 5050 or on any 8 GB card in this article. Every speed claim is missing until someone runs the model on the card.

The second limitation is the benchmark data. The scores come from NVIDIA’s harness and are reported in BF16. A quantized Q5_K_M file may score differently, and this article did not run any evaluation.

The third limitation is software. The vLLM path is not in a release at the time of writing, and the llama.cpp path depends on a pinned pull request. Support can change before you read this, so check the runtime release notes first.

The fourth limitation is the KV cache. The card does not give the architecture values needed to size it, so headroom for long contexts is unknown.

Decision framework by role

Individual developer with an 8 GB GPU. Use Q5_K_M. Keep the context under the length you have measured. Use one or two loops and measure the speed before you commit to a workflow. A threshold to watch: if the measured cache plus reserve exceeds the 4.33 GB headroom in Table 1 (estimate), shorten the context.

ML platform lead. Do not standardize on Dory 1.0 4B until the runtime you use ships support in a release. Test on your own tasks, because NVIDIA’s benchmark set does not cover your workload. A threshold: adopt only if the measured accuracy on your evaluation set is within the margin you accept against the model you run now.

DevOps or SRE. Plan for the runtime version as a dependency. Pin the version, because the support path is a pull request. Budget system RAM for any layers that run off the GPU.

Engineering manager or FinOps. An 8 GB consumer card is a low-cost test platform for a 4B model, but the cost of the test is the engineer’s time to verify speed and accuracy. Do not treat the hardware cost as the result.

FAQ: edge cases for running Dory 1.0 4B on small GPUs

Sources & further reading

• Nemotron Labs Dory 1.0 4B, BF16 model card (Hugging Face): parameter count, context length, looped architecture, tested hardware, loop trade-off, benchmark table.
• Nemotron Labs Dory 1.0 4B, GGUF repository (Hugging Face API): total file size of 3,673,288,864 bytes and the Q5_K_M file name.
• GeForce RTX 5050, NVIDIA: 2,560 CUDA cores, 8 GB GDDR6, 128-bit interface, 130 W total graphics power.
• llama.cpp Dory support pull request: Dory support, as referenced by the model card.
• vLLM Dory support pull request: Dory support under review, as referenced by the model card.

Related reading

• The Complete Hardware Guide for Running Powerful AI Models Locally (2026): the broader hardware tiers for local inference.
• vLLM vs SGLang vs TensorRT-LLM 2026: how the serving runtime changes throughput on the same hardware.
• KV Cache Memory: Llama 3 on H200 2026: how cache size grows with context and sequence count.
• H200 vs B200 vs H100 Cost per Token 2026: how throughput converts into cost per token on data-center cards.
• Edge AI vs Cloud GPU Inference 2026: when local inference is the better choice and when it is not.

Author: Iovanny Olguín Ávila

Computer Systems Engineer with an MSc in Computer Science. I apply quantitative analysis and data-driven methodologies to evaluate financial instruments, investment vehicles, and emerging technologies. My technical background allows me to cut through marketing language and analyze the actual mechanics of financial products — from HELOC structures to Medicare Advantage plan design to business credit card reward algorithms.