How to benchmark local AI on an RTX 5090

A model can fit on an RTX 5090 and still feel slow. If you ask it to review a long diff, the wait before the first useful line might matter more than how quickly the rest of the answer arrives. If it is extracting fields from hundreds of short documents, throughput and valid output matter more. Choose a task before choosing what to measure.

For a coding assistant, I would collect three prompts from work I actually do. A short debugging question, a review of a representative diff, and a request that includes several files. For each prompt, write down what a useful answer looks like before you test.

The RTX 5090 has 32 GB of GDDR7. The model weights take up part of that budget. The KV cache, which holds attention state for tokens already processed, takes more as context grows. Runtime buffers and other GPU workloads also compete for space. Serving multiple requests can increase memory pressure further.

Qwen’s published Qwen3-32B GGUF files give a concrete comparison. The Q4_K_M file is 19.8 GB and the Q8_0 file is 34.8 GB. Those are download sizes. The smaller file gives you room to test on a 32 GB card. The larger file is already too big to keep the whole model on that card, even before context and buffers. If you offload layers to the CPU, measure how much that changes the experience. A lower-precision model is worth testing against your actual quality bar.

Run each prompt once to warm up the model. Then record three comparable runs with the same model version, quantization, runtime, context setting, and output cap. Keep prompt-cache conditions consistent. Record cold startup separately if you regularly unload and reload the model.

• Time to first token tells you how long the request takes to start streaming. Test it with both a short prompt and the long input you actually expect.
• Generation tokens per second describe the stream after it starts. Use the runtime’s decode timing where available.
• Peak VRAM and CPU offload tell you whether the run leaves room for your browser, editor, or a second request. Sample GPU memory during the test and check the runtime logs for layer placement.
• Output checks tell you whether the run did the job. For a diff review, did it find the seeded bug and cite the right file? For extraction, did it return valid fields you can use?

Start with a context limit you can justify from your prompts. Increase it and rerun the long input. Then go back to your starting context and change the quantization. Keep concurrency at one until you understand the single-request numbers. If you intend to share the machine, rerun the winning configuration with two requests at once.

The configuration I would keep is the one that answers the long prompt within a tolerable wait, passes the output checks, and leaves enough GPU memory for the rest of the workstation. That comparison is more informative than a single peak speed from a short prompt.

NVIDIA GeForce RTX 5090 specifications

https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/

Qwen’s published Qwen3-32B GGUF variants

https://huggingface.co/Qwen/Qwen3-32B-GGUF

llama.cpp server options for context, offload, and cache settings

https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md