How much context really fits in 32 GB
Context is the conversation's memory — everything the model has in front of it while forming an answer. After the model weights, it is the second-largest consumer of card memory, and it is usually what decides whether a configuration starts at all. We checked where the limit lies at 32 GB. The answer turned out to be surprising: most often it is not on the hardware side.
Nine profiles out of ten are limited by the model, not the memory
Our machine exposes ten ready-made working profiles. Nine of them stand at 262,144 tokens of context — and that is not the cards’ limit, but the maximum these models were trained on. Above that value there is no way in, no matter how much memory you add.
That is the most important conclusion of the whole series of measurements: at 32 GB, today’s models in the 26–35 billion parameter class fit at the full context they were designed for. More memory would not buy a single extra token here.
The only memory-limited profile is the 31-billion model with image and video support — the vision module and the helper model’s separate file eat up space, so context tops out at 167,936 tokens. It is the tightest configuration in the whole lineup: under full load, 464 and 144 MB remain free on the cards. The next step up no longer loads.
What context costs
The math is linear and easy to remember: about 200–320 MB on each card for every 8,192 tokens of context. The spread comes from the model’s architecture — with image support, the helper model’s memory is added on top. And why memory counts separately on each of the two cards is explained in the article on splitting the model between cards.
The spread between models can be large. A model with a hybrid attention design uses about 35 kB per token, while a dense model uses about 51 kB. For that reason the latter stops at 208 thousand tokens instead of 262 thousand, even though it would happily keep going on its own. The model’s architecture thus decides context capacity just as strongly as the amount of memory.
How we measured — and why the obvious method lies
Three things turned out to be crucial, and each of them is a trap that is easy to fall into.
Free memory must be read after work, not after loading. The compute library reserves its working buffer lazily — only at the first real matrix multiplication. A model can therefore “load” with an apparent reserve and fall over at the first question.
Free memory must be read directly, not calculated. The card’s driver keeps about 423 MB for itself that cannot be allocated. The popular “total minus used” subtraction overstates the result by exactly that amount — showing 528 MB free where 106 actually remain.
The reserve turned out to be static. We threw long queries at the configurations — from a thousand to 227,000 tokens, with model reloads and repeated processing of the full context. A sample every half second, 14,771 measurements. The variation in free memory: at most 2 MB, and on two configurations exactly zero. Memory is reserved up front and does not move.
That is why our acceptance threshold is a modest 100 MB free on each card. If usage jumped around during work, we would need a reserve measured in gigabytes. It does not jump.
The surprise: a smaller context can leave less memory
The most counterintuitive result of the whole series. On one of the models we measured free memory at three context settings:
| Context | Free on card 1 | Free on card 2 |
|---|---|---|
| 262,144 | 1372 MB | 416 MB |
| 212,992 | 536 MB | 60 MB |
| 180,224 | 1026 MB | 870 MB |
Coming down from the context made things worse. The explanation: at certain values the compute buffers do not fit, and the server falls back to a simpler working mode that saves more than the larger conversation memory costs. Each of these results reproduces to the megabyte on subsequent loads.
The practical conclusion: context is chosen by measuring on a grid of values, not by extrapolation. The rule “less means safer” does not apply here.
Two things that gave memory back for free
One slot instead of four. By default the server prepares for four simultaneous users and reserves buffers for them — even when there is only one user. Limiting it to a single slot gave back 240–770 MB per card, with no change to the context and no change in speed. The price is a queue: a second query waits for the first. At home, that is a fair trade.
A more economical format for the conversation’s memory. Context can be kept at full precision or compressed. Compression costs about 12% of that buffer’s accuracy and yields noticeably more room — we use it everywhere we are fighting for context. We rejected stronger compression: it degrades answer quality in a way you can feel.
How much is 262 thousand tokens
It is hard to picture, so two points of reference. A context this size holds several hundred pages of text at once — the entire documentation of a medium-sized project, or a full set of contracts to compare. In the programming test, where the model wrote multi-module projects on its own, peak context usage averaged 107,000–175,000 tokens. In other words: even under intensive work, there was headroom left.
For comparison — the only Polish model we tested tops out at 32,768 tokens. Not because of the cards, but because that is how it was designed.
Measurements on 2 × RTX 5070 Ti 16 GB. Free memory read after full load, acceptance threshold 100 MB per card.