Qwen 7B thread weighs GPU scaling

The real bottleneck here is not just model size; it is context length, KV cache growth, and queueing policy. A 7B model can look small on paper, but once you add long structured generations and concurrency, capacity planning becomes a serving problem, not a parameter-count problem.

• –Qwen's own benchmark data shows 7B BF16 memory can start around 14.9 GB and climb past 40 GB at long context, while int4 quantization lowers the base footprint but does not eliminate KV cache growth.
• –vLLM's guidance is to size by GPU KV cache and the "maximum concurrency" it reports at runtime; if that number misses your target, add GPUs or nodes instead of assuming batching will fix memory limits.
• –For an 8GB 4060, aggressive quantization, shorter max outputs, and tight request caps are the first levers to pull before buying hardware.
• –Batch more when latency budgets are loose and requests are similar; scale out with more GPUs when p95 latency is already high or when longer contexts make batching less effective.
• –Cloud works well for bursty demand and fast experiments, but steady production inference usually wants reserved capacity or on-prem GPUs for cost predictability.