How to deploy and tune Qwen3.8-27B on one RTX Pro 6000

Qwen3.8-27B is 27B dense, published in FP8 under Apache-2.0, and the weights plus runtime overhead come to about 29 GiB. A single RTX Pro 6000 holds it with a large KV cache still to spare, on one process, with no tensor parallelism to configure. This post is the serving config EcoHash runs in production, the settings whose obvious alternative fails, and the one flag that took a single card from 999.9 to 1954.0 output tokens/sec.

Everything needed to run it well is in the download, including the draft head that doubles what one card serves.

Why this model suits self-hosting

At FP8 the checkpoint is 28.75 GiB on disk, and vLLM reports 29.34 GiB of weights and non-torch overhead once loaded. On a 96 GB card that leaves 53.8 GiB for KV cache, which is 767,317 tokens. Nothing has to be split across devices and nothing runs close to the edge.

Apache-2.0 means no usage terms to read. Context is 262,144 tokens natively, extensible to 1M. The official model card reports 89.2 on GPQA Diamond, 90.3 on LiveCodeBench v6, 61.7 on SWE-bench Pro, and 73.0 on Terminal Bench 2.1. Artificial Analysis puts it at 52 on their Intelligence Index, level with GPT-5.6 Luna and one point behind GLM-5.2 (753B) and DeepSeek V4 Pro (1.7T). Downloads passed three million within days of the 14 August 2026 release.

The checkpoint also carries its own multi-token-prediction draft head, 22 tensors and 0.44 GiB, in the same download. Speculative decoding normally means training or sourcing a second model; here it is a flag.

Plan for one quirk. Artificial Analysis needed 160 million output tokens to run this model through their index, against a 43 million median for comparable open-weight models, and Simon Willison found the same thing: it is excellent, but it defaults to wildly overthinking things. Because it reasons at length by default, decode throughput rather than prefill is what repays tuning.

The serving config

This is what runs in production, on vLLM 0.27.1 with driver 580.105.08 and CUDA 13.0.

vllm serve /path/to/Qwen3.8-27B-FP8 \
  --served-model-name qwen3.8-27b \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --safetensors-load-strategy prefetch \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Five of those lines are there because the obvious alternative failed.

• --dtype bfloat16, not float16. FP8 checkpoints dispatch to kernels that assert a bf16 output dtype. With float16 the engine loads every weight, reports success, then crash-loops at warmup. It cost a failed rollout on a sibling model before anyone connected the two.

• No --quantization flag. The checkpoint is already FP8 (e4m3, weight block size 128x128), so vLLM reads quantization_config from config.json and loads straight through. Passing the flag anyway makes it quantize at runtime on every start: 2105 seconds of weight loading against 17 seconds pre-quantized.

• --safetensors-load-strategy prefetch if the weights sit on network storage. vLLM does not always recognise a network filesystem, and sequential loading falls off a cliff past the eighth shard, from 94 seconds to 370 seconds per shard. This checkpoint is sharded per layer, so there are 66 of them.

• --tool-call-parser qwen3_xml, not hermes. Qwen3 emits its native XML tool format rather than Hermes JSON. With the wrong parser you get the raw <tool_call> text and tool_calls: null, and every agent integration breaks quietly.

• --max-num-seqs 128. This is a dense 27B, so every token touches every weight and the batching knee arrives earlier than it does on an A3B MoE, where 160 is the better setting.

--max-model-len 32768 is a deliberate cap rather than a limit of the model. It keeps the KV cache bounded when the card is shared. Raise it after checking the memory numbers, not before.

The flag worth tuning for

The draft head proposes several tokens, the model verifies them in one pass, and the accepted ones come out together. num_speculative_tokens, K below, is how many it proposes. Deeper drafts win more per step and get rejected more often, so K has an optimum that moves with the workload.

K=3 wins on all three workloads, so that is the setting. Returns are already flattening on chat, where K=2 to K=3 buys 6%, and the acceptance rate is what separates the workloads: predictable, low-entropy output drafts well, open-ended prose does not.

The effect holds as the card fills up. The dotted line is the 30 ms per-token budget that defines a usable interactive response.

Per-token latency drops from 22 ms to 13 ms on a single stream, and the number of concurrent requests one card holds inside that budget goes from 32 to 64. Aggregate output at the ceiling goes from 999.9 to 1954.0 tokens/sec. Measured in August 2026 on one card at 150-token replies; longer generations shift the numbers, and the full sweep is in the write-up.

The cost is cache. The draft head brings an attention layer that needs KV of its own, so capacity falls from 767,317 tokens to 592,164, a 23% cut. At 48 to 64 concurrent short requests that is not the binding constraint, but it would become one before pushing concurrency much higher.

The full write-up has the draft-depth sweep, acceptance rates by position, the whole concurrency table, the memory accounting, and the quality checks: Qwen3.8-27B with MTP speculative decoding. The raw CSVs are in the same repository.

Use the API instead

All of the above is already running. Calling qwen3.8-27b on EcoHash gets the tuned deployment with nothing to provision: 179 ms to first token, about 78 tokens/sec on a single stream, $0.30 per 1M input tokens and $0.60 per 1M output.

Peer numbers are as published by OpenRouter in August 2026. Time to a finished answer is the first token plus the rest of the stream, so a 500-token answer is roughly one step of an agent loop, and the whole field except one provider needs at least 1.7 times as long as EcoHash to produce it.

Blended at three parts input to one part output, that is $0.375 per 1M tokens against $1.05 to $1.21 elsewhere. Almost all of the gap is the output side, $0.60 against $3.00 to $3.40, and output is the side a reasoning model spends on. An agent making 400 requests an hour at 800 input and 2,000 output tokens each costs $0.58 an hour here and $2.70 at $0.45 / $3.20.

The API is OpenAI-compatible, so only the base URL and the model name change.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ecohash.com/v1",
    api_key="eco_...",  # create a key at console.ecohash.com
)

resp = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[
        {"role": "user", "content": "Refactor this function and explain the change."},
    ],
    stream=True,
)

for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Tool calling and structured outputs work, and images go in the same messages array. The shared API serves a 32k context rather than the checkpoint's 262k, which is what keeps the KV cache bounded on a shared card.

Or run it yourself on the same card

If the config above is what you want but the hardware is not, an RTX Pro 6000 workspace rents by the second at $1.89 per GPU-hour as of August 2026, with root access and a browser terminal. Pull the FP8 checkpoint, paste the vllm serve line, and the same deployment is yours to change: the full 262k context, your own --max-num-seqs, no shared-tenancy cap.

Which is the argument for self-hosting this particular model. One card covers it, and the tuning that moves the numbers is a flag you can read in a config file.

FAQ

Is this the full model or a quantized version? FP8, which is how the Qwen team publishes it. The precision is stated because not every provider states theirs.

Do I need to do anything to get the speculative decoding speedup on the API? No. It is on for every request to qwen3.8-27b.

Will it fit on a 48 GB card? The weights alone are about 29 GiB, so it loads, but the KV cache left over is small enough that concurrency and context both suffer. The 96 GB card is what makes the deployment comfortable rather than merely possible.

Does MTP change the output? Verification means the accepted tokens are the ones the model would have produced anyway. Answers matched on every checkable question in our quality pass; the write-up explains why a byte-for-byte comparison is not the right test.

Does it accept images? Yes. It is a vision-language model and takes images in the standard OpenAI content format.

Next steps

• Call the model: /models

• Rent the card and run the config above: /gpu-compute

• Read the benchmark: Qwen3.8-27B with MTP speculative decoding

• Reserve steady capacity: /dedicated-inference