Skip to main content
All posts

Blog

Best models to run on RTX Pro 6000 96GB

A task-by-task guide to choosing open models for a 96GB RTX Pro 6000 on EcoHash: coding, general chat, embeddings and rerankers for RAG, and speech models, all through an OpenAI-compatible API.

EcoHash Team7 min read

The right model depends on the task, not on the GPU. A 96 GB RTX Pro 6000 holds a 20B to 35B open model with room for context and batching, so you get to pick by the job instead of by a memory ceiling. For coding and agents that means Qwen3 Coder 30B; for low-cost chat, gpt-oss 20B; for retrieval, an embedding model with a reranker; for voice, Whisper and Kokoro. All of them run through the same OpenAI-compatible API at https://api.ecohash.com/v1, so you start on the shared API and move to a dedicated endpoint or a workspace later without touching your code.

Pick by task, not by memory. Coding: qwen3-coder-30b-a3b-instruct. Chat: gpt-oss-20b. Retrieval: jina-embeddings-v3 + bge-reranker-v2-m3. Voice: whisper-large-v3-turbo + kokoro-82m.

Model by task

  • Coding, agents, tool use · Model on EcoHash: Qwen3 Coder 30B · Model ID: qwen3-coder-30b-a3b-instruct

  • General chat, low cost · Model on EcoHash: gpt-oss 20B · Model ID: gpt-oss-20b

  • General chat, higher capability · Model on EcoHash: GLM 5.2, DeepSeek V4 Flash · Model ID: GLM-5.2, DeepSeek-V4-Flash

  • Embeddings for RAG · Model on EcoHash: Jina v3, Qwen3 Embedding · Model ID: jina-embeddings-v3, qwen3-embedding-8b

  • Reranking for RAG · Model on EcoHash: BGE reranker v2 m3 · Model ID: bge-reranker-v2-m3

  • Speech to text · Model on EcoHash: Whisper Large V3 Turbo · Model ID: whisper-large-v3-turbo

  • Text to speech · Model on EcoHash: Kokoro 82M · Model ID: kokoro-82m

The higher-capability chat models run through the hosted API, not on a single 96 GB card. Here is what the single-card models measure on EcoHash, end-to-end, in July 2026:

  • Coding · Model: qwen3-coder-30b-a3b-instruct · Measured on EcoHash: 30 ms TTFT, ~110 tok/s, 32k context

  • General chat · Model: gpt-oss-20b · Measured on EcoHash: 170 ms TTFT, ~125 tok/s

  • General chat · Model: llama-3.1-8b-instruct · Measured on EcoHash: 205 ms TTFT, ~143 tok/s

  • Speech to text · Model: whisper-large-v3-turbo · Measured on EcoHash: WER 4.37, ~0.5 s per 10 s of audio

  • Text to speech · Model: kokoro-82m · Measured on EcoHash: first audio ~120 ms

Against other providers of the same open models, EcoHash serves them at full BF16 precision and lands competitively on price and speed (llama-3.1-8b-instruct, EcoHash in purple):

Llama-3.1-8B price vs speed: EcoHash vs peers

Full data and charts: ecohash-benchmarks.

Facts

  • GPU: NVIDIA RTX Pro 6000 Blackwell Server Edition, 96 GB GDDR7

  • Single-card sweet spot: 20B to 35B open models

  • API base URL: https://api.ecohash.com/v1

  • Access: Shared API, dedicated endpoint, GPU workspace

  • Price: RTX Pro 6000 at $1.89 per GPU-hour (/pricing)

Who this is for

  • Developers choosing an open model for a coding, chat, retrieval, or voice feature.

  • Teams deciding what to run on a 96 GB card versus what to call through a hosted API.

What you can decide

  • Which model to start with for your task.

  • Whether it fits on a single RTX Pro 6000 or is better used through the hosted API.

  • When to move from the shared API to a dedicated endpoint or a workspace.

How to choose

Four questions get you most of the way:

  1. What is the task? Code, chat, retrieval, or speech. Start here, not with the hardware.

  2. Does it fit in 96 GB at your context length and batch size? A 20B to 35B model fits with room to spare, models near 70B need quantization, and 235B and up are better through the hosted API.

  3. What are your latency and throughput needs? Steady, high-volume traffic is the reason to reserve a dedicated endpoint.

  4. Do you need to self-host at all? For most teams the shared API is enough, and the GPU is EcoHash's problem, not yours.

Coding

Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct) is the default for code generation, refactoring, and tool-using agents. It is a Mixture-of-Experts model with about 30B total parameters, it fits comfortably on one 96 GB card, and on EcoHash it serves a 32,768-token context at about 110 tokens/sec single-stream with a 30 ms time to first token. See /models/qwen3-coder-30b-a3b-instruct.

For lighter or cheaper high-volume coding, gpt-oss 20B (gpt-oss-20b) is the smaller option. The three deployment paths for Qwen3 Coder are covered in Qwen3 Coder 30B on RTX Pro 6000.

General chat

Start with gpt-oss 20B (gpt-oss-20b) for low-cost chat and assistants. It fits on one card, has a long context, and reaches about 8,900 output tokens/sec under load.

For stronger general capability, EcoHash also serves larger models such as GLM 5.2 (GLM-5.2) and DeepSeek V4 Flash (DeepSeek-V4-Flash), and very large Mixture-of-Experts models like Qwen3-235B-A22B (Qwen3-235B-A22B). Those are not single-card deployments. Use them through the shared API or a dedicated endpoint and let EcoHash handle the hardware.

Retrieval (RAG)

A retrieval pipeline needs two models: an embedding model to index and search text, and a reranker to reorder the top results.

  • Embeddings: jina-embeddings-v3, jina-embeddings-v4, or the Qwen3 Embedding family (qwen3-embedding-8b, qwen3-embedding-4b, qwen3-embedding-0.6b).

  • Reranking: bge-reranker-v2-m3.

All of them run through the same key, so one service can embed, rerank, and generate the answer without stitching vendors together. See the RAG use case.

Voice

A voice agent uses three model types: speech to text, a chat model, and text to speech.

  • Speech to text: whisper-large-v3-turbo or qwen3-asr-1-7b.

  • Text to speech: kokoro-82m or qwen3-tts.

These speech models are small and do not need an RTX Pro 6000. They run through the API and move onto GPU capacity only when throughput grows. The full build is in Build a voice agent with Whisper, Kokoro, and an OpenAI-compatible API; see also /use-cases/voice-speech, /models/kokoro-82m, and /models/whisper-large-v3-turbo.

Calling these models

The API is OpenAI-compatible. A coding call:

from openai import OpenAI

client = OpenAI(base_url="https://api.ecohash.com/v1", api_key="eco_...")


resp = client.chat.completions.create(
model="qwen3-coder-30b-a3b-instruct",
messages=[
{"role": "user", "content": "Refactor this to be iterative:\n\ndef fib(n):\n    return n if n < 2 else fib(n-1) + fib(n-2)"}
],
)
print(resp.choices[0].message.content)

An embeddings call for a RAG index:

emb = client.embeddings.create(
model="jina-embeddings-v3",
input=["EcoHash serves open models through an OpenAI-compatible API."],
)
print(len(emb.data[0].embedding))

Shared API, dedicated endpoint, or workspace

  • Shared API: no GPU to manage, pay per token, good for building and variable traffic. See /inference.

  • Dedicated endpoint: reserved capacity for one model, steadier throughput and more predictable latency. See /dedicated-inference.

  • GPU workspace: an hourly RTX Pro 6000 for experiments, fine-tuning, and batch jobs. See /gpu-compute.

Limitations

  • Model fit depends on quantization, context length, and batch size. The ranges here are starting points to validate, not guarantees.

  • The higher-capability chat models are for the hosted API, not single-card serving.

  • Real performance depends on your setup. Reproduce any figure with its method and date.

  • Model IDs change as the catalog changes. Check /models before you build.

FAQ

What is the best model to run on an RTX Pro 6000 96GB? It depends on the task. Coding: qwen3-coder-30b-a3b-instruct. General chat: gpt-oss-20b. Retrieval: jina-embeddings-v3 with bge-reranker-v2-m3. Speech: whisper-large-v3-turbo with kokoro-82m.

What is the best coding model for the RTX Pro 6000? Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct). It fits on one 96 GB card and serves a 32,768-token context on EcoHash.

What is a good low-cost general chat model? gpt-oss 20B (gpt-oss-20b). It fits on one card and has a long context.

Can I run a 235B model on one RTX Pro 6000? No. Use a model like Qwen3-235B-A22B through the hosted API instead.

What models do I need for RAG? An embedding model and a reranker, for example jina-embeddings-v3 and bge-reranker-v2-m3, plus a chat model to write the answer.

Do speech models need an RTX Pro 6000? No. Whisper and Kokoro run through the API, and move onto GPU capacity only when throughput demands it.

How many models can I use with one API key? All of them. Every model shares one key and one base URL, https://api.ecohash.com/v1.

How much does this cost? API usage is billed per token; a GPU workspace runs at $1.89 per GPU-hour. See /pricing for current rates.

Next steps