Services

Resources

Company

/

Self-hosted LLM

Self-hosted LLM

A self-hosted LLM runs on GPUs you control, in your own rack or cloud account. You download open weights such as Llama, Qwen or gpt-oss and load them into an inference server. Your apps then call an endpoint on your own network. Teams do this when data may not leave, when they need a fine-tuned model no provider will serve, or when volume is high and steady.

How it works

The stack has three layers. GPUs hold the weights and the KV cache. An inference server such as vLLM runs LLM inference behind an OpenAI-compatible API, so client code only changes its base URL. In front sits an API layer that checks API keys, routes each model name to a running instance and counts tokens per user.

One2N skipped that third layer at first. In October it added two NVIDIA DGX Spark machines for its 30 users. Anyone who wanted a model would SSH into a Spark and run vllm serve on a free port.

That broke as more people joined. Nobody knew what was running without logging in. With no keys, nobody could tell who sent traffic, and trying a new model often meant stopping someone else's.

The fix was GPUStack v2.2.3 in front of both Sparks. People sign in with Google SSO and get their own API key, which can expire and reach only chosen models. A scheduler places each model on a Spark with room. The same AI gateway routes Bedrock models too, so clients need one hostname.

Why it matters in production

You pay for GPUs by the hour whether they are busy or idle, so break-even depends on volume. The One2N cost post lists Opus 5.5 output at $20 per million tokens and Haiku 5.5 at $0.50. Say your GPU serves 1 million output tokens an hour. It then has to cost less than 1 x $20 = $20 an hour to beat Opus, and under $0.50 to beat Haiku. That second bar is why self-hosting rarely wins against a small API model on price.

Add the engineers and the on-call rota to the GPU bill. Check managed options first, too: Bedrock runs models inside your cloud provider's boundary, and many regulated teams accept that.

See also

On the One2N blog

References

  1. vLLM documentation

  2. GPUStack on GitHub

Stack diagram of a self-hosted LLM: 30 users call one hostname. The top layer, GPUStack, handles keys, routing and tokens per user; below it the vLLM inference server offers an OpenAI-compatible API; at the bottom, two DGX Spark machines hold the weights and KV cache. Below the stack: at 1M output tokens an hour, the GPUs must cost under $20 an hour to beat Opus and, in red, under $0.50 to beat Haiku.