Services

Resources

Company

How we built a self-hosted LLM platform with GPUStack

How we built a self-hosted LLM platform with GPUStack

How we built a self-hosted LLM platform with GPUStack

How we built a self-hosted LLM platform with GPUStack

At One2N, we run many kinds of ML models, such as LLMs, text-to-video (T2V) models and text-to-speech (TTS) models, to build prototypes and run experiments. We used to run them on Vertex AI, Amazon Bedrock and a few other platforms, each with its own console, API keys and usage view.

In October, we added two NVIDIA DGX Spark machines to our setup. We wanted everyone at One2N to share them without stepping on each other’s toes. At first, anyone who wanted to run a model would SSH into one of the Sparks and start it with vllm serve or docker run on any free port. That worked for a few of us, but it broke down as more people joined in:

  • Only people with SSH access could deploy, so handing out SSH access became a bottleneck.

  • Nobody knew what was running without logging in to check.

  • There were no API keys, so we could not tell who sent traffic or enforce any governance.

  • Trying a new model often meant stopping someone else’s.

In this post, we explain how GPUStack brought our models on Bedrock and other cloud providers, along with the LLMs we self-host on our Sparks, under one platform. We also cover what it solved for us and what else it brings.

The options we considered

Before we picked a tool, we looked at options that would meet our needs today and still scale later.

Run vLLM directly

  • What it gives us: fast, OpenAI-compatible serving, with one vllm serve process per model.

  • Why not: vLLM is a serving engine, not a platform. Every model is its own process on its own port, with no users, per-user keys, usage tracking or scheduling across spark-1 and spark-2. Bedrock and Vertex AI stay completely separate.

vLLM with LiteLLM in front

  • What it gives us: one OpenAI-compatible gateway over Bedrock, Vertex AI and our vLLM endpoints. The open-source version includes virtual keys, budgets and spend tracking.

  • Why not: it only solves the routing half. Someone still has to start, stop and place every model on the Sparks by hand. LiteLLM’s SSO is also free for only up to 5 users, so our 30 users would need an enterprise licence.

Xinference

  • What it gives us: a supervisor-and-worker cluster with a web UI and OpenAI-compatible APIs for LLM, speech and multimodal models. Since version 3.0, it also has built-in users, permissions and API keys.

  • Why not: we found no way to put Bedrock or Vertex AI behind it, and its sign-in uses usernames and passwords rather than Google SSO.

Kubernetes-native serving (KServe, KubeAI, AIBrix)

  • What it gives us: production-grade model serving, autoscaling and routing on Kubernetes.

  • Why not: all of them need a Kubernetes cluster first, so we would have a control plane to run and upgrade, and our cloud provider models would still sit outside it. Engine support also varies by project. KubeAI, for example, runs vLLM and Ollama but not SGLang or TensorRT-LLM.

Build our own control plane

  • What it gives us: full control over deployment, routing, API keys and usage tracking.

  • Why not: we would be writing and maintaining the platform ourselves, instead of spending time on inference experiments.

GPUStack

  • What it gives us: it schedules models on our own GPUs, routes to cloud providers such as Bedrock through its AI gateway, and adds SSO, roles, per-user API keys, usage metering and benchmarking.

  • Why it fit: it was the only option that covered both halves, our own GPUs and our cloud providers, in one open-source platform.

The goal was not to add another layer of infrastructure for its own sake. We wanted to remove the manual work around model serving.

Why we chose GPUStack

GPUStack takes our GPUs, our models and cloud providers like Bedrock, and gives us one place to serve all of them through an API. GPUStack describes itself as a token factory, because every request and response can be metered in tokens.

  1. Its AI gateway could sit in front of Bedrock and other providers straight away, before we moved anything onto our own GPUs.

  2. We were already running vLLM and trying other inference engines. GPUStack runs those same engines for us, and we can add custom backends as well.

So we could keep what already worked and add the platform pieces we were missing around it.

How we set it up

We run GPUStack v2.2.3. The server is a single container on an E2E server inside our network, with Caddy in front of it for TLS, and both Sparks join it as Docker workers. GPUStack also exposes a Prometheus /metrics endpoint, which we use to monitor the models. We later connected it to our existing Prometheus and Grafana servers.

The architecture

The diagram below shows how GPUStack sits inside our network.

Requests from users, agents and scripts go over HTTPS to Caddy, then to the GPUStack server. Its Higress gateway routes each request to a model on spark-1 or spark-2, or to Amazon Bedrock. The scheduler places models on the two DGX Spark workers.

How a request flows

Here is the path a single request takes, from the user to the model and back:

  1. The request reaches Caddy over HTTPS. Caddy terminates TLS and forwards it to the GPUStack server.

  2. The API server validates the API key and reads the model name from the request.

  3. The Higress gateway routes the request to a running instance of that model on spark-1 or spark-2, or to Amazon Bedrock for a Bedrock model.

  4. If a model has more than one replica, the gateway balances requests across them.

  5. The model runs inference and streams the tokens back along the same path.

What GPUStack changed for us

Here is how GPUStack solved each problem we started with:

  • Models spread across platforms: the cloud platforms are now model providers in GPUStack, and the Sparks are its workers for the models we run in-house, so everything is managed from one place.

  • SSH access as a bottleneck: people sign in with their One2N Google account through SSO, and GPUStack's roles decide what each person can do: admins run the platform, organisation owners manage models, and members use them. Nobody needs SSH access to a Spark anymore.

  • No view of what was running: the GPUStack UI lists every model, the worker it runs on and its status. Clients only need the model name and a single DNS hostname for GPUStack.

  • No API keys or governance: each user has their own API key, and the Usage page shows tokens and requests by model, user and key.

  • Stopping each other’s models: the scheduler places a model on a Spark that has room, and the UI shows what is already running. Two machines still have limited memory, so we book slots for experiments.

It also gave us a few things we had not asked for:

  • Speech and video models: besides LLMs, we run Whisper and Kokoro for speech, and Wan 2.1 and other T2V models for video. They use the same OpenAI-compatible API, so teams do not need a different client for each type of model.

  • Limits on API keys: an API key can have an expiry date and access to only selected models.

  • Benchmarking from the UI: organisation owners can benchmark a model and see its throughput and latency on our own Sparks, without setting up a load generator by hand.

  • Automatic recovery: if a model instance crashes, GPUStack restarts it.

The screenshot below shows two views we use often: total tokens across the platform each day, and token usage for each model. Together they let us track token use across the organisation.

GPUStack Usage page, Tokens tab, 13 October to 11 November 2026: 18.7M total tokens, 8.3K API requests and 11 models used, a daily total-tokens chart, and a per-model token table led by gpt-oss-120b.

Conclusion

GPUStack started as a way to put our cloud-hosted models behind one endpoint, and it became the platform for our own GPUs as well. The models still run where they did before, on the cloud platforms and on our two Sparks. What changed is how people at One2N reach them, share them and test them.

This post covered why we built the platform and what we use it for. There is a lot more under the hood, and that is what the next posts will cover. We will show how we monitor GPUStack with our own Prometheus and Grafana servers, and what the per-model dashboards tell us. We will also cover how we scale models, from replicas on one Spark to one model served across both machines over the ConnectX-7 link with distributed vLLM.

Sources

If you liked this post, you might find these interesting too

At One2N, we run many kinds of ML models, such as LLMs, text-to-video (T2V) models and text-to-speech (TTS) models, to build prototypes and run experiments. We used to run them on Vertex AI, Amazon Bedrock and a few other platforms, each with its own console, API keys and usage view.

In October, we added two NVIDIA DGX Spark machines to our setup. We wanted everyone at One2N to share them without stepping on each other’s toes. At first, anyone who wanted to run a model would SSH into one of the Sparks and start it with vllm serve or docker run on any free port. That worked for a few of us, but it broke down as more people joined in:

  • Only people with SSH access could deploy, so handing out SSH access became a bottleneck.

  • Nobody knew what was running without logging in to check.

  • There were no API keys, so we could not tell who sent traffic or enforce any governance.

  • Trying a new model often meant stopping someone else’s.

In this post, we explain how GPUStack brought our models on Bedrock and other cloud providers, along with the LLMs we self-host on our Sparks, under one platform. We also cover what it solved for us and what else it brings.

The options we considered

Before we picked a tool, we looked at options that would meet our needs today and still scale later.

Run vLLM directly

  • What it gives us: fast, OpenAI-compatible serving, with one vllm serve process per model.

  • Why not: vLLM is a serving engine, not a platform. Every model is its own process on its own port, with no users, per-user keys, usage tracking or scheduling across spark-1 and spark-2. Bedrock and Vertex AI stay completely separate.

vLLM with LiteLLM in front

  • What it gives us: one OpenAI-compatible gateway over Bedrock, Vertex AI and our vLLM endpoints. The open-source version includes virtual keys, budgets and spend tracking.

  • Why not: it only solves the routing half. Someone still has to start, stop and place every model on the Sparks by hand. LiteLLM’s SSO is also free for only up to 5 users, so our 30 users would need an enterprise licence.

Xinference

  • What it gives us: a supervisor-and-worker cluster with a web UI and OpenAI-compatible APIs for LLM, speech and multimodal models. Since version 3.0, it also has built-in users, permissions and API keys.

  • Why not: we found no way to put Bedrock or Vertex AI behind it, and its sign-in uses usernames and passwords rather than Google SSO.

Kubernetes-native serving (KServe, KubeAI, AIBrix)

  • What it gives us: production-grade model serving, autoscaling and routing on Kubernetes.

  • Why not: all of them need a Kubernetes cluster first, so we would have a control plane to run and upgrade, and our cloud provider models would still sit outside it. Engine support also varies by project. KubeAI, for example, runs vLLM and Ollama but not SGLang or TensorRT-LLM.

Build our own control plane

  • What it gives us: full control over deployment, routing, API keys and usage tracking.

  • Why not: we would be writing and maintaining the platform ourselves, instead of spending time on inference experiments.

GPUStack

  • What it gives us: it schedules models on our own GPUs, routes to cloud providers such as Bedrock through its AI gateway, and adds SSO, roles, per-user API keys, usage metering and benchmarking.

  • Why it fit: it was the only option that covered both halves, our own GPUs and our cloud providers, in one open-source platform.

The goal was not to add another layer of infrastructure for its own sake. We wanted to remove the manual work around model serving.

Why we chose GPUStack

GPUStack takes our GPUs, our models and cloud providers like Bedrock, and gives us one place to serve all of them through an API. GPUStack describes itself as a token factory, because every request and response can be metered in tokens.

  1. Its AI gateway could sit in front of Bedrock and other providers straight away, before we moved anything onto our own GPUs.

  2. We were already running vLLM and trying other inference engines. GPUStack runs those same engines for us, and we can add custom backends as well.

So we could keep what already worked and add the platform pieces we were missing around it.

How we set it up

We run GPUStack v2.2.3. The server is a single container on an E2E server inside our network, with Caddy in front of it for TLS, and both Sparks join it as Docker workers. GPUStack also exposes a Prometheus /metrics endpoint, which we use to monitor the models. We later connected it to our existing Prometheus and Grafana servers.

The architecture

The diagram below shows how GPUStack sits inside our network.

Requests from users, agents and scripts go over HTTPS to Caddy, then to the GPUStack server. Its Higress gateway routes each request to a model on spark-1 or spark-2, or to Amazon Bedrock. The scheduler places models on the two DGX Spark workers.

How a request flows

Here is the path a single request takes, from the user to the model and back:

  1. The request reaches Caddy over HTTPS. Caddy terminates TLS and forwards it to the GPUStack server.

  2. The API server validates the API key and reads the model name from the request.

  3. The Higress gateway routes the request to a running instance of that model on spark-1 or spark-2, or to Amazon Bedrock for a Bedrock model.

  4. If a model has more than one replica, the gateway balances requests across them.

  5. The model runs inference and streams the tokens back along the same path.

What GPUStack changed for us

Here is how GPUStack solved each problem we started with:

  • Models spread across platforms: the cloud platforms are now model providers in GPUStack, and the Sparks are its workers for the models we run in-house, so everything is managed from one place.

  • SSH access as a bottleneck: people sign in with their One2N Google account through SSO, and GPUStack's roles decide what each person can do: admins run the platform, organisation owners manage models, and members use them. Nobody needs SSH access to a Spark anymore.

  • No view of what was running: the GPUStack UI lists every model, the worker it runs on and its status. Clients only need the model name and a single DNS hostname for GPUStack.

  • No API keys or governance: each user has their own API key, and the Usage page shows tokens and requests by model, user and key.

  • Stopping each other’s models: the scheduler places a model on a Spark that has room, and the UI shows what is already running. Two machines still have limited memory, so we book slots for experiments.

It also gave us a few things we had not asked for:

  • Speech and video models: besides LLMs, we run Whisper and Kokoro for speech, and Wan 2.1 and other T2V models for video. They use the same OpenAI-compatible API, so teams do not need a different client for each type of model.

  • Limits on API keys: an API key can have an expiry date and access to only selected models.

  • Benchmarking from the UI: organisation owners can benchmark a model and see its throughput and latency on our own Sparks, without setting up a load generator by hand.

  • Automatic recovery: if a model instance crashes, GPUStack restarts it.

The screenshot below shows two views we use often: total tokens across the platform each day, and token usage for each model. Together they let us track token use across the organisation.

GPUStack Usage page, Tokens tab, 13 October to 11 November 2026: 18.7M total tokens, 8.3K API requests and 11 models used, a daily total-tokens chart, and a per-model token table led by gpt-oss-120b.

Conclusion

GPUStack started as a way to put our cloud-hosted models behind one endpoint, and it became the platform for our own GPUs as well. The models still run where they did before, on the cloud platforms and on our two Sparks. What changed is how people at One2N reach them, share them and test them.

This post covered why we built the platform and what we use it for. There is a lot more under the hood, and that is what the next posts will cover. We will show how we monitor GPUStack with our own Prometheus and Grafana servers, and what the per-model dashboards tell us. We will also cover how we scale models, from replicas on one Spark to one model served across both machines over the ConnectX-7 link with distributed vLLM.

Sources

If you liked this post, you might find these interesting too

Share
Share
On this page
Section
On this page

Continue reading.

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!