Services

Resources

Company

Hermes agent works. Now we have to run it in production

Hermes agent works. Now we have to run it in production

Hermes agent works. Now we have to run it in production

Hermes agent works. Now we have to run it in production

Most people have tried autonomous agents for ops work, and they work in a demo. The harder question is whether they move the numbers a business cares about: how long onboarding takes, how quickly incidents get investigated, how smoothly infrastructure changes go from intent to reality.

This post covers how we ran the Hermes agent in production at a client for the last three months, and how we hardened it around security, reproducibility and controlled evolution.

Most automation in infrastructure is deliberately boring.

A health check runs every few minutes. A backup job follows a schedule. A Kubernetes controller observes some state and reconciles it toward another state. We know what the input looks like, we know what should happen, and we can usually describe the entire workflow before it starts.

That is exactly what makes deterministic automation reliable.

But a surprising amount of engineering work doesn’t look like that.

Someone notices that a Kafka connector has stopped working and asks why. Another engineer needs infrastructure for a new service but doesn’t know where in the IaC (infrastructure as code) repository it should be defined. Someone has several thousand lines of logs and wants to know what actually failed. Someone else wants to increase resources after a repeated high CPU or memory alert. An incident starts with one symptom, but the next step depends entirely on what the previous step reveals.

You can automate pieces of these workflows, but describing the complete decision tree quickly becomes harder than doing the investigation yourself.

We had already automated a fair amount at our customer: IaC, CI/CD, Kafka operations, Postgres user creation and more. But some tasks don’t fit the conventional automation route, or building a generic solution for them would cost too much time and engineering effort. Things like creating infrastructure from intent, or summarising logs and correlating them against the live code.

That was the gap we wanted to explore with Hermes.

Enter Hermes

Hermes is an open-source AI agent from Nous Research (github.com/NousResearch/hermes-agent). It can reason over a request, work out what it needs to do, call tools and build reusable skills from what it learns. Instead of defining every possible path beforehand, we expose capabilities and let the agent work out how to combine them.

We’ll walk through this in phases, to show how we ended up with the version that runs today.

It works on my laptop

The first version was straightforward. Hermes had a Slack interface, access to our code repositories and a model capable of reasoning through operational requests. Engineers could ask it to analyze code or open pull requests. No other tools were connected yet. The whole thing ran on an engineer’s MacBook, which isn’t exactly a good server, for obvious reasons.

Version one of Hermes: engineers ask in Slack, Hermes runs on an engineer's laptop, reads the code repositories and raises pull requests that a human merges. No ops tools are connected.

But it worked, and it showed it could deliver value. It was a pragmatic way to spend minimal time finding out whether Hermes was the right fit for our customer’s requirements.

A working agent isn’t necessarily a production system

At first, Hermes could only open pull requests against our infrastructure and application repos. That alone cleared the immediate bottleneck on creating infrastructure.

But the temptation with an agent is to think about what else you can give it.

If it can read connector state, let it restart connectors. If it understands Terraform, let it modify infrastructure. If it repeatedly solves the same problem, let it remember how. Add more Model Context Protocol (MCP) integrations, more memory and more tools, and presumably the agent becomes progressively more useful.

That reasoning works remarkably well right up until you ask what happens when the agent is wrong.

Hermes is self-evolving. When it performs a task, it can turn what it learned into a skill: effectively a reusable runbook that it applies when a similar request appears later.

Self-evolution is one of its most useful properties. It is also one of the first things that made us uncomfortable about running it in production.

Everyone lives with some recurring problems that are easier to manage than to solve. One of our customer’s third-party services exposed a Kafka connector endpoint that kept losing its connection from their end. While a long-term fix was being put in place, the quick workaround, so the analytics team wasn’t blocked, was to act on the alert and restart the Kafka connector. Hermes fit this use case perfectly.

Given a runbook and the right permissions, it completed the task, recognized that this was a recurring activity and created a skill for it. Future alerts and restart requests were handed off from a human to the agent.

Some incidents need one-off workarounds instead. They are good band-aids that let you act quickly and contain the damage, but band-aids need to be ripped off. One such instance was the customer’s Prometheus storage filling up and no longer accepting metrics. The stopgap was to clear its write-ahead log (WAL) directory. The proper fix was to put the right retention policies in place. Hermes still saw the stopgap as a skill worth reusing and added it to its working set.

From the agent’s perspective, both interactions look remarkably similar: a human requested something, a sequence of actions solved the problem, and the task succeeded.

But one of those procedures is organizational knowledge, and the other is a one-off fix that will probably never be repeated.

At an early stage of its evolution, an agent can’t reliably tell which action is worth storing as a skill and which one is a temporary fix that shouldn’t become a solution.

So the question wasn’t simply how to let Hermes learn. We had to answer: who gets to decide what Hermes is allowed to learn permanently?

That question became the basis of the entire production architecture.

We turned self-improvement into GitOps

We already had a mechanism for machines proposing changes that humans need to trust: Git.

So rather than letting Hermes make a newly generated skill permanent immediately, we put its skills under version control.

When Hermes believes it has learned something reusable, it can create or modify the corresponding skill. That change goes to the control-plane repository as a pull request. An operator can then inspect exactly what Hermes is trying to teach itself.

If the behaviour is genuinely reusable, the PR is merged. If Hermes has generalized from something specific to one incident, the PR is rejected.

Skill pull request flow: Hermes does a task, spots something reusable, opens a skill PR against the control-plane repo, and a human reviews it. Rejected PRs are closed; merged PRs become approved Git state that the reconciler syncs to Hermes.

The moment skills became Git-managed artifacts, we inherited review, history, diffs, attribution and rollback, without reaching for shiny new tools to solve an already solved problem. If GitOps is new to you, it is the practice of making a Git repository the single source of truth for what should be running.

What counts as a skill, and what doesn’t

So that every team uses the same criteria when gatekeeping how Hermes evolves, we wrote down what is skill-worthy and what isn’t.

A skill PR should be:

  • Recurring: the same situation keeps coming back. It’s not a one-time thing.

  • Deterministic: same inputs, same steps, same safe outcome every time.

  • Bounded: it has clear limits, with no unknown irreversible or destructive side effects.

  • Known context: nothing depends on one host, commit or timestamp. It works anywhere the pattern shows up.

These are signs that something should not become a skill:

  • One-off: it happened once, tied to a specific incident, and won’t recur the same way.

  • Destructive: the shortcut works now, but it has irreversible side effects or deletes data.

  • Masks the real fix: it treats the symptom and leaves the root cause in place.

Checklist for skill PRs. Make it a skill if it is recurring, deterministic, bounded and has known context, like restarting the flaky Kafka connector. Keep it out if it is one-off, destructive or masks the real fix, like clearing the Prometheus WAL to free disk.

Reconciliation

Putting skills in Git only solved half the problem. Git also had to become authoritative.

If Hermes could modify its local skills and keep using them while Git contained something else, then Git wasn’t a control plane. It was just a backup with good branding.

Here’s the scenario: an operator merges the Kafka-restart skill PR. Without a reconciler, the running Hermes never picks it up. Meanwhile it keeps a locally edited version that Git never approved. Git and Hermes drift apart.

Hermes can run jobs on a schedule, so our reconcilers are just scheduled scripts that pull approved state from GitHub and push proposed changes back to it.

Hermes can propose how it should evolve, but only approved changes become part of its durable behaviour, and the reconcilers enforce that.

The reconciler sits between the control-plane repo and the running Hermes. It pulls approved skills and memory from Git and applies them, and it sends new or edited skills back to the repo as pull requests, so Git and the agent never drift apart.

The reconciler syncs the approved changes and makes them part of Hermes’ skills or memory for future tasks.

We also point an LLM at the Hermes control-plane repo to check how Hermes is evolving, and whether the skills it has gained over the last few weeks are still relevant and worth keeping. This also catches approvals that slipped through. You could argue this level of control needs a lot of human involvement, but at this early stage it is well worth the time to build a solid foundation.

Usage patterns evolved

Once skills were governed, adoption grew, and requests fell into three buckets.

A request in Slack is classified as an infra change, a service change or a read-only question. Each has a codified path Hermes can use (a Terraform module, a runbook, a wired-up data source) or a gap where it stops.

1. Infra changes

  • Who asks: SRE and developer teams deploying new services or provisioning more infrastructure.

  • Example prompts: “Create a new RDS in dev us-east-1”, “For dev Elasticsearch, increase replicas from 2 to 3”.

  • Control: Hermes never applies infra changes. It uses existing Terraform modules, runs terraform plan and raises a PR against the relevant repo. A human merges it and triggers terraform apply.

2. Service queries and changes

  • Who asks: SRE and developer teams restarting services, or exploring how the product fits together.

  • Example prompts: “Restart the Kafka connector in dev”, “Scale up the payment service to 3 replicas”.

  • Control: for known runbooks with bounded, deterministic workflows, Hermes has direct access to run them.

3. Onboarding and read-only investigations

  • Who asks: SREs debugging production incidents, and new joiners learning a team’s stack.

  • Example prompts: “Check if there are any 5xx errors introduced after yesterday’s deployment for the login service”, “Where is the infra for the auth service hosted?”

  • Control: nothing changes. These are mostly Slack conversations. Hermes leans on the MCP servers that each tool exposes. For our monitoring product, its MCP server and tool calls did the heavy lifting.

Bringing non-IaC teams along through FOMO

For infra changes, we were dealing with two kinds of teams:

  • Legacy setups that didn’t use IaC and were set up by hand in the console (ClickOps).

  • Current setups that managed their infrastructure through IaC.

For the longest time, the legacy setups stayed as they were, because of the “don’t fix it if it ain’t broken” mindset. The infrastructure rarely changed beyond scaling up and down, and only a select few knew it in depth.

Watching the IaC teams use Hermes well gave the legacy teams a healthy dose of FOMO (fear of missing out). They imported their infrastructure into IaC so they could also maintain it by intent. That made the infrastructure easier to manage, removed the quirks that come with ClickOps, and helped new joiners understand critical legacy setups faster.

Who would’ve thought it would take an agent to change how humans behave?

Is everything GitOps?

If every operation requires a pull request and a human approval, we’ve built an extremely elaborate chatbot that files tickets.

Some operational actions are narrow, reversible and safe enough to expose directly.

At our client, Kafka connector restarts became a useful example.

Before, someone who hit a failed connector could end up waiting on the one engineer who knew the system and had the access. The restart itself might take seconds, but that dependency turned a tiny operation into waiting time.

Hermes changed that interaction.

An engineer can ask in Slack for the failing connector to be inspected and restarted. Hermes works out which connector is involved, checks its state through the available tools, calls the permitted restart operation and reports what happened.

The important part is what Hermes cannot do.

We didn’t hand it unrestricted Kafka access. The tool exposes only the capability the workflow needs. Similar limits apply elsewhere: read the relevant information, perform explicitly permitted non-destructive operations, and send potentially consequential changes through Git.

The useful security boundary became capabilities rather than environments.

Giving an agent SSH access to a production box and writing “please be careful” in the system prompt is reckless. Babysitting it on every task is unnecessary paranoia. We keep working to find the balance between the two.

That wraps up part 1. In part 2 we’ll cover cost, model usage, security and adoption numbers, and tie it together with a cost-benefit analysis for the customer.

If you liked this post, you might find these interesting too

Most people have tried autonomous agents for ops work, and they work in a demo. The harder question is whether they move the numbers a business cares about: how long onboarding takes, how quickly incidents get investigated, how smoothly infrastructure changes go from intent to reality.

This post covers how we ran the Hermes agent in production at a client for the last three months, and how we hardened it around security, reproducibility and controlled evolution.

Most automation in infrastructure is deliberately boring.

A health check runs every few minutes. A backup job follows a schedule. A Kubernetes controller observes some state and reconciles it toward another state. We know what the input looks like, we know what should happen, and we can usually describe the entire workflow before it starts.

That is exactly what makes deterministic automation reliable.

But a surprising amount of engineering work doesn’t look like that.

Someone notices that a Kafka connector has stopped working and asks why. Another engineer needs infrastructure for a new service but doesn’t know where in the IaC (infrastructure as code) repository it should be defined. Someone has several thousand lines of logs and wants to know what actually failed. Someone else wants to increase resources after a repeated high CPU or memory alert. An incident starts with one symptom, but the next step depends entirely on what the previous step reveals.

You can automate pieces of these workflows, but describing the complete decision tree quickly becomes harder than doing the investigation yourself.

We had already automated a fair amount at our customer: IaC, CI/CD, Kafka operations, Postgres user creation and more. But some tasks don’t fit the conventional automation route, or building a generic solution for them would cost too much time and engineering effort. Things like creating infrastructure from intent, or summarising logs and correlating them against the live code.

That was the gap we wanted to explore with Hermes.

Enter Hermes

Hermes is an open-source AI agent from Nous Research (github.com/NousResearch/hermes-agent). It can reason over a request, work out what it needs to do, call tools and build reusable skills from what it learns. Instead of defining every possible path beforehand, we expose capabilities and let the agent work out how to combine them.

We’ll walk through this in phases, to show how we ended up with the version that runs today.

It works on my laptop

The first version was straightforward. Hermes had a Slack interface, access to our code repositories and a model capable of reasoning through operational requests. Engineers could ask it to analyze code or open pull requests. No other tools were connected yet. The whole thing ran on an engineer’s MacBook, which isn’t exactly a good server, for obvious reasons.

Version one of Hermes: engineers ask in Slack, Hermes runs on an engineer's laptop, reads the code repositories and raises pull requests that a human merges. No ops tools are connected.

But it worked, and it showed it could deliver value. It was a pragmatic way to spend minimal time finding out whether Hermes was the right fit for our customer’s requirements.

A working agent isn’t necessarily a production system

At first, Hermes could only open pull requests against our infrastructure and application repos. That alone cleared the immediate bottleneck on creating infrastructure.

But the temptation with an agent is to think about what else you can give it.

If it can read connector state, let it restart connectors. If it understands Terraform, let it modify infrastructure. If it repeatedly solves the same problem, let it remember how. Add more Model Context Protocol (MCP) integrations, more memory and more tools, and presumably the agent becomes progressively more useful.

That reasoning works remarkably well right up until you ask what happens when the agent is wrong.

Hermes is self-evolving. When it performs a task, it can turn what it learned into a skill: effectively a reusable runbook that it applies when a similar request appears later.

Self-evolution is one of its most useful properties. It is also one of the first things that made us uncomfortable about running it in production.

Everyone lives with some recurring problems that are easier to manage than to solve. One of our customer’s third-party services exposed a Kafka connector endpoint that kept losing its connection from their end. While a long-term fix was being put in place, the quick workaround, so the analytics team wasn’t blocked, was to act on the alert and restart the Kafka connector. Hermes fit this use case perfectly.

Given a runbook and the right permissions, it completed the task, recognized that this was a recurring activity and created a skill for it. Future alerts and restart requests were handed off from a human to the agent.

Some incidents need one-off workarounds instead. They are good band-aids that let you act quickly and contain the damage, but band-aids need to be ripped off. One such instance was the customer’s Prometheus storage filling up and no longer accepting metrics. The stopgap was to clear its write-ahead log (WAL) directory. The proper fix was to put the right retention policies in place. Hermes still saw the stopgap as a skill worth reusing and added it to its working set.

From the agent’s perspective, both interactions look remarkably similar: a human requested something, a sequence of actions solved the problem, and the task succeeded.

But one of those procedures is organizational knowledge, and the other is a one-off fix that will probably never be repeated.

At an early stage of its evolution, an agent can’t reliably tell which action is worth storing as a skill and which one is a temporary fix that shouldn’t become a solution.

So the question wasn’t simply how to let Hermes learn. We had to answer: who gets to decide what Hermes is allowed to learn permanently?

That question became the basis of the entire production architecture.

We turned self-improvement into GitOps

We already had a mechanism for machines proposing changes that humans need to trust: Git.

So rather than letting Hermes make a newly generated skill permanent immediately, we put its skills under version control.

When Hermes believes it has learned something reusable, it can create or modify the corresponding skill. That change goes to the control-plane repository as a pull request. An operator can then inspect exactly what Hermes is trying to teach itself.

If the behaviour is genuinely reusable, the PR is merged. If Hermes has generalized from something specific to one incident, the PR is rejected.

Skill pull request flow: Hermes does a task, spots something reusable, opens a skill PR against the control-plane repo, and a human reviews it. Rejected PRs are closed; merged PRs become approved Git state that the reconciler syncs to Hermes.

The moment skills became Git-managed artifacts, we inherited review, history, diffs, attribution and rollback, without reaching for shiny new tools to solve an already solved problem. If GitOps is new to you, it is the practice of making a Git repository the single source of truth for what should be running.

What counts as a skill, and what doesn’t

So that every team uses the same criteria when gatekeeping how Hermes evolves, we wrote down what is skill-worthy and what isn’t.

A skill PR should be:

  • Recurring: the same situation keeps coming back. It’s not a one-time thing.

  • Deterministic: same inputs, same steps, same safe outcome every time.

  • Bounded: it has clear limits, with no unknown irreversible or destructive side effects.

  • Known context: nothing depends on one host, commit or timestamp. It works anywhere the pattern shows up.

These are signs that something should not become a skill:

  • One-off: it happened once, tied to a specific incident, and won’t recur the same way.

  • Destructive: the shortcut works now, but it has irreversible side effects or deletes data.

  • Masks the real fix: it treats the symptom and leaves the root cause in place.

Checklist for skill PRs. Make it a skill if it is recurring, deterministic, bounded and has known context, like restarting the flaky Kafka connector. Keep it out if it is one-off, destructive or masks the real fix, like clearing the Prometheus WAL to free disk.

Reconciliation

Putting skills in Git only solved half the problem. Git also had to become authoritative.

If Hermes could modify its local skills and keep using them while Git contained something else, then Git wasn’t a control plane. It was just a backup with good branding.

Here’s the scenario: an operator merges the Kafka-restart skill PR. Without a reconciler, the running Hermes never picks it up. Meanwhile it keeps a locally edited version that Git never approved. Git and Hermes drift apart.

Hermes can run jobs on a schedule, so our reconcilers are just scheduled scripts that pull approved state from GitHub and push proposed changes back to it.

Hermes can propose how it should evolve, but only approved changes become part of its durable behaviour, and the reconcilers enforce that.

The reconciler sits between the control-plane repo and the running Hermes. It pulls approved skills and memory from Git and applies them, and it sends new or edited skills back to the repo as pull requests, so Git and the agent never drift apart.

The reconciler syncs the approved changes and makes them part of Hermes’ skills or memory for future tasks.

We also point an LLM at the Hermes control-plane repo to check how Hermes is evolving, and whether the skills it has gained over the last few weeks are still relevant and worth keeping. This also catches approvals that slipped through. You could argue this level of control needs a lot of human involvement, but at this early stage it is well worth the time to build a solid foundation.

Usage patterns evolved

Once skills were governed, adoption grew, and requests fell into three buckets.

A request in Slack is classified as an infra change, a service change or a read-only question. Each has a codified path Hermes can use (a Terraform module, a runbook, a wired-up data source) or a gap where it stops.

1. Infra changes

  • Who asks: SRE and developer teams deploying new services or provisioning more infrastructure.

  • Example prompts: “Create a new RDS in dev us-east-1”, “For dev Elasticsearch, increase replicas from 2 to 3”.

  • Control: Hermes never applies infra changes. It uses existing Terraform modules, runs terraform plan and raises a PR against the relevant repo. A human merges it and triggers terraform apply.

2. Service queries and changes

  • Who asks: SRE and developer teams restarting services, or exploring how the product fits together.

  • Example prompts: “Restart the Kafka connector in dev”, “Scale up the payment service to 3 replicas”.

  • Control: for known runbooks with bounded, deterministic workflows, Hermes has direct access to run them.

3. Onboarding and read-only investigations

  • Who asks: SREs debugging production incidents, and new joiners learning a team’s stack.

  • Example prompts: “Check if there are any 5xx errors introduced after yesterday’s deployment for the login service”, “Where is the infra for the auth service hosted?”

  • Control: nothing changes. These are mostly Slack conversations. Hermes leans on the MCP servers that each tool exposes. For our monitoring product, its MCP server and tool calls did the heavy lifting.

Bringing non-IaC teams along through FOMO

For infra changes, we were dealing with two kinds of teams:

  • Legacy setups that didn’t use IaC and were set up by hand in the console (ClickOps).

  • Current setups that managed their infrastructure through IaC.

For the longest time, the legacy setups stayed as they were, because of the “don’t fix it if it ain’t broken” mindset. The infrastructure rarely changed beyond scaling up and down, and only a select few knew it in depth.

Watching the IaC teams use Hermes well gave the legacy teams a healthy dose of FOMO (fear of missing out). They imported their infrastructure into IaC so they could also maintain it by intent. That made the infrastructure easier to manage, removed the quirks that come with ClickOps, and helped new joiners understand critical legacy setups faster.

Who would’ve thought it would take an agent to change how humans behave?

Is everything GitOps?

If every operation requires a pull request and a human approval, we’ve built an extremely elaborate chatbot that files tickets.

Some operational actions are narrow, reversible and safe enough to expose directly.

At our client, Kafka connector restarts became a useful example.

Before, someone who hit a failed connector could end up waiting on the one engineer who knew the system and had the access. The restart itself might take seconds, but that dependency turned a tiny operation into waiting time.

Hermes changed that interaction.

An engineer can ask in Slack for the failing connector to be inspected and restarted. Hermes works out which connector is involved, checks its state through the available tools, calls the permitted restart operation and reports what happened.

The important part is what Hermes cannot do.

We didn’t hand it unrestricted Kafka access. The tool exposes only the capability the workflow needs. Similar limits apply elsewhere: read the relevant information, perform explicitly permitted non-destructive operations, and send potentially consequential changes through Git.

The useful security boundary became capabilities rather than environments.

Giving an agent SSH access to a production box and writing “please be careful” in the system prompt is reckless. Babysitting it on every task is unnecessary paranoia. We keep working to find the balance between the two.

That wraps up part 1. In part 2 we’ll cover cost, model usage, security and adoption numbers, and tie it together with a cost-benefit analysis for the customer.

If you liked this post, you might find these interesting too

Share
Share
On this page
Section
On this page

Continue reading.

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!