Services

Resources

Company

Illustration for Why healthy doesn’t mean what you think it means with AI workloads

Why healthy doesn’t mean what you think it means with AI workloads

Here’s a real but somewhat fictitious story (to protect the innocent) based on my interaction with a Head of SRE recently. Enjoy the read!

Head of SRE: You know we use the AI support assistant in our product, right? It ran into an issue on prod last week.

You: Oh! What was the issue?

Head of SRE: It started giving customers incorrect answers.

You: How did you know the answers were incorrect?

Head of SRE: Well, to be honest, we didn’t know until we noticed that people were complaining on social media.

You: I think you do have monitoring, right? Didn’t that help find this earlier?

Head of SRE: Umm, no. I mean we do have monitoring and we monitor API errors, latency etc. but that was all good. No obvious signs of what could be wrong.

You: What about the actual assistant responses? Were you monitoring those?

Head of SRE: Yeah, we had those but we weren’t monitoring their quality. How does one even do that?

You: Ah! Let’s see, do you capture the traces? If you have traces, for each request you can understand more about what the prompt was, which model was called and the model response itself. Even tool calls. In many cases, a single user interaction might involve many model calls and even more tool calls.

Head of SRE: Yeah, I get it, but this will only help with specific individual requests, right? How do we detect quality degradation overall? Do we just randomly look at some traces?

You: Yeah, that can be a good start, but you’ll need to think about evals as a first class problem when building AI applications. You’ll want to know things like: number of tool calls, the retrieved context, citation correctness if it’s RAG, etc. Maybe the users are asking the assistant questions that it doesn’t know how to handle. For example, users asking questions that aren’t relevant to your app’s domain.

Head of SRE: Could we use user feedback?

You: YES! That works too. But simple thumbs-up and thumbs-down feedback can be limited. If users edit a generated response in the next prompt, that correction may tell you more about what went wrong.

Head of SRE: So we’d need tracing, evaluations, and feedback collection.

You: Yes. And you’d ideally connect that information. If a user reports an incorrect answer, engineers should be able to trace what happened and understand why.

Head of SRE: Sounds like we need to rethink what application health means.

You: For AI applications, yes. Health is about the service being up, but with AI, you have to worry about correctness and quality of response as well. This should be captured in observability. Basically, your API can be perfectly healthy while your product is actually degraded in quality for the users.

I write such stories on software engineering.

There’s no specific frequency, as I don’t make these up.

If you liked this one, you might love - A curious story of debugging Machine Learning model performance

Follow me on LinkedIn and Twitter for more such stuff, straight from the production oven!

Share
Share

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Continue reading.