Congratulations! The difference between pure agentic exploration and deterministic steps is spot on. Runbooks give ops more confidence on the data exploration and save time/context.
Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first. Also how the runbooks can self heal if results from some steps in the middle are not expected.
>> how the runbooks can self heal if results from some steps in the middle are not expected.
Yeah this is a very interesting angle. Our primary mechanism here is via agent created auto-memories today. The agent keeps track of the most useful steps, and more importantly, dead end steps as it executes runbooks. We think this offers a great bridge to suggest runbook updates and keep them current.
>> Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first.
Really depends on runbook quality, so I don't have a straightforward answer. Of course, it's faster and cheaper if you have well defined steps in your runbooks. As an example, `check logs for service frontend, faceted by host_name`, vs. `check logs`. Agent does more exploration in the latter case.
Re: savings - it depends on the use case. For example, one of our users set up a small runbook to run a group-by-IP query for high-throughput alerts, since that was their most common first response to those alerts. That alone cuts out a couple of minutes of exploration per incident and removes the variability of the agent deciding what data to investigate and how to slice it.
In our experience, runbooks provide a consistent, fast, and reliable way of investigating incidents (or ruling out common causes). In their absence, the AI does its usual open-ended exploration.
Interesting! tbh, we don't have any runbooks and pretty minimal telemetry set up (we're a very small team :), do you have any recommendations on which telemetry service to use to get started? right now, our services run on a combination GCP Cloud Run + Vercel
Nice to see you here, Will! I’d generally recommend using open telemetry for instrumentation so that you keep the option of switching between telemetry vendors.
Re: runbooks, yeah even larger teams don’t have good ones to begin with. Relvy helps debug without runbooks as well - it might take longer to explore, but once you are happy with a particular investigation path the AI took, you can save it as a runbook for more deterministic future executions.
How does this differ from cursor cloud agents where I can hook up MCPs, etc and even launch the agent in my own cloud to connect directly to internal hosts like dbs?
Thanks. Yeah, Cursor / Claude code + MCP is powerful. We differentiate on two fronts, mainly:
1) Greater accuracy with our specialized tools: Most MCP tools allow agents to query data, or run *ql queries - this overwhelms context windows given the scale of telemetry data. Raw data is also not great for reasoning - we’ve designed our tools to ensure that models get data in the right format, enriched with statistical summaries, baselines, and correlation data, so LLMs can focus on reasoning.
2) Product UX: You’ll also find that text based outputs from general purpose agents are not sufficient for this task - our notebook UX offers a great way to visualize the underlying data so you can review and build trust with the AI.
To be clear, are the main differentiators basically better built-in MCPs and better UX? Not knocking just trying to understand the differences.
I have had incredible success debugging issues by just hooking up Datadog MCP and giving agents access to it. Claude/cursor don't seem to have any issues pulling in the raw data they need in amounts that don't overload their context.
Do you consider this a tool to be used in addition to something like cursor cloud agents or to replace it?
For the debugging workflow you described, we would be a standalone replacement for cursor or other agents. We don't yet write code so can't replace your cursor agents entirely.
Re: diffentiation - yes, faster, more accurate and more consistent. Partially because of better tools and UX, and partially because we anchor on runbooks. On-call engineers can quickly map out that the AI ran so-and-so steps, and here's what it found for each, and here's the time series graph that supports this.
Interesting that you have had great success with Datadog MCP. Do you mainly look at logs?
> For the X workflow, we would be a standalone replacement for other agents.
Imo, this is not what users want. They want extension to their agent. If a project tells me I have to use their interface or agentic setup, it's 95% not going to happen. Consider how many SaaS tools we already have to deal with, that many agents is not desirable, they all have their little quirks and take time to "get to know"
Instead, build extensions, skills, and subagents that fit into my agentic workflow and tooling. This will also simplify what you need to do, so you can focus on your core competency. For example, you should be able to create a chat participant in VS Code / Copilot, and take advantage of the native notebook and diff rendering, sharing the MCPs (et al) the user already has for their agents for their internal systems.
> They want extension to their agent. If a project tells me I have to use their interface or agentic setup, it's 95% not going to happen
Yes, there’s definitely friction there. It may be that the right form factor is that you trigger Relvy’s debugging agent via Claude code / Cursor .
Our early users are heavy on needing to look at the raw data to be able to review the AI RCA, so a standalone set up makes sense. Also, the dominant usage pattern is background agentic execution triggered by alerts, and not manual.
Yup, we are moving up the ladders of abstraction and will have our agentic team interfaces that include agents triggered outside of human input. It does not change things. As soon as I need to go into the code or to the agent to fix the problem, I'm back to copy and pasting, or switching to view, between multiple interfaces. That's the kind of stuff we loathe
Runbooks are great and all, but actions need to be taken and I'm not going to give all the vendor interfaces to the internal systems. They can be subagents in my system which already has the tools and permission gates needed, access to code and git for IaC changes, etc...
It seems like the way to go now, it's easier to get moving and show off an experience and the vision, but it's definitely not the operational way in prod for a lot of reasons, security being a paramount one.
I also do not discount that your SaaS can be easily replaced by an open sourced subagent team in the next couple of years.
heh, I was just about to post the following on your previous comment re: reproducible benchmark results. Thanks for posting the blog.
With the docker images that we offer, in theory, people can re-run the benchmark themselves with our agent. But we should document and make that easier.
At the end of it, you really would have to evaluate on your own production alerts. Hopefully the easy install + set up helps.
Interesting!
In my experience using custom harnesses has worked better eg: Stripe etc all did it custom largely because of the sensitive integrations. How would you handle that?
Do you mean how we connect to internal data? Today, you can connect any API endpoint to Relvy, so if you have internal business data / dashboards that you look at while debugging, Relvy can do the same if there's an API for it.
Most of our deployments are self-hosted, in which case the data stays locally (your chosen LLM provider exempted), if that's what you are asking.
Re: custom harnesses, imo maintaining them can be time consuming especially when things are changing very fast with AI. Bringing up a prototype is easy but a robust harness that handles the edge cases needs time and effort.
Congratulations! The difference between pure agentic exploration and deterministic steps is spot on. Runbooks give ops more confidence on the data exploration and save time/context.
Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first. Also how the runbooks can self heal if results from some steps in the middle are not expected.
>> how the runbooks can self heal if results from some steps in the middle are not expected.
Yeah this is a very interesting angle. Our primary mechanism here is via agent created auto-memories today. The agent keeps track of the most useful steps, and more importantly, dead end steps as it executes runbooks. We think this offers a great bridge to suggest runbook updates and keep them current.
>> Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first.
Really depends on runbook quality, so I don't have a straightforward answer. Of course, it's faster and cheaper if you have well defined steps in your runbooks. As an example, `check logs for service frontend, faceted by host_name`, vs. `check logs`. Agent does more exploration in the latter case.
We wrote about the LLM costs of investigating production alerts more generally here, in case helpful: https://relvy.ai/blog/llm-cost-of-ai-sre-investigating-produ...
Re: savings - it depends on the use case. For example, one of our users set up a small runbook to run a group-by-IP query for high-throughput alerts, since that was their most common first response to those alerts. That alone cuts out a couple of minutes of exploration per incident and removes the variability of the agent deciding what data to investigate and how to slice it.
In our experience, runbooks provide a consistent, fast, and reliable way of investigating incidents (or ruling out common causes). In their absence, the AI does its usual open-ended exploration.
Interesting! tbh, we don't have any runbooks and pretty minimal telemetry set up (we're a very small team :), do you have any recommendations on which telemetry service to use to get started? right now, our services run on a combination GCP Cloud Run + Vercel
Nice to see you here, Will! I’d generally recommend using open telemetry for instrumentation so that you keep the option of switching between telemetry vendors.
Re: runbooks, yeah even larger teams don’t have good ones to begin with. Relvy helps debug without runbooks as well - it might take longer to explore, but once you are happy with a particular investigation path the AI took, you can save it as a runbook for more deterministic future executions.
How does this differ from cursor cloud agents where I can hook up MCPs, etc and even launch the agent in my own cloud to connect directly to internal hosts like dbs?
Thanks. Yeah, Cursor / Claude code + MCP is powerful. We differentiate on two fronts, mainly:
1) Greater accuracy with our specialized tools: Most MCP tools allow agents to query data, or run *ql queries - this overwhelms context windows given the scale of telemetry data. Raw data is also not great for reasoning - we’ve designed our tools to ensure that models get data in the right format, enriched with statistical summaries, baselines, and correlation data, so LLMs can focus on reasoning.
2) Product UX: You’ll also find that text based outputs from general purpose agents are not sufficient for this task - our notebook UX offers a great way to visualize the underlying data so you can review and build trust with the AI.
To be clear, are the main differentiators basically better built-in MCPs and better UX? Not knocking just trying to understand the differences.
I have had incredible success debugging issues by just hooking up Datadog MCP and giving agents access to it. Claude/cursor don't seem to have any issues pulling in the raw data they need in amounts that don't overload their context.
Do you consider this a tool to be used in addition to something like cursor cloud agents or to replace it?
For the debugging workflow you described, we would be a standalone replacement for cursor or other agents. We don't yet write code so can't replace your cursor agents entirely.
Re: diffentiation - yes, faster, more accurate and more consistent. Partially because of better tools and UX, and partially because we anchor on runbooks. On-call engineers can quickly map out that the AI ran so-and-so steps, and here's what it found for each, and here's the time series graph that supports this.
Interesting that you have had great success with Datadog MCP. Do you mainly look at logs?
> For the X workflow, we would be a standalone replacement for other agents.
Imo, this is not what users want. They want extension to their agent. If a project tells me I have to use their interface or agentic setup, it's 95% not going to happen. Consider how many SaaS tools we already have to deal with, that many agents is not desirable, they all have their little quirks and take time to "get to know"
Instead, build extensions, skills, and subagents that fit into my agentic workflow and tooling. This will also simplify what you need to do, so you can focus on your core competency. For example, you should be able to create a chat participant in VS Code / Copilot, and take advantage of the native notebook and diff rendering, sharing the MCPs (et al) the user already has for their agents for their internal systems.
> They want extension to their agent. If a project tells me I have to use their interface or agentic setup, it's 95% not going to happen
Yes, there’s definitely friction there. It may be that the right form factor is that you trigger Relvy’s debugging agent via Claude code / Cursor .
Our early users are heavy on needing to look at the raw data to be able to review the AI RCA, so a standalone set up makes sense. Also, the dominant usage pattern is background agentic execution triggered by alerts, and not manual.
Yup, we are moving up the ladders of abstraction and will have our agentic team interfaces that include agents triggered outside of human input. It does not change things. As soon as I need to go into the code or to the agent to fix the problem, I'm back to copy and pasting, or switching to view, between multiple interfaces. That's the kind of stuff we loathe
Runbooks are great and all, but actions need to be taken and I'm not going to give all the vendor interfaces to the internal systems. They can be subagents in my system which already has the tools and permission gates needed, access to code and git for IaC changes, etc...
It seems like the way to go now, it's easier to get moving and show off an experience and the vision, but it's definitely not the operational way in prod for a lot of reasons, security being a paramount one.
I also do not discount that your SaaS can be easily replaced by an open sourced subagent team in the next couple of years.
They claim a 12% lead (from 36% to 48%) over Opus 4.6 in a RCA benchmark: https://www.relvy.ai/blog/relvy-improves-claude-accuracy-by-...
heh, I was just about to post the following on your previous comment re: reproducible benchmark results. Thanks for posting the blog.
With the docker images that we offer, in theory, people can re-run the benchmark themselves with our agent. But we should document and make that easier.
At the end of it, you really would have to evaluate on your own production alerts. Hopefully the easy install + set up helps.
Interesting! In my experience using custom harnesses has worked better eg: Stripe etc all did it custom largely because of the sensitive integrations. How would you handle that?
Do you mean how we connect to internal data? Today, you can connect any API endpoint to Relvy, so if you have internal business data / dashboards that you look at while debugging, Relvy can do the same if there's an API for it.
Most of our deployments are self-hosted, in which case the data stays locally (your chosen LLM provider exempted), if that's what you are asking.
Re: custom harnesses, imo maintaining them can be time consuming especially when things are changing very fast with AI. Bringing up a prototype is easy but a robust harness that handles the edge cases needs time and effort.
Congrats on the launch! I dig the concept, seems like a good tool :)
Thank you :)
Woohoo!!! Congrats on the big launch y'all
This is a big one!! Congratulations guys :) Rooting for you!