During his KubeCon EU keynote, Vijay Samuel, Principal MTS Architect at eBay, shared his team’s experience of enhancing the incident response capabilities of the SRE teams by incorporating ML and LLMs building blocks. Through their journey, they realised that LLMs are not a silver bullet but can help engineers in complex scenarios, primarily if multiple capabilities are composed.
His introductory part framed the complexity of today’s platforms based on eBay’s infrastructure growth experience in the last five years. Their platform increased consistently, reaching more than four thousand microservices. The data generated by those translates into 15 petabytes of logs, 10 billion active time series per day, and 10 million spans per second (2% sample).
Samuel: How many dashboards should I look at before formulating a hypothesis?
Incident manual triage based solely on human capabilities was cumbersome and error-prone at this scale as it mainly involved trial and error. Samuel and his team experimented with different machine-learning approaches to enhance the process. The first step in moving away from static threshold-based alerts was Groot. This system attached a root cause to every alert that impacts a business KPI and even had auto-remediation capabilities for minor issues. Its anomaly detection algorithm decreased the time to detect incidents to under four minutes. As the system was learning from what he had experienced already, it failed whenever a new type of incident occurred.
Samuel and his team tried to embrace the LLM promise but quickly learned first-hand what hallucinations mean when trying to rewrite the Prometheus postings index using ChatGPT. He learned from his experiments that interacting with LLMs involves a lot of randomness whenever the context is broad (“triage the site” or “triage the alert”). Still, the output becomes more accurate if you prompt very deterministically with a very “crisp” context.
After their experiments, they concluded they could use LLMs’ capabilities for small amounts of information to assist them in their investigations. “Explainers” are small tools that focus on small amounts of information to provide more context while investigating an incident. They have capabilities that provide trace, log metric, or change explanations, all building blocks for the observability mechanisms.
Even though their progress convinced them that AI and engineers are complementary, the LLM’s limitations and misses were still massive for more significant information. Given the complexity of the ecosystem, they needed to divide and conquer the problem. Initially, they just extracted the critical path, inspired by Uber’s CRISP whitepaper, ensuring that all less important spans were removed. Afterwards, through “few shot prompting,” they provided examples of how SRE would normally triage those incidents to allow the algorithm to learn.
Further, to work with the context window limitations, they “dictionary encoded” everything and split the critical path into “upstream and downstream” chunks. Based on these, they generated partial explanations for each of those and combined them into a full explanation. The full explanation(using their basic building blocks, the explainers) allowed them to understand whether there were multiple performance issues.
They were composed into more complex evaluation mechanisms using each capability as a building block. Simple explainers were aggregated to analyse and explain more complex dashboards, and further triage workflows were built to analyse multiple dashboards. Also, alerts incorporated faulty traces whenever available to provide as much context as possible.
Samuel concludes his presentation by mentioning the potential evolutions of the system and what they learned from building it. For instance, adding all the metric metadata in a vector database would allow them to gain more insights, such as “how many searches failed” in a given period or “why was an SLO violated?” However, they also realised that LLMs are not a silver bullet but should be used for their strengths: simple reasoning, summarisation, internal knowledge search and even code generation at points. He also hints that broad OpenTelemetry adoption and query language standardisation would help make straightforward assumptions across the industry.
