Artificial intelligence and machine learning in observability

^{The Essential Guide to Observability}

Artificial intelligence and machine learning in observability

Data deluge

Digital services emit huge amounts of data from every interaction with the customer. This data, coming in the forms of logs, metrics, and traces, is generated by infrastructure (servers, virtual machines, containers, cloud, etc.), applications (your code, databases, web servers, etc.), clients (web, mobile, etc.), and security technologies (endpoint, SSO, etc.).

To make sense of this data, artificial intelligence (AI) and machine learning (ML) are critical components of observability.

Collected and analyzed effectively, this data can dramatically improve your team’s ability to support a high-quality customer experience.

Because of the volume, velocity and variety of data, modern observability solutions use AI/ML to sift through complex datasets to identify root-cause problems more efficiently, make recommendations and automate analyses

Most organizations have already started their journey to observability by cobbling together a collection of open source and proprietary "point products."

The more modern of these "point products" will have an ML/AI attached to it that is tuned to solve a specific use case within a specific domain, such as infrastructure monitoring, or application performance monitoring (APM).

This tuning means that it's great at understanding one type of metric, but not necessarily good at understanding how those metrics might relate to other types of telemetry.

End-to-end visibility cannot be realized in silos. Not only is it impractical, but it is inefficient to get answers from dispersed data. Insights from telemetry must draw from cross-platform AI/ML algorithms that detect anomalies and connect dots, processing data in volumes far beyond the human brain's capacity. For algorithms to perform to their potential, all data should be centralized.

^{The Essential Guide to Observability}

For algorithms to perform to their potential, all data should be centralized.”

^{The Essential Guide to Observability}

In his groundbreaking book, "The Myth of the Superhuman AI," Kevin Kelly talks about the fact that intelligence is a "combinatorial continuum."

Put more simply, the more information you feed into an AI, the more relationships it will be able to find and the more intelligent it will become.

Organizations looking to maximize their observability efforts will be well-served with a single unified repository of telemetry data, accompanied by a domain-agnostic AI that can look across all the data with full fidelity to detect and identify patterns and relationships.

AIOps platform offerings fallinto two categories: domain-agnostic and domain-centric solutions. However, requirements for increased flexibility for processing highly diverse datasets are having a significant impact on the market and shifting AIOps platforms toward domain-agnostic functionality.”

Gartner
Market Guide for AIOps Platforms
By Analysts Pankaj Prasad, Padraig Byrne,
Josh Chessman Published 6 April 2021

^{The Essential Guide to Observability}

How do you make your applications observable?

To make your apps observable, you need to design them so they emit enough useful signals to describe their state. That needs to happen across your delivery chain, so the app is equally observable on a dev’s laptop, in pre-prod and in production instances. This requires tight integration with something on the code level across all your apps.

Enter OpenTelemetry.

With the help of AI, observability solutions help humans make sense of all of this data.