<![CDATA[Notes by Doctor Droid]]>https://notes.drdroid.iohttps://cdn.hashnode.com/res/hashnode/image/upload/v1723284089567/30d18ee5-02b3-42cd-a3d6-b53a35289f14.pngNotes by Doctor Droidhttps://notes.drdroid.ioRSS for NodeFri, 01 May 2026 12:59:05 GMT60<![CDATA[Context Engine: How DrDroid's AI Agent leverages the Continuously Improving Knowledge Graph]]>Executive Summary

Production systems is a not a single component.

Kubernetes is a tangibly identifiable component with deterministic ways of operation. So is a database. And so is code. But Production Systems is a complex orchestration of different and discrete components that interact with each other to produce desired results for a business or user.

Joining together pieces, some of which are continuously changing, means that mathematically, the risk of failure is non-linearly higher than the risk of the individual components.

To tackle that, it is quintessential to design, iterate and architect systems with resilience, fallbacks, reliability and visibility.

DrDroid helps engineers leverage AI effectively for automating and accelerating operations. To enable high quality results, DrDroid has a stateful knowledge layer (context engine) about different components of production systems for agents.

The need for context engine

Engineers are using DrDroid to debug production alerts and get sharp data backed RCAs. To ensure that the agent is able to run complex investigations, is grounded in data and is context aware, we realised that a lot of reality is hidden in the day-to-day changes in the system. Here a few examples of typical changes, in sequence of most obvious to less obvious:

• Code Changes: Regular changes in a service in accordance with deliverables and goals

• Resource changes: A database table gets its memory upto the limit

• Indirect code changes: A third party dependency being used downstream in a service changes their API payload

• Indirect resource changes: A certificate expires on one of the machines where a critical service is running

• User change: A different user triggers an edge case in a product journey that never got used until now, leading to a failure for the user

None of these issues are unheard of, or shockingly new to an engineer. The problem is that the impact could be visible at a very different point in the system compared to where the issue originated.

In fact, for very different reasons, the same outcome/impact/alert might get triggered at different times.

Being able to mitigate and avoid such issues often require more information than just the alert. This debugging is often a combination of system knowledge, deductive reasoning, tribal knowledge, specialised skills and real-time situational context.

Added to that, engineers are expected to do this in high pressure situations or unexpected times as there are often business guarantees associated with these issues.

Components within a context engine

Our context engine is designed to proactively process and maintain an update knowledge layer about a company.

Here are some of the examples of what the context engine comprises of:

1. System Knowledge:

   1. Service Catalog: A list of every service, it's related teams and repositories, monitoring context and dependencies. This can auto-generated by fetching data across multiple tools like an APM or ArgoCD or Github repositories or Kubernetes. You can further edit the details if needed.

   2. Infrastructure inventory: A list of all the resources, from Databases to VMs to kubernetes deployments and namespaces.

2. Pre-instrumented metrics knowledge: The context engine is aware of all the metrics that are instrumented, as well as the dashboards and it's panels. This ensures that agent does not have to query prometheus by first principle but can re-use your existing dashboards too!

3. Codebase context: The context engine continuously maintains knowledge of the capabilities and features within each repo, their structure. It additionally also stores correlations with other repositories (depending on the information available). Read more on source code security.

4. Third party dependencies: Within DrDroid, you can maintain a list of all 3rd parties that your application depends upon.

5. Assistance for deductive reasoning:

6. Service & infrastructure correlations: By leveraging the context of your inventory across infrastructure & services, and your traces/service maps, correlations between services and infrastructure is auto-processed. This can be built using your existing service maps from APM / traces or using network mappers in your kubernetes infrastructure (available with our reverse proxy service)

https://www.youtube.com/watch?v=Bt-G0ZRaj-8&t=2s

2.  **Pre-compiled log patterns and fingerprints:** Your log patterns are overtime stored within the context engine, ensuring that the agent can re-use the past experience of log analysis to improve the future log search queries.

3.  **Pre-mapped data flows and critical user journeys:** The agent maintains context about the product/architecture and can even keep context of critical workflows across repositories. Additionally, these can be edited to add additional insights and knowledge.

1. Tribal Knowledge:

2. Pre-existing runbooks and SOPs: All your existing runbooks and SOPs can be added to the context engine, either via syncing it with your wiki or uploading/creating documents in the account.

3. Past Incidents & investigations: The context engine is continuously updated as the agent runs alert investigations and RCAs. Insights of correlations and issues identified are stored for easy referencing and searching for the agent in the future.

4. Alert Patterns: The context engine stores alerts from Slack channels and webhooks, ensuring the agent can see the pattern of a specific alert and differentiate an anomaly from a regular alert.

5. Communication rosters: The context engine stores information about the teams and the users within the teams. Users can be auto-synced from Slack where as the teams can be simply configured in the platform.

6. Specialised skills

7. Internal tools specific knowledge: You can store guidelines about how your internal CLI / tools operate. This would enable the agent to use the tools effectively, ensuring your production workflows can be replicated here.

8. Best practices on trace analysis: Analysing traces can often be taxing for any engineer and require deep focus to identify the anomaly in them, across the time and spans. The context engine has specialised tools for trace analysis, enabling it to get insights on large volume of spans in a short duration with reduced token consumption.

9. Observability tools usage: Every tool has it's own data structures and entity design. Best practices around using tools like Signoz, NewRelic, AWS, Kubernetes, etc. are pre-baked into the context engine.

10. Real-time situational context

    1. Code Changes: The context engine continuously processes PRs, commits and merges across repositories. This ensures that the agent can get a quick glance on recent changes.

11. Infrastructure Changes: Addition of inventory in the infrastructure, from a database in your cloud to a namespace in your kubernetes cluster, the diff is continuously tracked.

12. Ongoing issues and alerts: Alerts are processed and saved in real-time for the agent to use in real-time.

13. Vendor downtime events: The context engine is continuously querying the status pages of the third party vendors identified in the platform. This ensures that the agent can be notified upfront during an investigation if a downstream vendor API is impacted.

Where is the context engine used?

How does the context engine get used in the platform:

1. Investigation Agent: The investigation agent is our flagship agent which can help you investigate any issue in your production system - from alert investigation & service degradation to security review & cost analysis.

2. Alert Classification/Triaging Agent: Just send all your alerts to DrDroid. When you receive a large stream of alerts, this agent can classify your alert to be a noisy alert (and suppress it) or let that alert come through to you. It grounds every recommendation in actual system data. As you get started, it gives benefit of doubt to an alert being critical (so that a critical issue isn't missed) but over time, it learns from already triaged alerts, data queried and user feedback received. Users can also add specific "notes" at alert type level so that agent doesn't have to think from first principles when it gets started.

3. Alert Grouping Agent: If you have 100+ relevant alerts coming in, rarely are they 100 unique issues. More often than not, these might end up becoming 7 or 19 issues. The Grouping agent listens to your alerts and tries to group it into actionable buckets - it often does it by root cause (found by investigation agent) or by the impacted component / where it fits in the topology. This could ensure that during an incident or when you get a surged stream of alerts where most of them are due to the same issue, the one other signal doesn't get missed.

4. Continuous Improvement Agent: The core principles behind engineering operations is to react, learn and improve. Some teams often get too occupied with firefighting and do not get the bandwidth to work on proactive improvements of the system. This agent gives suggestions for improving your alerts, missing observability data for any of your new services, gives cost or security posture recommendations.

5. Meta agent: This agent helps you to improve context on the DrDroid platform based on ongoing activity, which will in-turn, make all the other agents more powerful.

How is the context engine implemented?

Read more about the design of the context engine and it's architecture in this blog.

]]>https://notes.drdroid.io/context-engine-how-drdroid-s-ai-agent-leverages-the-continuously-improving-knowledge-graphhttps://notes.drdroid.io/context-engine-how-drdroid-s-ai-agent-leverages-the-continuously-improving-knowledge-graph<![CDATA[How DrDroid AI SRE Agent is specialised for Production Incidents & On-call Investigations]]>By working with 100s of engineers and their debugging problems, we iterated over DrDroid. The investigation agent assists engineers with complex analysis which are critical, time sensitive and have little room for error. Here are some of the things that help the investigation agent perform well:

1. Specialized Debugging Tools & Skills

Production incidents often require analyzing large volumes of logs, traces, and metrics. This can be token-intensive and time-consuming. Most LLMs and agentic frameworks hit context window limits quickly, can't process production-scale data, and lose quality when analyzing large datasets.

How DrDroid Solves This:

Pre-Built Aggregate Analysis Tools

Instead of feeding raw logs into an LLM, DrDroid has tools designed specifically for handling large-volume logs with:

• Built-in log aggregation and pattern detection

• Complex trace analysis across distributed systems

• Large-volume metrics analysis with outlier detection and ML techniques

Specialized Investigation Skills

Our agent has domain-specific skills built from working with hundreds of engineers on real debugging problems:

• How to query and analyze traces in Signoz or Datadog

• How to navigate APM data efficiently

• How to correlate metrics across multiple monitoring tools

Real Impact: The agent can process 100,000+ log lines in seconds and surface the 5 relevant errorssomething that would exhaust a generic LLM's context window.

2. Code & Application Awareness

Most LLMs and agents start every investigation from zero, with only the context of the prompt and sometimes markdown files.

How DrDroid Solves This:

Automatic Code Context Generation

Even before your first chat with the agent, DrDroid builds knowledge of:

• What each repository does

• What capabilities, APIs, features, and workflows each repo covers

• Programming languages, frameworks, and file structures

• Connections between multiple repositories (discovered via traces and logs)

Business Workflow Understanding

You can ask DrDroid to build context around critical business and product workflows. The agent understands:

• "The checkout flow involves payment-service, inventory-service, and notification-service"

• "When users report 'payment stuck,' check these three services in this order"

Real Impact: When an alert fires on "payment-service," DrDroid already knows what that service does, which other services depend on it, and where to look for root causes.

3. Infrastructure & Resource Awareness

An LLM with MCP connections doesn't know which apps run in which Kubernetes clusters, which databases are in which cloud providers, or how your infrastructure is organised. It needs to query multiple tools (costing time and token) and explore it before it's able to answer that.

How DrDroid Solves This:

Auto-Discovery of Infrastructure

DrDroid continuously maps:

• Apps hosted in different Kubernetes clusters

• Databases and their cloud providers

• Service dependencies and communication patterns

• Network topology and resource relationships

Service Map & Dependency Graph

The agent can answer questions like:

• "Which services depend on the payments database?"

• "If eu-west-1 goes down, what's affected?"

• "Show me all services running in the production cluster"

Real Impact: During an incident, the agent instantly knows the blast radius and which downstream services might be affected without you having to explain your architecture.

4. Past Alert & Incident Pattern Recognition

With generic agents, every investigation is independent. It has no memory of past incidents or patterns.

How DrDroid Solves This: Searchable Alert History

The agent has access to:

• All alerts since platform enablement

• Past incidents and their resolutions

• RCAs and postmortems (from Confluence, docs, or previous investigations)

• Understanding patterns in alerts

When similar issues occur, the agent can say:

• "This looks similar to the incident from Jan 15th where the Redis cache was full"

• "Last time this alert fired, the root cause was a config change in service X"

Real Impact: Repeat incidents get resolved faster because the agent learns from past investigations.

5. Continually Learning System

DrDroid improves with every investigation.

Active Learning from Your Environment

The agent continuously creates notes and memory from:

• Recent commits and merges in your applications

• Investigations and conversations with the agent

• Human conversations in Slack channels (optional)

Contextual Memory Storage

Everything is stored with metadata:

• Timestamp

• Related entities (services, databases, clusters)

• Related team and people

• Relevant tags and categories

Real Impact: The agent gets smarter every week. After a month, it knows your environment better than most new engineers.

6. Context Compaction (1M+ Token Conversations)

Typically, agents do the following with the problem of large context windows:

• Summarize the entire conversation (losing critical context) or

• Hit token limits and can't continue

• Slow down dramatically as conversations grow

With production telemetry data, this can often happen.

How DrDroid Solves This:

Intelligent Compression Without Context Loss

• Tool calls are compressed (only IDs and summaries preserved)

• Reasoning and train of thought remain intact (no summarization)

• Agent maintains full context even beyond 1M tokens

• Smart Tool-Level Compaction

Our tools have built-in context management:

• Logging tool has grep/search capability over large volumes

• Agent can "eyeball and search" logs instead of loading everything into context

Real Impact: You can have a 2-hour debugging session with 500+ tool calls, and the agent never loses context or slows down.

7. Multi-Channel Conversations with Shareability

The agent is designed to work from your place of convenience:

• Slack DMs

• Thread replies to alerts

• Web UI

• CLI (coming soon)

• API triggers

• Voice calls (coming soon)

Seamless Sharing

Any investigation can be:

• Shared with teammates for review

• Linked in postmortems

• Referenced in future incidents

Real Impact: When someone gets paged, they can see the auto-investigation that already ran in the Slack threadno need to DM the agent separately.

8. Automated Investigations

DrDroid can run proactively or via automated triggers enabling proactive visibility for your team:

• Alert fires in PagerDuty/OpsGenie Investigation starts automatically

• Cron-based health checks Agent investigates on schedule

• Custom triggers via API or webhooks

Real Impact: Agent can detect issues even without alerts; By the time you open the alert, the agent has already investigated and summarized the likely root cause.

9. Smart Model Switching (85% Cost Savings)

LLMs have been commoditised and the SOTA model is not necessarily required for every investigation. DrDroid smartly chooses between different LLMs based on investigation complexity

• Simple tasks Faster, cheaper models

• Complex reasoning State-of-the-art models

Real Impact: Up to 85% token savings compared to always using frontier models, with no degradation in investigation quality.

10. Dedicated File System & Memory

Memory management for a large scale infrastructure requires a structured approach.

DrDroid has a Persistent Knowledge Base - All context, memory, investigations, and alerts are stored and accessible:

• Agent can navigate past investigations like files

• Search across all historical data

• Reference previous findings instantly

Real Impact: "Show me all investigations related to database timeouts in the last 30 days" returns instant results.

11. Coding Sub-Agent for Hotfixes

Coding agent operates very different from a production investigation agent. DrDroid comes pre-packaged with a coding agent connected to the investigation agent.

Real-Time Coding Agent When needed:

• Spins up a coding agent in an ephemeral sandbox

• Reviews the full repository

• Creates hotfix PRs with proper context

Real Impact: During an incident, the agent can say "I found the bug in payment-service line 247here's a PR to fix it."

12. Remote Machine & Kubernetes Access

Often, data needs to be reviewed on a remote machine or kubernetes cluster for production incidents. These might be inaccessible or sensitive.

How DrDroid Solves This: Direct Infrastructure Access without token access to the agent

• Execute commands on remote machines via SSH (keys are not exposed to the agent)

• Query read-only Kubernetes clusters directly

• Access VMs and clusters within your VPC via reverse proxy

Real Impact: "Check disk space on prod-api-01" Agent SSHs in, runs the command, and returns results. No manual execution needed.

13. Image Support for Dashboard Analysis

You might want to debug an issue with a screenshot shared by the customer as the starting point.

How DrDroid Solves This: DrDroid agent support image processing from Slack or UI.

• Your product showing an error

• A Grafana dashboard

• A monitoring alert

The agent analyses it and continues the investigation from there.

Real Impact: "Here's what the user is seeing" Agent understands the UI issue and investigates the backend cause.

14. Granular Access Control & RBAC

Production systems debugging come with sensitive data and access management. DrDroid ensures only the right people have the right access while debugging.

How DrDroid Solves This:

• Read commands: Execute without approval (safe exploration)

• Write commands: Require RBAC approval per your policy

• SSO integration: Syncs with your internal permissions

• Audit logs: Track who did what

Real Impact: Junior engineers can investigate safely, while dangerous operations require senior approval.

15. Third-Party Vendor Status Tracking

Often production incidents can be partly caused due to 3rd party downtimes or issues.

How DrDroid Solves This: Connected to Vendor Statuspages

• Tracks 150+ status pages for your third-party vendors: Stripe, AWS, Datadog, MongoDB Atlas, etc.

• Flags when vendor issues might be causing downstream impact

Real Impact: "Is this our issue or Stripe's?" Agent checks Stripe's status page and correlates timing.

16. Automated Quality Evaluation

Production Agents need to come with quality guarantees for the team to track and trust.

How DrDroid Solves This: LLM-Based Evals on Every Investigation

Every investigation is automatically evaluated for:

• Accuracy Safety Errors or hallucinations

• Central teams get visibility into investigation quality and improvement opportunities.

Real Impact: Platform team can see "Investigation quality is 94% this month, down from 97% last monthlet's review the low-scoring investigations."

17. User Feedback & Team Visibility

Context within DrDroid can continuously improve over time. But for that to improve, tracking and acting upon user feedback is critical.

How DrDroid Solves This: Collaborative Quality Control

• Every investigation can be upvoted/downvoted

• Feedback back to central team helps improve agent context

Real Impact: Central team & managers have visibility on confidence and impact of AI on the engineers.

18. Reasoning Lifecycle & Audit Trail

Production incident investigations cannot be led to be incorrect due to "hallucinations" or "guesses" by an LLM. DrDroid ensures that every reasoning and logic by the LLM is grounded in facts and data.

How DrDroid Solves This: Transparent Investigation Path

The agent tracks:

• What data it queried

• Why each data point was relevant

• What hypothesis it built from each finding

• How it reached its conclusion

Real Impact: You can backtrack through the investigation to validate correctness, spot gaps, or understand the agent's reasoning.

Summary: Why DrDroid is Purpose-Built for Production

What This Means for Your Team:

• Every investigation starts with past context

• You do not have to guide the LLM or explain your architecture every time

• It can handle production-scale logs or metrics

• It supports automation or proactive help

• It works across different channels where your team lives

Ready to See the agent?

DrDroid is a purpose-built investigation agent that understands your infrastructure, learns from your incidents, and gets smarter every day.

Next steps:

• Watch platform demo videos

• Check our MCP Servers & integrations

• Read the documentation

• See customer case studies

Want to see how it works with your stack?

Setup & go live takes 1-2 hours for smaller teams, < 1 week for enterprises. Get started here.

]]>https://notes.drdroid.io/how-drdroid-ai-sre-agent-is-specialised-for-production-incidents-on-call-investigationshttps://notes.drdroid.io/how-drdroid-ai-sre-agent-is-specialised-for-production-incidents-on-call-investigations<![CDATA[DrDroid: How AI SRE Helps Engineers who are on-call for production monitoring]]>Day 0 Incident Response & Firefighting

Something is broken right now. You need answers fast.

Day 1 Operational Tasks & Maintenance

No fire, but you need to keep things running smoothly.

Day 2 Automation, Optimization & Reliability

You want to stop doing repetitive things and build resilience.

What's Next?

DrDroid helps your team move from reactive firefighting to proactive operations. Whether you're debugging an incident at 3 AM or building automation to prevent the next one, DrDroid becomes your team's operational co-pilot.

Ready to see how DrDroid works with your stack?

• Watch platform demo videos

• Check our integrations

• Read the documentation

• See customer case studies

Want to try it out? Setup takes 1-2 hours and you'll see value from your first investigation. Get started here.

]]>https://notes.drdroid.io/drdroid-how-ai-sre-helps-engineers-who-are-on-call-for-production-monitoringhttps://notes.drdroid.io/drdroid-how-ai-sre-helps-engineers-who-are-on-call-for-production-monitoring<![CDATA[KubeCon + CloudNativeCon Europe 2026 Guide – Amsterdam
]]>Agenda Strategy, Tracks, Networking & SRE Playbook

Updated

March 2026 6 min read

On this page

• Doctor Droids Guide to KubeCon + CloudNativeCon Europe 2026

• Overview

• Event Schedule

• Who Should Attend (and Who Can Skip)

• How to Build Your Agenda (Without Overloading Yourself)

• Track Strategy for SRE & Platform Teams

• Co-located Events: Where Specialists Get Leverage

• Solutions Showcase: How to Avoid Vendor Fatigue

• Networking Strategy for Engineering Leaders

• Amsterdam Logistics Checklist

• Post-Conference Execution Plan

• Visit the Doctor Droid Booth

Doctor Droids Guide to KubeCon + CloudNativeCon Europe 2026

KubeCon + CloudNativeCon Europe is heading to Amsterdam, Netherlands, from 2326 March 2026.

If youre on platform engineering, SRE, DevOps, or cloud architecture teams, this event is still one of the best places to compress a year of learning into four days.

This guide is for practitioners who want outcomes, not conference FOMO:

• what to prioritize

• how to choose tracks

• how to plan networking

• and how to convert conference notes into production improvements

Overview

KubeCon + CloudNativeCon is the flagship conference organized by the Cloud Native Computing Foundation (CNCF). It gathers thousands of engineers, maintainers, and infrastructure leaders working on Kubernetes and the broader cloud-native ecosystem.

The event typically features:

• Major Kubernetes and CNCF ecosystem announcements

• Technical deep-dives from engineers running large-scale production systems

• Co-located events focused on specialized technologies

• The Solutions Showcase, where cloud-native vendors demonstrate tooling across observability, security, infrastructure automation, and platform engineering.

For engineering teams operating production Kubernetes environments, KubeCon acts as a yearly checkpoint for infrastructure strategy.

Event Schedule

According to the Linux Foundation event page, KubeCon + CloudNativeCon Europe 2026 is scheduled in Amsterdam from 2326 March 2026.

Expected structure:

Day 0 (Monday)

Pre-event programming and co-located events

Days 13 (TuesdayThursday)

Keynotes, breakout sessions, and Solutions Showcase

If youre traveling internationally, plan to arrive by Sunday evening so you can still attend Mondays co-located tracks.

These smaller events often contain some of the most advanced implementation discussions of the entire conference.

Who Should Attend (and Who Can Skip)

Attend if your team is currently dealing with:

• Reliability issues across Kubernetes clusters

• Scaling bottlenecks (multi-cluster, noisy neighbors, cost/perf tradeoffs)

• Incident response gaps between alerts and root cause

• AI adoption in platform engineering workflows

• Security or compliance friction in cloud-native stacks

You can skip (or send one delegate) if:

• your stack is stable and not evolving

• youre not planning infrastructure changes in the next 612 months

• or you mainly need vendor procurement meetings

KubeCon ROI is highest when your team has active infrastructure pain and concrete architecture decisions pending.

How to Build Your Agenda (Without Overloading Yourself)

Most engineers fail KubeCon by overbooking sessions.

A better approach:

1) Define 23 decision themes before the event

Examples:

• Should we move from single-cluster to multi-cluster for resilience?

• How should we harden runtime security without killing developer speed?

• Where can AI actually reduce MTTR in incident response?

Use these themes to filter sessions.

If a talk doesnt help a current architecture decision, skip it.

2) Split your day into three buckets

Each day should include:

• One depth session (deep technical talk)

• One trend session (strategy or ecosystem direction)

• One practical session (case study with production lessons)

This avoids the common trap of attending 100% hype talks or 100% dense internals.

3) Reserve white space

Keep at least 90 minutes per day unscheduled.

Some of the highest-signal learning at KubeCon comes from hallway conversations, not slides.

Track Strategy for SRE & Platform Teams

If your mandate is reliability + developer speed, these themes should be top priority.

Reliability engineering in Kubernetes

Look for sessions covering:

• failure domains and blast radius control

• progressive delivery and rollback safety

• SLO design in distributed systems

• incident retrospectives with architecture changes

AI for operations (without magic claims)

Prioritize talks that show:

• real workflows (alert triage, runbook assistance, anomaly explanation)

• measurable outcomes (MTTR reduction, false positive reduction, toil reduction)

• guardrails like human-in-the-loop systems and auditability

Scaling and performance

Focus on talks about:

• control-plane scaling

• multi-tenancy isolation

• workload scheduling optimization

• cost/performance tuning with real production metrics

Security and policy at scale

Strong sessions usually cover:

• software supply chain controls

• runtime policy enforcement

• identity and secrets management patterns

• tradeoffs between security and developer experience

Co-located Events: Where Specialists Get Leverage

Monday co-located events are often where advanced implementation patterns surface early.

If your team has niche challenges around:

• service meshes

• observability pipelines

• policy engines

• platform APIs

these tracks may provide better signal than general keynotes.

Recommendation:

Send at least one engineer to co-located sessions and have them summarize the takeaways for your team.

Solutions Showcase: How to Avoid Vendor Fatigue

The expo floor can become overwhelming quickly.

Treat it like a technical discovery sprint.

Before visiting booths, define your constraints:

• existing observability stack

• data residency and compliance requirements

• budget range

• integration requirements (Datadog, Grafana, PagerDuty, CloudWatch, Slack, etc.)

Ask every vendor the same 5 questions

1. What production scale do your reference customers run?

2. What does deployment look like in week one?

3. What are the common failure modes of your product?

4. How do you integrate with existing incident workflows?

5. What metrics improve in 30 / 60 / 90 days?

If answers remain high-level, move on.

Networking Strategy for Engineering Leaders

Skip generic networking.

Optimize for targeted conversations.

Try to meet:

• 3 peers running similar Kubernetes scale

• 2 teams that recently migrated tooling youre evaluating

• 2 maintainers from critical OSS dependencies

Questions worth asking

• What broke after rollout?

• What did you underestimate?

• What would you do differently in year two?

Answers to these questions can save months of trial and error.

Planning Your Amsterdam Experience

Where to Stay

Amsterdam offers many accommodation options close to the event venue.

Options include:

• Hotels near the conference center

• Short-term apartment rentals via Airbnb or Booking.com

• Budget hostels for solo travelers

Booking early is recommended because KubeCon events tend to sell out nearby hotels quickly.

Getting There

Amsterdam is served by Amsterdam Schiphol Airport (AMS), one of Europes largest and most connected airports.

From Schiphol Airport:

• Direct trains connect to Amsterdam Central Station

• Metro and tram networks provide quick access to most parts of the city

Public transit is generally the easiest way to move around during the conference.

Visiting Amsterdam

If you have extra time, Amsterdam offers plenty to explore:

• Rijksmuseum Dutch art and history

• Anne Frank House historic museum and cultural landmark

• Canal boat tours through the historic city center

• Jordaan district for cafs and restaurants

Evening community meetups during KubeCon are often hosted across the city.

Amsterdam Logistics Checklist

Practical tips for conference week:

• Arrive at least one day early for registration and timezone adjustment

• Stay near the venue or along a direct transit line

• Keep evening slots free for community meetups

• Carry a lightweight note template for each session

Suggested note template:

• Problem addressed

• Architecture pattern used

• Scale context

• Results or metrics

• Relevance to your environment

Post-Conference Execution Plan (The Part That Matters)

Conference ROI is realized after you get back.

Within 72 hours

1. Consolidate notes into themes (reliability, AI, scaling, security).

2. Rank ideas by effort vs impact.

3. Choose 2 quick wins and 1 strategic bet.

Within two weeks

• Run one architecture review based on KubeCon learnings

• Launch one pilot experiment with explicit success metrics

• Share an internal write-up:

What we learned, what were changing, expected impact.

Without this step, even great conference insights decay quickly.

Visit the Doctor Droid Booth

Doctor Droid is the AI-powered Slack bot for faster incident diagnosis.

It helps engineering teams identify the root cause of production issues automatically by analyzing alerts, logs, and system signals.

At KubeCon + CloudNativeCon Europe 2026, stop by the Doctor Droid booth to:

• See live demos of AI-driven incident investigation

• Explore how teams reduce MTTR and alert fatigue

• Grab exclusive Doctor Droid swag and giveaways

You can also schedule a one-on-one demo with our team:

https://calendly.com/siddarthjain/doctor-droid-discovery-call

Get Early Access & Updates

If you're interested in Doctor Droid demos, credits, or updates during KubeCon, sign up here:

https://forms.gle/hPzaMa4YRqDLzHZg6

[P.S. - You get 20% off on tickets as well :) ]

Source Note

Event date and high-level schedule are based on the official Linux Foundation KubeCon + CloudNativeCon Europe event page (accessed March 2026).

]]>https://notes.drdroid.io/kubecon-cloudnativecon-europe-2026-guide-amsterdamhttps://notes.drdroid.io/kubecon-cloudnativecon-europe-2026-guide-amsterdam<![CDATA[Backtesting AI Agents: How SRE Teams Prove Reliability Before Production
]]>

AI agents are finally showing up inside real incident workflows. One agent triages alerts, another scrapes dashboards, a third drafts the remediation plan. Yet 62% of organizations experimenting with agents admit they still cannot run them reliably in production because demos rarely expose variance, safety, or cost failures ( Codebridge).

Backtesting is how SRE teams close that gap. Instead of lets ship and see, you treat agents like a new microservice: define reliability budgets, hammer them with synthetic and real traces, and fail the build until you trust every path.

This guide shows how to build an AI-agent backtesting program that mirrors load testing for infrastructure. It leans on Codebridges reliability dimensions, the AI Reliability Institutes 30-point checklist, modern agent-observability stacks, and DrDroids native context graph plus guardrail center.

1. The wake-up call: why AI agents need pass^k reliability

A single happy-path demo is meaningless when the production pager expects deterministic success. Codebridges recent survey highlights the reliability delta clearly:

- Prototype bias. Teams measure whether a workflow completes once, under ideal prompts, then extrapolate to production. In reality, single-run success rates of 60% often translate to only 25% full consistency when you rerun the same scenario 10+ times ([ Codebridge])

- Cost spikes hide in the tail. Architectures like Reflexion or self-reflection loops inflate token usage by 5.12 for marginal accuracy gains; without cost-normalized evaluation you do not see the runaway invoice until after launch (same source).

- Trust is earned, not promised. Venture teams Codebridge interviewed said more than 70% of execs only greenlight broader automation once they see formal evidence of safety controls, loop detection, and kill switches.

Treat agent validation like you treat capacity planning. Define agent SLOs (Mean Time to Context, Agent-Assisted MTTR, Unauthorized Action Budget). Require pass^k (all trials succeed) instead of pass@k (one success out of many). Every failed attempt becomes a regression test before the agent is allowed anywhere near the on-call rotation.

2. Five reliability dimensions to measure every run against

Codebridge frames reliability as a system property, not just accuracy. Their five dimensions map cleanly to the levers SRE teams already manage:

Backtesting should emit metrics for each dimension. Examples:

- Consistency: For every golden scenario, run 20 Monte Carlo trials. Alert if success <95% or if token usage swings >30% between runs.

- Robustness: Randomly perturb prompts (create a rollback vs. can you undo the deploy). Evaluate success delta and force remedial prompt hardening when regression >10%.

- Predictability: Require agents to emit confidence scores for risky actions. Route anything under 0.7 to human approval. Compare claimed confidence to measured success to compute Brier scores.

- Safety: Enforce negative constraints in tests (Do not email this alias, Do not touch prod DB) and fail the build if the agent even attempts the blocked action.

- Infrastructure: Track per-session token, tool, and latency budgets inside DrDroids guardrails center. Attempts to exceed a $2 reasoning budget trigger the kill switch before the vendor invoice hits.

3. Designing the backtest dataset: golden, edge, adversarial, regression

A strong dataset mirrors the risk surface. Codebridge recommends this split ([ same source]):

- 20% Golden paths. Known-good workflows that mirror typical incidents.

- 30% Edge cases. Ambiguous alerts, partial telemetry, missing runbooks.

- 20% Adversarial. Prompt injections, malicious tool outputs, conflicting human directives.

- 30% Regression. Every failure ever seen in prod becomes a permanent test.

Layer in AI Reliability Institutes 30-point checklist to make sure you are covering loop detection, denial-of-wallet defenses, zombie-process cleanup, policy insubordination, and kill switches ([ AIRI]. DrDroids droidctx makes populating these scenarios easier because it keeps a living graph of alerts, dashboards, service owners, and incident annotations. You can:

1. Auto-generate golden cases from resolved incident timelines (alerts + deploy notes + Slack transcript).

2. Synthesize edge cases by perturbing telemetry (drop 20% of log lines, rename dashboards) and exporting them into the test harness.

3. Maintain adversarial suites by piping AI Reliability Institutes negative-constraint tests (ignore the guardrail) straight into the prompt injection lane.

4. Promote regressions automatically every time an agent fails in staging or prod; DrDroids Slack-native workflows capture the trace and push it into the regression bucket.

4. Layered graders: deterministic checks, agent-as-a-judge, and humans

A dataset without trustworthy graders is just fan fiction. Codebridge outlines a layered verification model that mirrors classic testing pyramids:

1. Deterministic graders (code) verify objective outcomes: did the runbook markdown change, did the Kubernetes deployment roll back, did the SQL diff match expectations.

2. LLM-as-a-judge (AaaJ) handles subjective traits like clarity of Slack updates or whether the hypothesis actually explains the alert. Codebridge cites AaaJ frameworks achieving ~90% agreement with humans when they gather their own evidence, while cutting review cost by 97%.

3. Human-in-the-loop remains the final gate for irreversible actions (database writes, customer communications, pager handoffs).

DrDroid bakes these layers into its guardrail center:

- Guarded tool schema: Every tool call runs through JSON schema validation; failing schema equals instant fail.

- Agent approval workflows: High-risk actions appear in Slack with context, metrics, and a CONFIRM field so humans cannot rubber-stamp blindly.

- Trace exports: Each run captures the entire reasoning trace so deterministic, model-based, and human graders all work from the same evidence.

5. Tooling landscape: sim rigs, observability stacks, and when to extend beyond DrDroid

Even with DrDroids native tracing, teams often mix in specialist eval stacks for breadth. The Maxim AI roundup of agent-testing platforms is a useful cheat sheet ( GetMaxim):

- Maxim AI. Full lifecycle (experiment simulate evaluate observe) with distributed tracing, llm-as-a-judge, and AI gateway controls. Great when product managers need no-code scenario builders.

- Langfuse. Open-source tracing for teams who want to self-host every span.

- Arize. Extends classic ML observability (drift, dashboards) into LLM workloads, ideal for enterprises already running Arize for models.

- Opik (Comet). Lightweight trace logging plus evals when you need quick wins.

- DeepEval. Pytest-style evaluator infrastructure for engineering-heavy orgs building custom metrics.

How this pairs with DrDroid:

- Use DrDroid for incident-native context (alerts + deploys), permissions, and Slack workflows.

- Pipe traces to Langfuse/Maxim if you need deeper span-level analytics or cross-product dashboards.

- Feed evaluation metrics back into DrDroids SLO board so on-call engineers see Agent backtest coverage: 86% alongside service health.

6. Operationalizing backtests with DrDroid

Heres a practical loop SRE teams can implement in a sprint:

1. Ingest traces. Enable DrDroids trace exporter for every staging and prod run. Capture prompts, tool calls, guardrail hits, latency, and cost.

2. Generate scenarios. Use the captured traces plus droidctx to auto-build the golden/edge/adversarial suite. Store them in a repo so they version with code.

3. Wire graders. Start with deterministic checks (e.g., pytest verifying Grafana API responses). Add llm-as-a-judge jobs via Maxim or DeepEval for subjective signals. Route high-risk failures to a Slack approval queue.

4. Automate pass/fail gates. Add a backtest job to CI that runs the full suite on every scaffold change. Block merges unless success 95%, safety violations = 0, cost variance <30%.

5. Publish SLOs. DrDroid dashboards should show Agent MTTC, Assisted MTTR, unauthorized-action budget, and coverage (# of alerts where agents participated). Treat SLO breaches exactly like service SLO breaches: open incidents, run postmortems, add regressions.

6. Keep humans in control. The AIRI checklist mandates kill switches, loop detection, DoW limits, and policy-insubordination tests. DrDroids guardrail center exposes all of them in one UI so on-call engineers can yank access in <200 ms if the agent drifts.

Backtesting isnt a one-time certification. Its a living discipline where every production event becomes a new test. When you plug DrDroids context engine, guardrails, and aggregated observability into that loop, AI agents stop being unpredictable copilots and become accountable teammates who earn their time on the pager.

Once you have datasets, graders, and tooling in place, the next step is designing the evaluation pipeline itself.

Evaluation architecture: how agent backtests actually run

Backtesting requires more than datasets and metrics. Reliable agent systems separate execution, trace capture, and evaluation into a structured pipeline.

A typical evaluation architecture looks like this:

Scenario Dataset

Simulation Harness

Agent Execution

Trace Capture

Evaluation Pipeline

CI Pass/Fail Gate

Each layer plays a specific role in validating reliability.

This separation ensures engineers can test agents the same way they test distributed systems.

Instead of manually inspecting runs, every execution generates structured traces and evaluation metrics.

Testing taxonomy for AI agents

Backtesting is only one layer of the testing strategy. Mature teams build a testing pyramid similar to traditional software engineering.

Each layer catches different classes of failures.

1. Unit tests

Unit tests validate the smallest components of the agent system.

Typical unit tests include:

• tool schema validation

• prompt template formatting

• guardrail logic

• JSON output validation

Example:

assert tool_schema.validate(agent_output)

These tests are deterministic and run in milliseconds.

They prevent simple failures from reaching higher-level tests.

2. Integration tests

Integration tests validate interactions between agents and infrastructure tools.

Examples:

• querying observability dashboards

• executing Kubernetes rollbacks

• posting Slack updates

• retrieving runbooks

These tests confirm that the agent can actually interact with the systems it relies on.

Failures here often come from:

• API schema changes

• authentication issues

• permission errors

3. Simulation tests

Simulation tests run agents in controlled synthetic environments.

Typical simulation features:

• replay alert streams

• mock tool responses

• inject telemetry noise

• simulate partial failures

Example simulation scenario:

Alert: CPU spike on checkout-service
Telemetry: 20% logs missing
Tool latency: +2 seconds

The goal is to test robustness under imperfect conditions.

Simulation environments often expose reasoning failures that do not appear in ideal demos.

4. Backtests

Backtests replay real incidents from production.

These are the most valuable tests because they contain realistic context:

• real alerts

• real dashboards

• real Slack conversations

• real deploy timelines

The agent attempts to resolve the incident using the same information that engineers had during the original outage.

Backtests validate:

• decision quality

• operational safety

• cost stability

This is where pass^k reliability becomes important.

If an agent succeeds once but fails on repeated runs, it cannot be trusted in production.

Eval-as-a-judge in the evaluation pipeline

Many agent outcomes cannot be evaluated using deterministic checks.

For example:

• Is the root cause hypothesis plausible?

• Is the Slack update clear to on-call engineers?

• Did the agent follow incident response policy?

This is where Eval-as-a-Judge (EaaJ) is useful.

An evaluation model reviews the agent output and scores it according to defined criteria.

Example evaluation prompt:

You are an SRE evaluating an incident response.

Alert:
CPU spike on checkout-api

Agent response:
"Root cause likely a memory leak introduced in version v1.3.2."

Evaluate:
1. Is the hypothesis plausible?
2. Is the remediation safe?
3. Did the response follow policy?

Return:
score (0-1)
justification

Eval-as-a-judge works well because it can evaluate semantic correctness and reasoning quality, which deterministic tests cannot capture.

Best practice is to combine three layers:

This layered grading system dramatically improves evaluation reliability.

Incident replay: the most powerful backtesting tool

The most valuable evaluation dataset is your own incident history.

Replay systems reconstruct the context of past outages using:

• alerts

• logs

• dashboards

• deployment events

• Slack threads

Agents then attempt to resolve the incident as if it were happening live.

Benefits include:

• realistic test scenarios

• automatic regression generation

• continuous learning from production failures

Every production incident can become a permanent regression test for the agent.

Over time, the backtest suite becomes a living archive of operational knowledge.

]]>https://notes.drdroid.io/backtesting-ai-agents-how-sre-teams-prove-reliability-before-productionhttps://notes.drdroid.io/backtesting-ai-agents-how-sre-teams-prove-reliability-before-production<![CDATA[AI in Engineering: 6 Trends That Will Define 2026]]>The way engineering teams build, ship, and operate software is undergoing a fundamental shift. In 2025, we saw AI move from code autocomplete to genuine collaboration. In 2026, that collaboration becomes autonomy.

Here are six trends we're anticipating that will reshape how engineering teams work this year.

1. Agents Will Ship With Built-in Accountability

The first generation of AI agents were black boxes. They'd take an instruction, disappear into a loop, and return somethinghopefully useful, often not. Engineers had no visibility into what the agent tried, why it failed, or whether its approach was even sensible.

That changes in 2026. The next wave of agents will come with testing frameworks, goal tracking, and structured logs built in. Think of it as observability for AI workflows. Every action logged. Every decision traceable. Every failure reviewable.

This isn't just nice-to-have tooling. It's the minimum bar for agents that operate in production environments where accountability matters. Teams won't trust agents they can't audit.

2. AI-Generated Code Will Be Structurally Better

Early AI code generation optimized for "does it work?" The result was functional but often messyinconsistent patterns, poor separation of concerns, and the kind of technical debt that compounds quietly.

The models shipping in 2026 are trained differently. They've internalized architectural patterns, not just syntax. They understand that a 500-line function is a code smell. They know when to extract a service, when to add an interface, and when to leave well enough alone.

The practical result: fewer bugs at the source. Not because AI doesn't make mistakes, but because well-structured code has fewer places for bugs to hide.

3. Complex, Multi-Step Tasks Will Actually Complete

Ask an AI agent to "refactor this module" or "migrate this service to the new API" and, until recently, you'd get partial results at best. The agent would lose context, get stuck, or quietly drift off-goal.

2026 brings agents that maintain coherence across longer task horizons. They break complex work into subtasks, checkpoint progress, and recover from failures without starting over. They can hold a goal in mind across dozens of operations and hundreds of files.

This is the difference between a tool that helps with tasks and one that completes them.

4. Autonomous AI Will Take Primary On-Call

This is the trend that will feel most uncomfortableand most inevitable.

AI agents are already triaging alerts, correlating signals, and suggesting root causes. The next step is giving them the authority to act. Not just "here's what might be wrong" but "I've identified the issue, applied the fix, and I'm monitoring for recurrence."

For well-understood failure modes with established runbooks, there's no reason a human needs to wake up at 3 AM. The agent can handle it, escalate if it's uncertain, and hand off a detailed incident report in the morning.

The human on-call role shifts from first responder to supervisorstill accountable, but not necessarily awake.

5. Day-to-Day Operations Will Run on Autopilot

Beyond incident response, there's a long tail of operational work that consumes engineering time: dependency updates, certificate rotations, capacity adjustments, config drift remediation, and the endless stream of small fixes that never quite make it to the sprint.

AI agents will absorb this work in 2026. Not as a batch job that runs once, but as a continuous process. The agent monitors, identifies issues, proposes fixes, andwith appropriate guardrailsapplies them.

Engineers review the changelog. They don't write it.

6. Always-On Agents Will Work in Shifts

The most significant shift is temporal. Today's AI interactions are synchronous: you prompt, it responds, you review. That loop keeps humans in the critical path.

The agents arriving in 2026 can work asynchronously for extended periodshours, not minutes. You define a goal, provide constraints, and the agent works toward it continuously. It checks in when it needs input, escalates when it hits uncertainty, and otherwise just keeps going.

Imagine starting your day with a summary: "Overnight, I completed the database migration, ran the regression suite, fixed two failing tests, and deployed to staging. Ready for your review."

That's not a vision. That's a product roadmap.

What This Means for Engineering Teams

These trends point in one direction: AI as a genuine team member, not just a tool.

The teams that thrive in 2026 will be those that figure out the right division of labor. What decisions require human judgment? What work can be fully delegated? How do you maintain accountability when an agent is acting autonomously?

The answers will vary by team, by codebase, and by risk tolerance. But the question is no longer whether AI will take on meaningful engineering work. It's how quickly your team will adapt to working alongside it.

Building reliable AI agents for production operations requires deep infrastructure context. At DrDroid , we're building the agentic context engine that makes autonomous incident response possible. Learn how teams are already putting AI on-call.

]]>https://notes.drdroid.io/ai-in-engineering-6-trends-that-will-define-2026https://notes.drdroid.io/ai-in-engineering-6-trends-that-will-define-2026<![CDATA[How to Build an AI Agent in Slack [DIY Guide]]]>Objective

By the end of this DIY guide, youll have:

1. A Slack Bot

2. A backend that can

• Listen to user messages or alerts in a channel and take agentic action based on prompts or steps that you might have in mind.

• Query your Grafana instance, analyse logs/dashboards and send info about anomaly in reply to an alert/message in your Slack channel

Pre-requisites

1. Python uv (package manager )

2. Ngrok to expose the slackbot server. Setup instructions

3. Grafana (Optional)

Step 0: Clone the repo:

Repository Link - https://github.com/DrDroidLab/slack-ai-bot-builder

Step 1: Building the Slack Bot with an integrated backend

The backend repo setup here, behaves in multiple ways:

• Acts as an MCP Client for any AI calls you might want to make

• Acts as a server to accept webhooks from Slack and manage configurations

Setting up the Ngrok Tunnel

Expose port 5000 of your localhost, using the command:

ngrok http 5000

You will receive a HTTPS URL of the form https://abc123.ngrok.io

Which is pointing to port 5000 of your system.

Note: We have not setup a server running on port 5000 yet, but that is fine since ngrok is independent of that, and exposes the port regardless.

Creating the Slack Application

1. Go to Slack API Apps https://api.slack.com/apps

2. Click on Create App, and select the option From a manifest

3. Copy the manifest json from the repository and replace the placeholder with the HTTPS URL you got from ngrok. Include the https:// part as well.

4. Install the app in your workspace.

5. Copy the credentials for the slack application into credentials.yaml

6. app_id, app_name and signing secret can be found in the basic information tab.

7. Bot-auth-token can be found in the Oauth & Permissions tab.

8. openai_key from openai - if you plan to use AI based workflows.

9. Create a channel called #drdroid-slack-bot-tester in your slack workspace & add the bot to the channel.

Setting up the bot server

Run the following commands to set up your virtual environment and activate it.

uv env venv
source .venv/bin/activate

Install dependencies using:

uv sync

Now we can finally run the bot server using:

uv run python app.py

The server is now running on port 5000, and exposed to the outside world via your ngrok tunnel.

Testing the bot

Add the bot to the drdroid-slack-bot-tester workspace that you had previously created.

And just type in a hi. The bot should send you a sample response.

Step 2: Inte grating AI

There is already an example workflow for AI (name: "chatbot") in the workflows.yaml. It is just boilerplate code, you can modify it as required. For now, you can tag your bot in the drdroid-slack-bot-tester, and chat with it.

• Add your OpenAI/LLM key

For example:

Message in Slack: chatbot How to debug Kubernetes CrashLoopBackOff error? @bot
Message in Slack: chatbot I'm getting this alert. What does it mean? @bot

Step 3: Making the bot an Agent by giving AI access to different tools (Grafana for demo)

MCP Servers help abstract out any API and making them accessible to AI.

Setting up Grafana MCP Server

Clone the following repository:

Repository URL - https://github.com/DrDroidLab/grafana-mcp-server

Navigate into the root directory of the repository.

Populate the src/grafana_mcp_server/config.yaml with your grafana credentials.

Install and setup the dependencies using:

uv venv .venv
source .venv/bin/activate
uv sync

Run the MCP server:

uv run -m src.grafana_mcp_server.mcp_server

Your MCP server is now running on port 8000.

Creating an AI Grafana workflow in slack-bot-builder

There is already an example workflow for grafana ai in the workflows.yaml.

We will be running the script scripts/grafana_ai_tool.py in this workflow, it is just boilerplate code, you can modify it as required.

Now you can tag your bot in the drdroid-slack-bot-tester, and ask it to do various things from grafana.

For example:

Message in Slack: Fetch me logs from the currencyservice in grafana ai.
Message in Slack: Fetch and analyse the Go Microservices dashboard from grafana ai

Next Steps:

Now that youve been able to setup a bot, here are a few things you can do:

• Productionise it from your current ngrok setup to a static endpoint

• Integrate with Grafana or open source MCP servers of your favourite tool that you want to leverage for automation

• Add custom prompts and scripts

Stuck anywhere? Ask on our Discord

]]>https://notes.drdroid.io/how-to-build-an-ai-agent-in-slack-diy-guidehttps://notes.drdroid.io/how-to-build-an-ai-agent-in-slack-diy-guide<![CDATA[GitOps for Alerting: How to Manage Alert Rules Like Code]]>It's 2 AM. Production is on fire. You need to adjust an alert threshold that's been firing false positives all week.

You log into Grafana, click through three nested menus, find the alert, and bump the threshold from 80% to 85%. Crisis averted. You go back to bed.

Two weeks later, during a postmortem, someone asks: "Who changed the CPU alert threshold? And why?"

Silence. Nobody remembers. There's no history. No context. No way to know if this was a temporary hack or a deliberate tuning decision. Worse, when you refresh your staging environment, the old threshold returns because the change only lived in the production UI.

Sound familiar? You're not alone. This is how most teams manage alertsand it's fundamentally broken.

Why Managing Alerts in Dashboards Doesn't Scale

We've spent the last decade moving infrastructure to code. Terraform for cloud resources. Helm charts for Kubernetes. Ansible for configuration. Yet somehow, our alert rulescritical infrastructure that wakes up engineersstill live in UI dashboards like it's 2010.

The problems compound quickly:

No version history: When did this alert last change? Who changed it? Why? Your Grafana dashboard shrugs.

No peer review: A junior engineer can accidentally change a critical alert threshold with zero oversight. Try doing that with production code.

No rollback capability: That "quick fix" that made things worse? Good luck remembering the old values.

Environment drift: Production alerts diverge from staging. Dev environments have different rules. Chaos ensues.

No ownership tracking: Who owns this alert? Which team should review changes? The UI doesn't care.

Your infrastructure evolves constantly. Services scale. Traffic patterns shift. Performance characteristics change. But alerts configured through dashboards remain frozen in time, slowly becoming less relevant until they're just noise.

Here's the thing: alert rules are infrastructure-as-code too. They define critical system behavior. They impact your team's quality of life. They deserve the same rigor as any other code.

Enter GitOps for alertswhere alert definitions live in version control, changes happen through pull requests, and every modification is tracked, reviewed, and reversible.

What is GitOps for Alerting?

GitOps for alerting is beautifully simple: store your alert rules as code in Git, manage changes through pull requests, and deploy automatically. Just like any other infrastructure.

Most modern monitoring tools already support this:

• Prometheus: Alert rules in YAML files

• Alertmanager: Routing configuration as code

• Grafana: Alerts exportable as JSON

• Datadog: Monitors manageable via Terraform

• New Relic: Alerts configurable through their API/Terraform

Here's what a typical structure looks like:

/alerts/
  frontend-service.yaml
  database.yaml
  redis.yaml

/teams/
  payments/
    api-alerts.yaml
    database-alerts.yaml
  platform/
    infrastructure-alerts.yaml
    kubernetes-alerts.yaml

A Prometheus alert rule might look like:

groups:
  - name: frontend-service
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
          service: frontend
          team: frontend-team
        annotations:
          summary: "High error rate on {{ $labels.instance }}"
          description: "Error rate is {{ $value }} (threshold 0.05)"
          runbook: "https://wiki.company.com/runbooks/frontend-errors"
          owner: "frontend-oncall@company.com"
The benefits are immediate:

Traceability: Every change is a commit. Git blame tells you who changed what and when.

Peer review: Alert changes go through PR reviews. No more accidental 3 AM threshold adjustments.

Consistency: Deploy the same alerts across all environments. No more production/staging drift.

Rollback capability: Bad change? git revert and you're back to working alerts.

Documentation: PR descriptions explain why changes were made. Context is preserved forever.

But GitOps Alone Isn't Enough

Here's the plot twist: GitOps for alerts solves the how but not the what.

You now have beautiful, version-controlled alert rules. Every change is reviewed and tracked. But you still don't know which rules need updating.

Your Git repo becomes a graveyard of alert rules that might or might not be relevant:

• That CPU alert from 2019 when you ran on smaller instances

• The memory warning tuned for your old Java app (you've since moved to Go)

• The latency threshold set when you had 100 users (you now have 10,000)

You've traded one problem for another. Instead of stale alerts in dashboards, you have stale alerts in Git. They're better organized, sure, but still noisy.

This is where most GitOps alerting stories end. Teams implement the framework but lack the feedback loop to keep it healthy. Alert rules accumulate like sediment. Engineers suffer in silence because "at least it's in Git now."

Using Alert Insights to Drive GitOps Changes

Let real alert data guide your pull requests

The missing piece is data. You need to know which alerts are actually problematic before you can fix them. This is where DrDroid's Alert Insights transforms GitOps from a theoretical improvement into a practical solution.

Alert Insights analyzes your live production alerts and tells you:

• Which alerts fired most frequently last week: Your noisiest offenders, ranked

• Which alerts were ignored: Clear signal of rules that need removal

• Which alerts lack owners or runbooks: Quality issues to address

• Suggested changes: Specific recommendations to mute, tweak, or archive

Now GitOps becomes powerful. You're not guessing which alert rules to updateyou have data.

Workflow Example:

Monday: Run Alert Insights

`Top 3 Noisy Alerts:

1. redis_memory_warning - 127 fires, 0 actions taken

2. api_latency_high - 89 fires, acknowledged but not investigated

3. cpu_usage_critical - 45 fires, all during deploy windows`

Tuesday: Create targeted PRs

git checkout -b fix/reduce-redis-memory-noise
# Edit alerts/redis.yaml
# Increase threshold from 70% to 80% based on actual usage patterns
git commit -m "Increase Redis memory threshold to reduce false positives

Alert Insights showed 127 fires with 0 actions last week. Analysis shows Redis memory naturally spikes to 75% during cache warmup."

Wednesday: Review and merge

• Team reviews the PR

• Links to Alert Insights data provide context

• Changes deploy automatically

Thursday: Validate impact

• Alert noise drops immediately

• Next week's Alert Insights confirms improvement

The feedback loop is complete. You're not just organizing alerts betteryou're systematically improving them based on real data.

🛠 Want a GitOps-ready alert audit? 👉 Run DrDroid's Alert Insights and get actionable suggestions in minutes.

Recommended Practices for GitOps Alert Management

🔍 Use clear filenames per service/component

Don't create a monolithic alerts.yaml. Break rules into logical groups:

/alerts/
  services/
    payment-api.yaml
    user-service.yaml
  infrastructure/
    kubernetes-nodes.yaml
    database-cluster.yaml
  business/
    checkout-flow.yaml
    user-engagement.yaml

🔄 Add labels/tags to help Alert Insights map alerts to owners

Every alert should include:

yaml

labels: team: payments service: payment-api environment: production severity: P2

This metadata powers Alert Insights' analysis and recommendations.

🧪 Validate rules with test alerts in staging

Before merging, trigger test conditions:

# Simulate high error rate
curl -X POST http://prometheus:9090/api/v1/series \
-d 'match[]=up{job="frontend"}'
🔁 Link PRs to weekly alert review

Common GitOps Pitfalls to Avoid

Bulk silencing alerts without context

"Let's just comment out all the noisy alerts" is tempting but dangerous. Use Alert Insights to understand why alerts are noisy before acting.

Committing rules without reviews

The whole point of GitOps is peer review. Don't bypass it with direct commits, even for "quick fixes."

No tagging = Alert Insights can't map alerts to services

Without proper labels, you lose the ability to analyze alerts by team, service, or severity. Enforce tagging standards.

Alert rules diverging across environments

Use templating to keep staging and production alerts synchronized:

# values-prod.yaml
cpu_threshold: 80
memory_threshold: 85

# values-staging.yaml
cpu_threshold: 90 # Higher tolerance in staging
memory_threshold: 90

Final Take Let Data Drive Your Alert Rule Changes

GitOps gives you the framework for managing alerts professionally. Version control, peer review, and rollback capabilities bring alerts into the modern era.

But framework without data is just organized chaos. You need to know which alerts to fix, how to fix them, and whether your fixes worked.

Alert Insights provides that missing data layer. It tells you which alert rules are hurting your team, suggests specific improvements, and validates that your changes actually reduced noise.

Together, they create a powerful feedback loop:

1. Alert Insights identifies problematic alerts

2. GitOps enables reviewed, tracked changes

3. Automated deployment ensures consistency

4. Next week's Alert Insights validates improvement

This isn't theoretical. Teams using this approach report 50-70% reduction in alert noise within weeks. On-call engineers sleep better. Real incidents get proper attention. Alert quality becomes a measurable, improvable metric.

Your alerts deserve the same engineering rigor as your code. GitOps provides the foundation. Alert Insights provides the intelligence. Together, they transform alerting from a necessary evil into a competitive advantage.

Want to make smarter, reviewable changes to your alerts? 👉 Run AIOps and let your alerts tell you what to fix.

]]>https://notes.drdroid.io/gitops-for-alerting-how-to-manage-alert-rules-like-codehttps://notes.drdroid.io/gitops-for-alerting-how-to-manage-alert-rules-like-code<![CDATA[3 Tools That Help Reduce Alert Fatigue (With Trade-offs)]]>We live in the age of "vibecoding."

Your engineers ship features at lightning speed. AI copilots autocomplete entire functions. CI/CD pipelines deploy to production in minutes. Modern development has become a symphony of efficiency, with developers operating at 10x the speed of just five years ago.

But there's one part of your stack that's stuck in 2010: your alerts.

While your team vibecodes their way through complex distributed systems, your alerting engine still screams about every CPU spike, memory blip, and network hiccup like it's the apocalypse. It's like having a Ferrari engine attached to horse-and-buggy wheels. The cognitive dissonance is jarringand it's killing your team's productivity.

Why Alert Fatigue Is a Real Problem in 2025

Here's the absurd reality: The same engineer who just deployed a sophisticated ML model in production gets woken up at 3 AM because a health check endpoint took 501ms instead of 500ms to respond. The developer who elegantly orchestrated a microservices migration gets paged because a pod restartedsomething Kubernetes is literally designed to do automatically.

Modern infrastructure has exploded in complexity. You're running hundreds of microservices, each generating alerts. Kubernetes adds its own layer of notifications. Cloud providers, APMs, and security tools all want their voice heard. The result? An endless stream of "urgent" notifications flooding Slack channels and PagerDuty rotations.

But unlike your codebasewhich has intelligent linters, smart IDEs, and AI-powered suggestionsyour alerts remain dumb. They can't distinguish between:

• A temporary spike during garbage collection vs. a memory leak

• A planned scaling event vs. an unexpected traffic surge

• A self-healing Kubernetes pod restart vs. a critical service failure

The real problem: You don't know which alerts matter anymore.

Your engineers have adapted the only way they canby tuning out. When every alert claims to be critical but most are noise, even genuine emergencies get ignored. It's the monitoring equivalent of crying wolf, except the wolf is paging your on-call engineer every 30 minutes.

What you need aren't more dashboards visualizing the chaos. You need intelligent tools that understand context, learn patterns, and show you what's noisy and help you take action. Let's examine three approaches to bringing your alerts into the modern era.

Tool #1 DrDroid

Best for: Real-time visibility into noisy alerts, across any stack

DrDroid represents the first generation of truly intelligent alerting tools. While your engineers use AI to write code faster, DrDroid uses intelligence to make your alerts smarter.

The platform integrates with your existing stackSlack, Prometheus, New Relic, OpenTelemetry, and more. But what sets it apart is the Alert Insights feature, which applies actual intelligence to your alert patterns:

• Which alerts are flapping? Just like a smart IDE highlights code smells, DrDroid identifies alerts that repeatedly fire and resolveclear indicators of misconfiguration.

• Which alerts are being ignored? By analyzing engineer behavior, it spots alerts that get dismissed without action. If developers ignore an alert 100% of the time, why is it still paging them?

• Which alerts lack runbooks or clear owners? Nothing frustrates a vibecoding engineer more than context-switching to an alert with zero information about what to do.

DrDroid doesn't just identify problemsit suggests fixes:

• Automatically mute alerts during deployment windows

• Disable alerts that have never correlated with customer impact

• Add intelligent conditions (like requiring sustained threshold breaches)

• Enrich alerts with missing context, runbooks, and correlation data

The platform's auto-debugging capabilities are particularly impressive. When an alert fires, DrDroid automatically pulls relevant logs, metrics, traces, and even recent code changes. It's like having an AI copilot for incident response.

Consider this scenario: Your payment service alerts on high latency every day at 2 PM. DrDroid notices the pattern, correlates it with a scheduled batch job, and suggests either suppressing the alert during that window or adjusting the threshold. What took hours of manual analysis now happens automatically.

Trade-offs

DrDroid is built for modern, Slack-first teams. If your organization has traditional processes requiring all alerts to flow through legacy ITSM tools, adoption might face resistance. As a newer platform, some enterprise compliance features are still maturing.

🧠 Want to know which alerts your team should disable? 👉 Explore DrDroid's Alert Insights loved by SREs to reduce alert fatigue.

Tool #2 BigPanda

Best for: Enterprise-scale alert correlation

BigPanda takes a different approachusing machine learning to group related alerts into incidents. When a database issue triggers alerts across 20 services, BigPanda recognizes the pattern and presents them as one incident.

For large enterprises with complex systems, this correlation can help. The platform learns relationships between components and can reduce the number of incidents operators review. It also integrates deeply with enterprise tools like ServiceNow and Dynatrace.

Trade-offs

Here's where the contrast with modern development becomes stark. While your engineers deploy code in minutes, BigPanda requires months of setup. While developers use intuitive tools that work out-of-the-box, BigPanda demands extensive metadata configuration and alert standardization.

More critically, BigPanda doesn't make individual alerts smarterit just groups dumb alerts better. Those flapping alerts your engineers hate? Still firing, just bundled together. It's like organizing spam into folders instead of fixing your spam filter.

The platform is also expensive, often requiring dedicated administrators and cross-team coordination. For teams used to the speed of modern development, BigPanda's implementation timeline feels like stepping back in time.

Tool #3 PagerDuty Analytics

Best for: Trend visibility inside the PagerDuty ecosystem

PagerDuty Analytics provides retrospective dashboards showing alert volume, MTTR, and on-call load. For teams already using PagerDuty, it offers visibility into historical patterns and trends.

The analytics can be useful for quarterly reviews and capacity planning. You can see which services generate the most alerts and track improvements over time.

Trade-offs

The limitations mirror the gap between modern development and legacy monitoring. While your engineers get real-time feedback from their tools, PagerDuty Analytics is retrospective only. It tells you that Service X generated 500 alerts last month but not which ones were false positives or what to do about them.

It only analyzes alerts flowing through PagerDuty, missing Slack notifications and other channels. The insights are descriptive, not prescriptiveyou see the problem visualized but get no help fixing it. And it requires expensive premium tiers, adding cost without adding intelligence.

Comparison Table

Final Thoughts Your Alerts Should Be as Smart as Your Code

We've entered an era where engineers can literally describe what they want to build and watch AI generate the code. They deploy with confidence, iterate rapidly, and ship features that would have taken months in mere days.

Yet these same engineersthese 10x vibecoding machinesare still being interrupted by alerts that would have been considered noisy a decade ago.

The disconnect is unsustainable. You can't run a modern engineering organization with stone-age alerting. Your monitoring needs to evolve to match the sophistication of your development practices.

BigPanda and PagerDuty show you the problem in high resolution. Only DrDroid's Alert Insights actually makes your alerts smarter identifying what's broken, why it's noisy, and exactly how to fix it.

The future of monitoring isn't better dashboards or fancier grouping algorithms. It's intelligent systems that understand context, learn from patterns, and proactively help you maintain signal-to-noise ratio. It's alerts that are as smart as the engineers they're interrupting.

Your team deserves alerting infrastructure that matches their development velocity. Stop letting 2010-era alerts slow down your 2025 engineering team.

💡 Ready to reduce alert fatigue the smart way? 👉 Start using Alert Insights to find and fix noisy alerts today no config needed.

]]>https://notes.drdroid.io/3-tools-that-help-reduce-alert-fatigue-with-trade-offshttps://notes.drdroid.io/3-tools-that-help-reduce-alert-fatigue-with-trade-offs<![CDATA[A Practical Framework to Reduce Alert Noise (Without Missing Incidents)]]>Every SRE has been there.

Fed up with alert fatigue, you go on a muting spree. That flaky health check? Silenced. The CPU warning that fires during deploys? Disabled. The memory alert that triggers during garbage collection? Gone.

For a blissful week, your on-call rotation is peaceful. Engineers are sleeping through the night. Slack channels are quiet. Life is good.

Then it happens. A real incident slips through. Customer complaints pour in. Your CEO wants answers. And suddenly, those "noisy" alerts you disabled don't seem so unnecessary anymore.

Here's the uncomfortable truth: anyone can reduce alert noise by turning off alerts. The real challengethe one that separates good SRE teams from great onesis reducing noise without sacrificing coverage.

Why Reducing Alert Noise Is Harder Than It Sounds

The naive approach to alert fatigue is seductively simple: just turn off the annoying alerts. But this creates a dangerous blind spot. That CPU alert might be noisy 99% of the time, but what about the 1% when it signals a real problem?

The opposite extreme isn't better. Some teams, burned by missed incidents, keep every alert active "just in case." They end up with hundreds of alerts that cry wolf, training engineers to ignore everythingincluding real emergencies.

The solution isn't choosing between noise and coverage. It's building a systematic approach that maintains visibility while eliminating false positives. High-performing SRE teams follow a 4-phase framework that transforms chaotic alerting into intelligent monitoring.

This framework isn't theoreticalit's battle-tested by teams managing hundreds of services in production. And with modern tools like Alert Insights, you can measure and validate your improvements with data, not guesswork.

Let's dive into each phase.

Phase 1 Start with Coverage, Not Silence

The mistake most teams make: start muting

When alert fatigue hits, the instinctive response is to start silencing alerts. It feels productiveeach muted alert is one less interruption. But this approach is backwards.

Before you disable a single alert, you need to understand what you're actually trying to monitor. This means mapping your core Service Level Objectives (SLOs) and Service Level Indicators (SLIs) to your alerting strategy.

Most teams rely too heavily on infrastructure alertsCPU usage, memory consumption, disk space. These are important, but they're indirect signals. A service can have high CPU usage while serving customers perfectly. Conversely, it can have normal resource usage while completely failing its primary function.

Instead, start with user-facing signals:

• Failed user logins (not just authentication service uptime)

• Checkout completion rates (not just payment gateway availability)

• API response times at the 95th percentile (not just average latency)

• Database query failures (not just connection pool metrics)

Map these business-critical indicators first. Only after you have comprehensive coverage of what matters should you start tuning what doesn't.

Principle: Only tune alerts after coverage is solid. It's better to have noisy but comprehensive alerting than quiet but blind monitoring.

Phase 2 Assign Ownership

Every alert should have an owner, a service, and a runbook

Here's a dirty secret of most alerting systems: nobody owns the alerts. They fire into shared channels where responsibility diffuses across the team. When everyone is responsible, no one is accountable.

This shared ownership model is why alerts never improve. The payment team ignores database alerts because "that's infrastructure's problem." The infrastructure team ignores API latency alerts because "that's the app team's issue." Meanwhile, both alerts keep firing, and your on-call engineer suffers.

The fix is radical but simple: every alert must have a single owner. Not a team, not a rotationa specific service and the team that owns it. This means:

• No more #alerts-general channels where everything dumps

• No more "infrastructure noise" channels that everyone mutes

• Each team gets their own alert destinations

• Each team is accountable for their signal-to-noise ratio

Implement this with proper tagging:

alert: HighAPILatency service: payment-api team: payments owner: payments-team@company.com escalation: payments-oncall severity: P2

When alerts have clear ownership, magic happens. The payments team suddenly cares about that flapping API alert because it's waking them up, not some random SRE. They'll fix it, tune it, or justify why it needs to stay.

Tip: Alerts without owners almost never get fixed. They become background noise that everyone learns to ignore.

Phase 3 Enrich, Then Tune

Rich alerts = less cognitive load = faster response

Now that you have coverage and ownership, it's time to make your alerts actually useful. A bare-bones "Service X is down" notification forces engineers to context-switch, investigate, and piece together what's happening. Rich alerts provide everything upfront.

Essential enrichment includes:

• Runbook links: Step-by-step remediation instructions

• Severity levels: Is this customer-impacting or internal-only?

• Business impact: How many users affected? Which features degraded?

• Recent changes: Did a deployment just go out?

• Historical context: Has this happened before? How was it fixed?

But richness isn't verbosity. Don't dump entire log files into alerts. Instead, provide precisely what's needed for rapid decision-making.

Only after enrichment should you start tuning:

Add intelligent conditions: Instead of alerting on every spike, require sustained problems:

• Alert only after 3 consecutive failures

• Require issues to persist for 5 minutes

• Use percentage-based thresholds (5% of requests failing vs. 10 absolute failures)

Adjust thresholds based on reality: That 80% CPU alert made sense with your old infrastructure. But if your auto-scaling kicks in at 70%, you're alerting on normal operations.

Add flapping protection: If an alert fires and resolves repeatedly, it needs damping:

• Require state changes to persist before alerting

• Group rapid-fire alerts into single notifications

• Add cooldown periods between alerts

Insight: Context beats volume every time. One well-enriched alert is worth ten noisy notifications.

Phase 4 Use Data to Improve Over Time

Enter: Alert Insights by DrDroid

Here's where most frameworks fail: they're static. Teams implement phases 1-3, declare victory, and move on. Six months later, they're back to alert fatigue because systems evolve but alerts don't.

You need a continuous feedback loopa way to measure what's working and what's still broken. This is where Alert Insights becomes your secret weapon.

After implementing your alert structure, Alert Insights provides ongoing intelligence:

• Which alerts are firing too often? That P1 alert that fires 50 times per week probably needs adjustment

• Which ones are being ignored? If engineers acknowledge but never act on an alert, it's pure noise

• Which lack runbooks or clear owners? Gaps in your enrichment strategy become visible

• What can be safely muted, disabled, or improved? Data-driven recommendations, not guesswork

The workflow becomes systematic:

Every sprint:

1. Review Alert Insights dashboard

2. Identify the top 3 worst offenders

3. Fix ownership, enrichment, or tuning for those alerts

4. Validate improvements in the next sprint

5. Repeat

This creates a virtuous cycle. Your alerts get better every sprint. Your on-call experience improves measurably. And you maintain coverage while reducing noise.

🧠 Want a clear report on which alerts are hurting your team? 👉 Run DrDroid Alert Insights no config required.

Bringing It All Together Your Team's Framework

Here's your systematic approach to intelligent alerting:

This isn't a one-time projectit's an ongoing practice. Just like you continuously refactor code, you need to continuously refine alerts. The difference is that now you have a framework and the data to guide your decisions.

Final Thought You Can't Fix What You Don't See

Most teams exist in one of two failure modes. They either suffer in silence with alert fatigue, accepting it as the cost of observability. Or they oversimplify their alerting, creating dangerous blind spots that only become visible during incidents.

Real success looks different: high signal, low noise, and fast resolution. It's alerts that wake you up only when customer impact is imminent. It's notifications that include everything needed to respond. It's a system that improves continuously based on data, not opinions.

This framework gives you the path. Phase by phase, you can transform your alerting from a source of frustration into a competitive advantage. But frameworks only work when you can measure their impact.

Let Alert Insights be your guide. It shows what's working, what's broken, and exactly how to improve. No more guessing which alerts to tune. No more hoping you haven't created blind spots. Just data-driven improvements that make your team's life better.

Your engineers deserve better than alert fatigue. Your customers deserve better than missed incidents. This framework delivers both.

🛠 Tired of guessing which alerts are noisy? 👉 Try Alert Insights and start tuning your alerts based on real data.

]]>https://notes.drdroid.io/a-practical-framework-to-reduce-alert-noise-without-missing-incidentshttps://notes.drdroid.io/a-practical-framework-to-reduce-alert-noise-without-missing-incidents<![CDATA[KubeCon + CloudNativeCon Europe 2025 Guide – London]]>Doctor Droids Guide to KubeCon + CloudNativeCon Europe 2025

Welcome to our complete guide for navigating KubeCon + CloudNativeCon Europe 2025 in London, England, running from 14 April 2025. Whether youre a seasoned cloud native pro or new to the Kubernetes world, this guide has everything you need to maximize your conference experience. And dont forget visit the Doctor Droid booth at the event! Show us this blog to score an exclusive 20% discount on your ticket plus a chance to receive special Doctor Droid credits!

Overview

KubeCon + CloudNativeCon is the premier conference for Kubernetes, cloud-native technologies, and open source innovations. Organized by the Cloud Native Computing Foundation (CNCF), this flagship event gathers thousands of developers, engineers, and industry leaders to share ideas, network, and explore the latest trends shaping the future of cloud computing. In the heart of London, expect inspiring keynotes, deep-dive sessions, and engaging co-located events that cover everything from AI/ML integration to edge computing and beyond.

Access Types

KubeCon + CloudNativeCon Europe 2025 offers a single, all-inclusive pass that grants access to:

• All keynote sessions, breakout tracks, and panel discussions

• Hands-on labs and workshops for a practical dive into cloud-native solutions

• Co-located events hosted by CNCF and industry partners

Additionally, there are special discounted options available for students and academic participants. For more details, check out the official KubeCon + CloudNativeCon Europe website for registration options.

Exclusive KubeCon + CloudNativeCon Europe 2025 Discount Save 20% on Tickets, Courtesy of Doctor Droid!

Thats right Doctor Droid is proud to sponsor KubeCon + CloudNativeCon Europe 2025, and were offering you an exclusive 20% discount on your ticket! Heres how to claim your savings:

1. Fill out our quick Google Form with your basic information.

2. Receive your discount code directly in your inbox.

3. Register on the official event website and enjoy your 20% savings!

Hurry up secure your discount today and get ready for an unforgettable experience in London!

Speakers & Tracks at KubeCon + CloudNativeCon Europe 2025

The conference features an impressive line-up of industry thought leaders and technical experts across multiple tracks, including:

• Kubernetes Operations: Best practices in deployment, scaling, and security.

• Cloud Security: Deep dives into safeguarding cloud native environments.

• AI & Machine Learning: Innovations transforming how we manage and operate Kubernetes.

• Edge Computing: Exploring the future of distributed computing in real-world scenarios.

Be sure to check out the detailed schedule on the official event page for a complete list of sessions and speakers.

Planning Your London Experience

Where to Stay

London offers a range of accommodation options to suit every budget:

• Hotels: Browse options on Booking.com, Agoda, or Airbnb for a comfortable stay near the event venue.

• Short-term Rentals: Consider serviced apartments if you prefer a homier experience during your stay.

Getting There

London is well connected by international air travel:

• Heathrow Airport (LHR) The largest and busiest airport, with easy public transit into central London.

• Gatwick Airport (LGW) A convenient alternative for many international travellers.

Plan your journey ahead to make the most of your time in this vibrant city!

Visiting London

Culinary Delights

Londons food scene is as diverse as it is delicious. Whether youre looking for Michelin-starred restaurants or quirky food markets, here are some recommendations:

• Restaurants: Try iconic spots in Soho or trendy eateries in Shoreditch.

• After-Hours: Explore vibrant nightlife in Camden or the West End for live music and cocktails.

• Coffee Shops: Recharge at local favorites like Monmouth Coffee or The Attendant for a caffeine boost.

Weekend Plans

When youre not immersed in conference sessions, take some time to explore Londons rich history and culture:

• The British Museum: Discover art and antiquities from around the world.

• Tower of London: Step back in time with a visit to this historic fortress.

• Buckingham Palace & Changing of the Guard: A must-see for first-time visitors.

• London Eye: Enjoy panoramic views of the city skyline.

Visit the Doctor Droid Booth

Doctor Droid is the intelligent Slack bot that accelerates incident diagnosis by automatically pinpointing the root cause of production issues. Simply tag the bot in your alert messages, and let it do the heavy lifting!

Stop by our booth at KubeCon + CloudNativeCon Europe 2025 to discover:

• Live Demos: See Doctor Droid in action and learn how it can transform your incident response.

• Puzzles & Giveaways: Test your skills and win exciting Doctor Droid goodies.

• $500 Doctor Droid Credits: Show this blog at our booth and receive $500 in credits to supercharge your troubleshooting capabilities.

For those interested in a one-on-one demo, pre-book a meeting with us here.

Get ready to experience the future of cloud native computing in one of the worlds most exciting cities London awaits at KubeCon + CloudNativeCon Europe 2025!

Happy conferencing, and see you in London!

]]>https://notes.drdroid.io/kubecon-cloudnativecon-europe-2025-guide-londonhttps://notes.drdroid.io/kubecon-cloudnativecon-europe-2025-guide-london<![CDATA[Tools can't buy you good MTTR.. but these 3 practices can]]>Context

Its a scenario weve all witnessed: teams equipped with cutting-edge observability tools still struggling to catch issues before customers notice.

Theyve invested heavily in top-tier APM solutions, container and infrastructure monitoring, and log accessibility. Yet, their on-call engineers remain overwhelmed. Incidents happen more frequently than anyone would like, and the spotlight they find themselves in post-incident is never the kind they want.

For engineering teams, being called out for production issues is a tough pill to swallow. The key lies in post-incident action plans that lead to meaningful, systematic improvements.

While production incidents cant be entirely eliminated, well-thought-out preventive measures can dramatically improve operational health.

Tools Are the BaselineNot the Answer

While tools are essential, they primarily address infrastructure or service-level issues. However, most real-world incidents cascade across multiple stacks, often affecting features, products, or customer experiencesareas that are rarely solved by out-of-the-box tools.

To reduce MTTR, teams need processes that improve detection, diagnosis, and resolution speed.

Measures for reducing MTTR drastically:

Here are three processes that I have seen help teams in improving MTTR significantly:

1. Improving Actionability of Alerts (Faster Detection)

• Trustworthy alerts are a cornerstone of effective incident management. Engineers need a single source of truth to detect issues earlybefore customers or business stakeholders notice.

• Poorly configured alerts can destroy this trust, leading teams to rely on escalations from support or business teams instead. Monitoring alert quality is critical. For example, many companies using Doctor Droid track alert quality to ensure non-actionable alerts dont erode confidence in their systems.

2. Instrumenting Custom Metrics

• Custom metrics are invaluable for tracking operational health and catching issues tied to features and product breakages. Unlike generic service-level metrics, custom metrics provide leading indicators that can help teams spot potential failures before they escalate.

• By focusing on metrics relevant to their features and customer experience, teams can gain clarity and react faster.

3. Faster Fixing Through Runbooks and Quick Links

• Developer experience during on-call is often overlooked. Simple resources like runbooks or quick links for known issues can dramatically reduce the cognitive load on engineers.

• For example, a link to a pre-built log query can save critical minutes during an incident. These tools empower teams to pinpoint issues faster, enabling quicker resolutions.

Conclusion:

No matter how much you spend on tools, improving MTTR requires engineering investment in processes that enhance detection, diagnosis, and resolution. Custom metrics, actionable alerts, and developer-friendly resources are what truly make the difference.

Engineering teams that focus on these practices find themselves more prepared, more resilient, and better positioned to handle the inevitable challenges of production.

Want to monitor your alerting quality and improve MTTR? Doctor Droid has helped 40+ companies take their incident management to the next level. Get started for free and improve your alerts today!

]]>https://notes.drdroid.io/tools-cant-buy-you-good-mttr-but-these-3-practices-canhttps://notes.drdroid.io/tools-cant-buy-you-good-mttr-but-these-3-practices-can<![CDATA[KubeCon + CloudNativeCon India 2024 Guide -- Delhi]]>Doctor Droids Guide to KubeCon + CloudNativeCon India 2024

Welcome to our complete guide for navigating KubeCon + CloudNativeCon India 2024! Heres all you need to know to make the most of this event in Delhi, India, from December 11-12, 2024. Dont forget to visit Doctor Droid at Boothshow us this blog for a chance to receive $500 worth of Doctor Droid credits!

Overview

KubeCon + CloudNativeCon is the leading conference for Kubernetes, cloud-native technologies, and open-source solutions. Hosted by the Cloud Native Computing Foundation (CNCF), this event gathers thousands of developers, engineers, and business leaders to exchange knowledge, network, and discover the future of cloud-native ecosystems.

Access Types

KubeCon + CloudNativeCon India 2024 offers one access pass to cater to all types of attendees:

Includes access to all sessions, keynotes, and co-located events

• Academic Pass: Discounted pass for students looking to dive into cloud-native technologies.

• Individual pass: Perfect for attendees who are paying for the conference by themselves.

Be sure to review the registration options on the official KubeCon + CloudNativeCon India 2024 website.

Exclusive KubeCon + CloudNativeCon India 2024 Discount Save 20% on Tickets, Courtesy of Doctor Droid!

Thats rightDoctor Droid is a proud sponsor of KubeCon + CloudNativeCon India 2024, and were hooking you up with an exclusive 20% discount on your tickets! Heres how to secure your spot and save big:

1. Fill out this google form with your basic info.

2. Get your 20% discount code delivered straight to your inbox.

3. Register for KubeCon + CloudNativeCon India 2024 and enjoy the savings!

Dont sit on thisgrab your discount before its gone!

Speakers & Tracks at KubeCon + CloudNativeCon India 2024

KubeCon + CloudNativeCon features keynotes from industry leaders and breakout sessions across multiple tracks, including:

• Kubernetes Operations: Topics around the deployment, scaling, and security of Kubernetes.

• AI and Machine Learning: How AI and ML are transforming Kubernetes.

• Edge Computing: Use cases and solutions for Kubernetes at the edge.

• Cloud Security: Tools and practices to ensure security in a cloud-native environment.

Planning for KubeCon + CloudNativeCon India 2024 Delhi Logistics

Stays near Delhi

Delhi has several convenient options for accommodation. Whether you prefer hotels or Airbnbs, here are a few recommendations:

• Hotels: You can find a number hotels on booking.com, agoda.com or makemytrip.com

• Airbnb: Airbnb has a healthy number of properties available in the city

Airports nearby

• Indira Gandhi International Airport (DEL) is the nearest airport from the convention center

Visiting Delhi

Restaurants

Delhi offers a diverse culinary scene. Recommended spots include:

After Hour Locations

Coffee Shops

Need a caffeine boost? Check out:

• Blue Tokai Coffee: Known for freshly roasted coffee.

• Third Wave Coffee: Multiple locations to elevate your coffee experience

Weekend Plans

Consider exploring Delhis culture and history over the weekend. Popular options include:

• Red Fort: A red coloured fort in the old Delhi to experience the culture and food of Delhi

• Akshardham Temple: A stunning modern temple complex showcasing Indias rich cultural heritage through intricate architecture, gardens, and a beautiful water show. Its a must-visit for its grandeur and serenity.

There are many more places to explore and you can checkout while in delhi

1. India Gate: A monumental war memorial built in honor of Indian soldiers, surrounded by beautiful lawns, making it a great spot for a peaceful evening stroll.

2. Lotus Temple: Known for its stunning flower-like shape, this temple is a peaceful place for meditation.

3. National Museum: A treasure trove of Indias art, history, and culture spanning millennia.

4. Dilli Haat: A market with traditional handicrafts and foods from various Indian states. A must-visit for authentic Indian souvenirs.

Visit Doctor Droid Booth

Doctor Droid is a root cause identification slack bot which can assist on-call engineers diagnose incidents and find root cause really fast. All you need to do is reply to your alert message in slack by tagging the bot. If you are interested to get a demo and explore more about Doctor Droid, visit us at Booth in the venue!

Pre-book a meeting with us using this link.

Stop by Booth to discover how Doctor Droids automated RCA can help you debug & fix your production issues faster! What else is up for grabs at the event?

• Puzzles & Goodies: Test your mental muscle and win some amazing gifts.

• $500 Doctor Droid Credits: Show this blog at our booth to receive $500 in Doctor Droid credits!

]]>https://notes.drdroid.io/kubecon-cloudnativecon-india-2024-guide-delhihttps://notes.drdroid.io/kubecon-cloudnativecon-india-2024-guide-delhi<![CDATA[KubeCon + CloudNativeCon North America 2024 Guide -- Salt Lake City, Utah]]>Doctor Droids Guide to KubeCon + CloudNativeCon North America 2024

Welcome to our complete guide for navigating KubeCon + CloudNativeCon North America 2024! Heres all you need to know to make the most of this event in Salt Lake City, Utah, from November 12-15, 2024. Dont forget to visit Doctor Droid at Booth Q45show us this blog for a chance to receive $500 worth of Doctor Droid credits!

Overview

Access Types

KubeCon + CloudNativeCon North America 2024 offers different access passes to cater to all types of attendees:

• Full Access Pass: Includes access to all sessions, keynotes, and co-located events

• KubeCon + CloudNativeCon Only Pass: Includes access to all sessions, keynotes, excluding co-located events

• Academic Pass: Discounted pass for students looking to dive into cloud-native technologies.

• Individual pass: Perfect for attendees who are paying for the conference by themselves.

Be sure to review the registration options on the official KubeCon + CloudNativeCon North America 2024 website.

Exclusive KubeCon + CloudNativeCon North America 2024 Discount Save 20% on Tickets, Courtesy of Doctor Droid!

Thats rightDoctor Droid is a proud sponsor of KubeCon + CloudNativeCon North America 2024, and were hooking you up with an exclusive 20% discount on your tickets! Heres how to secure your spot and save big:

1. Fill out this google form with your basic info.

2. Get your 20% discount code delivered straight to your inbox.

3. Register for KubeCon + CloudNativeCon North America 2024 and enjoy the savings!

Dont sit on thisgrab your discount before its gone!

Co-Located Events at KubeCon + CloudNativeCon North America 2024

Expand your KubeCon + CloudNativeCon experience by joining co-located events, each tailored to specific interests within the cloud-native realm:

• AppDeveloperCon: Focuses on tools and techniques for building cloud-native applications.

• ArgoCon: Dive into Argo workflows, events, and continuous delivery for Kubernetes.

• BackstageCon: Explore Backstages developer portal and best practices for engineering platforms.

• Cilium + eBPF Day: A deep dive into Cilium and eBPF for networking and security.

• Cloud Native & Kubernetes AI Day: Discuss AI and ML workloads on Kubernetes.

• CloudNative StartupFest: Networking and insights for startup founders and innovators.

• Cloud Native University: Educational sessions on the fundamentals of cloud-native technologies.

• Data on Kubernetes Day: Explore data management practices and tools for Kubernetes.

• EnvoyCon: Dedicated to the Envoy proxy community, focusing on networking and observability.

• Istio Day: Learn about Istio and service mesh technologies in cloud-native environments.

• Kubernetes on Edge Day: Explore the role of Kubernetes in edge computing.

• Observability Day: A day centered around observability tools and practices in the cloud.

• OpenFeature Summit: Discussions on feature flagging and experimentation in cloud-native setups.

• OpenTofu Day: Open-source infrastructure management and IaC best practices.

• Platform Engineering Day: Dedicated to platform engineering in cloud-native environments.

• WasmCon: Focuses on WebAssembly and its role in cloud-native development.

KubeCon + CloudNativeCon North America 2024 Unofficial Conference Parties

After a day of learning and networking, unwind at the official conference parties.

• Check out Conference Parties for the latest info on social events.

• Check out events on lu.ma too to keep exploring.

Speakers & Tracks at KubeCon + CloudNativeCon North America 2024

KubeCon + CloudNativeCon features keynotes from industry leaders and breakout sessions across multiple tracks, including:

• Kubernetes Operations: Topics around the deployment, scaling, and security of Kubernetes.

• AI and Machine Learning: How AI and ML are transforming Kubernetes.

• Edge Computing: Use cases and solutions for Kubernetes at the edge.

• Cloud Security: Tools and practices to ensure security in a cloud-native environment.

Planning for KubeCon + CloudNativeCon North America 2024 Utah Logistics

Stays near Salt Lake City

Salt Lake City has several convenient options for accommodation. Whether you prefer hotels or Airbnbs, here are a few recommendations:

• Hotels: As of now, all hotels on booking.com as well as most other websites are sold out right now.

• Airbnb: Airbnb still has a healthy number of properties available in the city, although most of them are not within walking distance of the Convention Center.

Airports nearby

• Salt Lake City International Airport (SLC) is the nearest airport, just 5 miles from downtown.

Visiting Salt Lake City

Restaurants

Salt Lake City offers a diverse culinary scene. Recommended spots include:

• Red Iguana: Known for authentic Mexican cuisine.

• The Copper Onion: A top choice for American fare.

• Takashi: Excellent sushi in the heart of the city.

After Hour Locations

• Beer Bar: Perfect for a laid-back evening with a variety of beers.

• The Bayou: Offers an extensive selection of brews and Cajun-style food.

Coffee Shops

Need a caffeine boost? Check out:

• La Barba Coffee: Known for artisanal coffee.

• Publik Coffee Roasters: A great spot to unwind with quality brews.

Weekend Plans

Consider exploring Utahs natural beauty over the weekend. Popular options include:

• Bonneville Salt Flats: A unique desert landscape.

• Big Cottonwood Canyon: Ideal for scenic drives and hikes.

Visit Doctor Droid Booth - Q45

Doctor Droid is a root cause identification slack bot which can assist on-call engineers diagnose incidents and find root cause really fast. All you need to do is reply to your alert message in slack by tagging the bot. If you are interested to get a demo and explore more about Doctor Droid, visit us at Booth Q45 in the venue!

Pre-book a meeting with us using this link.

Stop by Booth Q45 to discover how Doctor Droids automated RCA can help you debug & fix your production issues faster! What else is up for grabs at the event?

• Puzzles & Goodies: Test your mental muscle and win some amazing gifts.

• $500 Doctor Droid Credits: Show this blog at our booth to receive $500 in Doctor Droid credits!

]]>https://notes.drdroid.io/kubecon-cloudnativecon-north-america-2024-guide-salt-lake-city-utahhttps://notes.drdroid.io/kubecon-cloudnativecon-north-america-2024-guide-salt-lake-city-utah<![CDATA[Keeping keys secure without slowing your iteration speed]]>Context

At Doctor Droid, we are building a cutting-edge AI recommendation platform for on-call teams. Whenever an alert or ticket is raised, Doctor Droid:

• Looks for all past investigations and see if it finds anything similar

• Looks for SOPs for the issue at hand (these SOPs are also created by Doctor Droid by reading past Slack threads & existing docs)

• Executes autonomous investigation for popular infrastructure & microservices symptoms.

Problem Statement

This requires a fair bit of experimentation with our early adopters and extensive usage of Jupyter Notebooks. Often as the Notebooks are not connected to a cloud environment, how does one go about managing secrets and ensure they are not lying around anywhere? I wanted a solution where I could have access to keys JUST-IN-TIME (get it just when I need to run it) and become unavailable right after.

Solution

With Infisical, I found a convenient solution for this issue. Heres how it works:

1. Step 1: Configure keys

2. Step 2: Use APIs to retrieve keys securely on-the-go

```python url = "https://app.infisical.com/api/v1/auth/universal-auth/login"

payload = 'clientSecret=xxxx&clientId=yyyyy' headers = { 'Content-Type': 'application/x-www-form-urlencoded' }

response = requests.request("POST", url, headers=headers, data=payload)

access_token = json.loads(response.text)['accessToken']

url = "https://app.infisical.com/api/v3/secrets/raw/KEY\_NAME?workspaceId=xxxx&environment=dev"

payload = {} headers = { 'Authorization': f'Bearer {access_token}' }

response = requests.request("GET", url, headers=headers, data=payload)

KEY_VALUE = json.loads(response.text)['secret']['secretValue']

```

Benefits of using Infisical:

1. Change environment and get updated key

2. Quarantine keys easily: If youve been close to any production incident, youll know that being able to flush keys in a jiffy is super important and at the same time, super difficult because of its underlying dependencies across the stack. Using Infisical gives me the buffer of instantly disabling access by disabling infisical key/secret

3. Free to get started: Its an opensource project with a convenient cloud option

4. Too many features (although I have only used like 5% of the platform probably) so I feel like as my requirements expand, Ill learn about new things easily

5. Helpful team / community: They have a community, a prompt support team and well-written documentation.

]]>https://notes.drdroid.io/keeping-keys-secure-without-slowing-your-iteration-speedhttps://notes.drdroid.io/keeping-keys-secure-without-slowing-your-iteration-speed<![CDATA[How to investigate Sentry Alert with Doctor Droid]]>Sentry

Sentry is one of the best tools in the industry right now for error and exception tracking. It has high quality SDKs across the stack and has great integrations as well as a powerful dashboard for users to learn about an exception in the code. You can read more about Sentry here.

Debugging an exception in Sentry

An exception could arise because of n-different reasons. It could be because of code change in the place where the exception came, it can be a code change upstream, it could be bad user input or it could even be just an edge case that hadnt appeared until now. In case of canary deployments, it could get even trickier as different containers or users could be on different versions of the code.

In production, investigation of a simple looking Sentry issue can span across multiple data sources & contexts:

• Your infrastructure & deployment resources like Kubernetes

• Your code repository to check the code for recent changes or even analysing the flow of data

• Your database/logs to check for user entered data

• Discussion with internal team members regarding expected behaviour

Using Doctor Droid to debug the issue

Initiating an investigation

You can start investigation of an alert directly from the home page which has all the recent alerts.

Once an investigation is created, this is what it looks like.

Here are the key elements of the investigation panel:

• The alerts that it's investigating

• The recommended investigation strategy and preliminary data for your evaluation

• Additional panels related to related investigations or alerts

Investigation Strategy

Now what is it that it's able to fetch till now? Depending on the alert context, the platform recommends different steps.

It identified that the first thing that it should check is the stack trace itself in Sentry. So it goes and fetches the stack trace from Sentry, including the culprit.

Once it's able to fetch that, it goes and looks for the recent code changes for the same stack trace that it identified within the your GitHub repository. It shows you the recent commits, the URLs, and you can go and check if this this is potentially something that was done in the last couple of days.

You can also then go and check if within your Kubernetes infrastructure if there was any recent deployment that could be correlated with it. Given that this is related to the prototype instance, we can see that here there is a couple of releases for prototype in the last one hour. So it could potentially be the reason for this instance, for this alert to actually come up. And now you have all the data here.

You can ask it for more data, chat with it, ask for more data.

And what's what's also good is that it'll give you references to either existing playbooks, dashboards, or any other data point that your system already has. So we have integrations with almost every tool that your monitoring and observability stack would potentially have.

And we also have options for you to self host these integrations so that the data remains within your own plane.

Try it today

https://www.youtube.com/watch?v=tx_x8BHCK38

If if this is something that looked exciting to you, we have a lot more demos coming up, like how do you auto investigate an API latency alert, or how do you investigate CPU utilisation alerts on your databases.

Visit www.drdroid.io and try it out for yourself for your own stack. We have a free trial that we provide for the tool. So if you have any questions, please reach out to us, and we'll be happy to answer.

]]>https://notes.drdroid.io/how-to-investigate-sentry-alert-with-doctor-droidhttps://notes.drdroid.io/how-to-investigate-sentry-alert-with-doctor-droid<![CDATA[Dr. Patternson: How Meta reduced their MTTR by 50% using AIOps]]>Introduction

For Meta, reducing downtime has been crucial to ensuring millions (or should I say Billions?) of users have a seamless experience. Recently, Meta shared about one of their internal platforms that helped reduce MTTR by ~50% for critical alerts.

This blog explores how Meta accomplished this by leveraging AI, machine learning & runbook automation to transform its incident response processes, making them faster and more efficient.

Let's dive into the key components that enabled this efficiency gain.

Objective

Imagine trying to find a needle in a haystackblindfolded. Thats what incident management can feel like without the right tools. Metas objective was to take off that blindfold and make the process of finding and fixing problems as swift and accurate as possible. Their goal? Cut down the Mean Time to Resolution (MTTR) by half, so that when things go wrong, they can be fixed faster than you can say downtime.

By harnessing the power of AI and machine learning, Meta aimed to automate the grunt work of incident managementspotting issues, figuring out whats broken, and fixing itall without requiring a superhero on standby. This isnt just about cool tech; its about making sure users experience as little disruption as possible, turning potential disasters into minor hiccups that barely anyone notices.

What Meta built?

Alright, let's peek under the hood of Meta's incident-busting machine. They didn't just slap on a new coat of paint; they rebuilt the entire engine. Here are the three turbocharged components that turned their incident response from a clunky old jalopy into a sleek, AI-powered sports car:

Component 1: Automated Runbooks

Remember those old-school detective novels where the brilliant sleuth solves the case with a magnifying glass and a pipe? Well, Meta created a digital Sherlock Holmes, minus the pipe smoke. They call it Dr. Patternson (Dr. P for short), and it's like having a tireless detective on call 24/7.

Dr. P is an automated runbook system that encodes expert knowledge into executable investigation workflows. It's like giving every on-call engineer a cheat sheet written by the smartest person in the room. With its own SDK, simplified APIs, and ML algorithms, Dr. P can quickly analyze data, correlate events, and generate findings faster than you can say "Elementary, my dear Watson."

But wait, there's more! Dr. P comes with a fully managed platform that deploys these runbooks, monitors for issues, and even triggers investigations automatically when an alert fires. It's like having a whole team of digital detectives working round the clock, leaving no log unturned.

Component 2: Analysis Algorithms Service

If Dr. P is the detective, then the Analysis Algorithms Service is its time machine. This nifty piece of tech allows Meta's engineers to zoom through vast amounts of data at warp speed. Picture this: You've got more data than stars in the sky, and you need to find that one glowing red dot that's causing all the trouble. That's where this service comes in. It's packed with ML algorithms for dimensional analysis, time series analysis, anomaly detection, and more. But the real magic is in its pre-aggregation layer, which shrinks datasets by up to 500 times! It's like compressing the entire library of Congress into a pocket-sized book, without losing a single word.

The result? Insights that used to take hours now pop up in seconds. It's so fast, you might think it's predicting the future. (Spoiler alert: it's not. That's still on the roadmap for 2025.)

Component 3: Event Isolation Assistance

Last but not least, we have the Event Isolation Assistance. Think of it as a super-smart metal detector for that proverbial needle in a haystack.

This system uses ML models to rank thousands of events and pinpoint the root cause of an incident. It's like having a psychic on your team, except this one actually works. By focusing on config-based and code-based isolation, it can filter out 80% of the uninteresting events during an active investigation. That's right, it separates the wheat from the chaff, leaving engineers with a much smaller, much more suspicious pile of events to investigate.

But it doesn't just point fingers. The system provides annotations explaining its reasoning, making it transparent and trustworthy. It's like having a really smart friend who not only tells you the answer but also shows their work.

Guided Investigations:

Sometimes, even the smartest AI needs a human touch. That's where Guided Investigations come in. Think of it as a choose-your-own-adventure book, but for fixing tech problems.

These decision trees provide step-by-step workflows that help investigators narrow down the root cause of an issue. It's like having a seasoned pro whispering in your ear, guiding you through the digital labyrinth. By combining automated workflows with human expertise, these guided investigations can tackle complex issues that might stump a fully automated system.

And the best part? They're right where you need them, integrated with Meta's detection systems. It's like having a tech support genie, ready to pop out whenever an alert goes off. No need to rub a lamp just click a button!

Current state of Investigations at Meta:

So, where does Meta stand now in their AIOps journey? Let's just say they've gone from digital chaos to zen master status.

Today, Meta's foundational systems are more popular than cat videos (well, almost). Hundreds of teams have adopted them, running over 500,000 analyses per week. That's more check-ups than a hypochondriac gets in a lifetime!

The impact? A cool 50% decrease in MTTR for critical alerts across the company. It's like they've upgraded from a horse-drawn buggy to a supersonic jet when it comes to fixing problems. Take the Ads Manager team, for instance. They've gone from spending days investigating issues to resolving them in minutes. It's like they've traded in their magnifying glass for a high-powered microscope with AI-assisted focusing.

Implementing your Own Dr. Patternson using Doctor Droid:

If you want to implement a solution like Dr. Patternson within your team without investing the time or cost like Microsoft, you might want to explore Doctor Droid.

Doctor Droid is a AI-assisted intelligence platform to help engineering teams reduce investigation time of production issues by 10x. Here's what you can do with Doctor Droid:

(a) Codify your investigation mental models:

Doctor Droid PlayBooks is an Open-Source On-call automation platform. With one click, you can run your investigation steps and have all the diagnosis data across all tools, directly fed in response to your alerts.

(b) Leverage your past knowledge to get intelligent suggestions:

Doctor Droid's AIOps Platform can provide your on-call engineers with intelligent recommendations by leveraging the knowledge that you already have accessible in your system over the past time.

Both A & B combined, you are effectively going to end up with an equivalent of Dr. Patternson.

Try it out today by signing up here!

Conclusion:

Meta's AIOps journey has transformed their incident response from a digital firefight into a well-oiled machine. Let's break down the impressive results:

• 50% reduction in Mean Time to Resolution (MTTR) for critical alerts across the company

• Over 500,000 automated analyses run per week

• 80% of uninteresting events filtered out during active investigations

• Ads Manager team improved investigation time from days to minutes

• Nearly 50% of previously manual investigations now automated

These statistics paint a picture of a dramatically more efficient system, but what does it mean in the real world?

For Meta, it means:

• Fewer service disruptions for millions of users

• Faster resolution when issues do occur

• Engineers spending less time on repetitive tasks and more on innovation

• Improved overall system reliability and user experience

The secret sauce? A combination of:

1. Automated Runbooks (Dr. Patternson)

2. Analysis Algorithms Service

3. Event Isolation Assistance

4. Guided Investigations

This powerful quartet has turned Meta's incident response into a symphony of efficiency, conducting a harmonious blend of AI automation and human expertise.

As we look to the future, Meta's AIOps journey serves as a beacon for the tech industry. It shows us that with the right tools and approach, we can tame the chaos of complex systems and create a more reliable digital world. So the next time you scroll through your feed without a hitch, remember - there's a good chance Meta's AIOps team had a hand in making that seamless experience possible.

]]>https://notes.drdroid.io/dr-patternson-how-meta-reduced-their-mttr-by-50-using-aiopshttps://notes.drdroid.io/dr-patternson-how-meta-reduced-their-mttr-by-50-using-aiops<![CDATA[RCACoPilot: A breakdown of how Microsoft built their Automated RCA Bot]]>Introduction

Big Tech companies often have scale enough to justify allocating resources to building internal tools. In this blog, we discuss about RCACoPilot -- an automated incident classification and investigation engine built by Microsoft to improve the lives for their on-call engineers.

Less than a year ago, Microsoft published a research paper discussing RCACoPilot. It's a longish paper ~ 16 pages, so I decided to condense it into a shorter blog.

Context

Picture this: You're an on-call engineer at Microsoft. It's 3 AM, and suddenly, alerts start blaring. Something's wrong with the email service (which delivers over 150 billion messages daily) that millions of people rely on. Your job? Figure out what's causing the issue and fix it ASAP. No pressure, right?

This scenario plays out all too often at Microsoft & at most companies. A company's systems are only getting more complex by the day, and on-call engineers were drowning in a sea of alerts, logs, and metrics. This often leads to escalations and time being spent on tickets rather than planned work. They needed a way to streamline the process and a way to quickly make sense of all this information and zero in on the root cause of problems. That's what RCACoPilot tries to solve for them.

Why should you even read this article? The Results

Let's cut to the chase: How well does RCACopilot perform & is it even worth reading about their approach? Their performance is pretty impressive, as it turns out.

Tested on 653 real-world incidents from Microsoft's email service (which handles about 150 billion messages daily), RCACopilot achieved:

• 76.6% accuracy in predicting root cause categories

• A Macro-F1 score of 0.533, showing good performance across various incident types

• Significantly reduced MTTR:

• Auto-diagnosis ran with an average time of 1-10 minutes depending on the complexity of the incident handlers (more on it below).

• An average classification time of just 4.2 seconds per incident.

[Quick note on that Macro-F1 score: It's a measure that gives equal importance to each category, regardless of how often it appears. A score of 0.533 tells us that RCACopilot performs well across various incident types, not just the common ones. This is crucial in a system where rare, critical issues are just as important as frequent, minor ones.]

These numbers outperformed all baseline methods, including traditional machine learning approaches and non fine-tuned GPT models.

But the real proof is in the deployment. Parts of RCACopilot have been in use at Microsoft for over four years, across more than 30 teams. On-call engineers report significant time savings in incident management tasks, from diagnosis to mitigation.

What this basically means is that in less than a few minutes, most of the likely investigation steps are run, analysed and the engineers are told a likely incident Root Cause category.

RCACoPilot -- The Architecture

Now, let's peek under the hood of RCACoPilot. Think of it as a super-smart detective for computer problems. Here's a simple breakdown of how it works:

1. Diagnosis Identifier: In the original paper, this is part of what they call the "Diagnostic Information Collection Stage". It is responsible to take in an incident from their alerting tool, parse the incident context and match it to the existing playbooks (called Incident Handlers) basis the mapping logics defined.

2. Diagnosis Data Fetch & Summarisation: The playbook identified in the previous section is custom defined by the on-call engineers ( called OCEs). The system now executes all the steps pre-configured in their playbook -- from fetching logs & metrics to running more steps.

3. Incident Predictor: Now on top of the diagnosis data that's fetched, a couple of things are done: converting it into an embedding and comparing the embedding with past embeddings. These embeddings are now leveraged to identify the potential Root Cause for this issue. This corresponds to the "Root Cause Prediction Stage" in the paper.

The actual benefit of RCACoPilot lies in how the entire pipeline works out. When an alert comes in, the Incident Handler kicks into gear, following predefined playbooks to gather relevant data. This could involve querying databases, analyzing log files, or even running diagnostic scripts.

Component 1: Diagnosis Identifier

Incident Parser:

The Incident is the entry point of RCACoPilot. It's connected to the existing alerting systems in the company through triggers or webhooks. It then parses the incident to identify the key entities of interest.

Handler:

A Handler is set of steps pre-defined to be run for a specific type of investigation. This is effectively a programmatic SOP for a certain type of issue. These are NOT AI GENERATED. These are something that every on-call engineer documents -- the investigation strategies for a type of issue within their service. Types of Actions

The Incident Handler uses several types of actions to investigate and respond to incidents:

1. Scope Switching Action: This allows the handler to adjust its focus dynamically. It might start by looking at a single server, then expand to an entire cluster if needed.

2. Query Action: Think of this as the handler's way of asking questions. It can pull data from various sources like databases, log files, or even run scripts to gather system information. The results come back as key-value pairs, giving the handler structured data to work with.

3. Mitigation Action: Sometimes, the handler can take steps to address the problem directly. This could involve restarting a service, clearing disk space, or even calling in specialized teams for complex issues.

This is what an incident handler looks like for too many messages stuck in the delivery queue alert.

Component 2: Diagnosis Data Fetch & Summarisation

In this part, the system is executing a series of commands as per the incident handler definition.

• It has access to different internal / external tools with the relevant information.

• Each step in handler is structured and mapped to a technical task (be it an API call, a log fetch, a metric fetch or any other task).

• The system automatically interacts with every different data tool and then fetches it.

These are some of the data points that the system can fetch from the handlers (playbooks):

• Logs: Application logs, system logs, security logs.

• Metrics: Performance metrics and resource utilization stats.

• Traces: Detailed records of how requests flow through the system.

• Configuration data: Current system settings that might be relevant.

Component 3: Incident Predictor

The ML part of RCACopilot happens in what we call "Incident Predictor". This is where it uses Large Language Models (LLMs) to make sense of all the data collected by the Incident Handler.

Here's how it uses LLMs to analyze incidents:

1. Summarization: First, the LLM takes all the diagnostic information collected and creates a concise summary. This step is crucial because it condenses vast amounts of data into something manageable for both the AI and human engineers.

2. Similarity Matching: Next, RCACopilot creates an embedding to represent incidents as points in a high-dimensional space and then uses a Nearest Neighbour Search Algorithm. ELI5: similar incidents will be close to each other in this space.

Using this technique, RCACopilot finds past incidents that are most similar to the current one. This is important because similar past incidents can provide valuable clues about the current problem.

3. Chain-of-Thought Prompting: This is where things get really interesting. RCACopilot uses a technique called "Chain-of-thought" prompting. Instead of just asking the LLM "What's the root cause?", it prompts the model to think through the problem step-by-step, much like a human engineer would.

It does this by showing the LLM examples of how similar past incidents were solved. This is akin to training a junior engineer by walking them through past case studies before asking them to solve a new problem.

4. Root Cause Prediction: Based on this careful analysis, the LLM then predicts the most likely root cause of the incident. But it doesn't stop there.

5. Explanation Generation: Crucially, the LLM also generates an explanation for its prediction. This isn't just a black box spitting out an answer - it's more like a colleague explaining their reasoning. This explanation helps human engineers understand and verify the AI's conclusion.

Feedback loop: While it doesn't learn in real-time, feedback from engineers can be used to periodically retrain and improve the model. This means RCACopilot can get better over time, learning from each incident it analyzes.

By leveraging the power of LLMs in this way, RCACoPilot can quickly analyze complex incidents, drawing insights from vast amounts of data and past experiences. It's like having an AI assistant that has seen every incident your organization has ever faced, can think through problems step-by-step, and can clearly explain its reasoning. This not only speeds up incident resolution but also helps engineers learn and improve their own diagnostic skills.

Implementing your Own RCACoPilot using Doctor Droid:

If you want to implement a solution like RCACoPilot within your team without investing the time or cost like Microsoft, you might want to explore Doctor Droid.

(a) Codify your investigation mental models:

Doctor Droid PlayBooks is an Open-Source On-call automation platform. With one click, you can run your investigation steps and have all the diagnosis data across all tools, directly fed in response to your alerts.

(b) Leverage your past knowledge to get intelligent suggestions:

Doctor Droid's AIOps Platform can provide your on-call engineers with intelligent recommendations by leveraging the knowledge that you already have accessible in your system over the past time.

Both A & B combined, you are effectively going to end up with an equivalent of RCACoPilot.

Try it out today by signing up here!

]]>https://notes.drdroid.io/rcacopilot-a-breakdown-of-how-microsoft-built-their-automated-rca-bothttps://notes.drdroid.io/rcacopilot-a-breakdown-of-how-microsoft-built-their-automated-rca-bot<![CDATA[How to setup your dev environment for editing & contributing code?]]>Playbooks is a web server application that interacts with a Django API server via Nginx. It also includes salary workers for scheduling asynchronous tasks and a persistence layer consisting of Postgres and Redis cache.

https://www.youtube.com/watch?v=OiwgXSncNuo

Setup Phase

Step 1: Ensure a running instance of Postgres and Redis on the local machine. Installation guides are available for different machine types to set up Postgres and Redis. Alternatively, Docker can be used for the same purpose.

Step 2: Use the DB Docker Compose file located in the Docker folder to set up Postgres and Redis.

Step 3: Verify the setup by checking Docker Desktop to see if both instances are running.

Build Phase

Step 4: Create all the tables and relations between them that the API server will use by running the command

python manage.py migrate

Step 5: Confirm the creation of all tables and relations in the Postgres DB using a tool like Postico.

Run Phase

Step 6: Start the API server on the local machine using the below command

python manage.py runserver

The server will run on port 8000, but the port can be changed according to preference.

Step 7: Go to the web folder and use the following command to bring up the React application

npm start

Step 8: Inside the web folder, use the command shown to start the Nginx server. This ensures the React application can connect with the Django application.

nginx -c $PWD/nginx.local.conf -g "daemon off;"

Step 9: Once both servers are running, open a browser and go to localhost to see the running application.

Testing Changes

Step 10: Test changes by creating a playbook and an HTTP task. For example, remove the method field in the API task and save the changes.

Step 11: Django will load the changes and restart the server. Refresh the application platform and try to create an API task. The method field should be missing.

Step 12: To restore the method field, undo the changes, save, and refresh the page. Try to create the task again.

For further assistance with local setup, refer to the details mentioned in the github project. The contribution page includes a link to the playbook architecture and local setup guide.

]]>https://notes.drdroid.io/how-to-setup-your-dev-environment-for-editing-contributing-codehttps://notes.drdroid.io/how-to-setup-your-dev-environment-for-editing-contributing-code<![CDATA[What is a PlayBook and what is core components of a playbook?]]>A playbook is a set of instructions that a Doctor Droid bot or an on-call engineer follows during a production incident.

https://www.youtube.com/watch?v=T9KfunP9juA

A playbook consists of tasks. A task is an instruction that's executed through the portal. Let's create a task.

A task could involve fetching a metric from CloudWatch or running a kubectl command on a server. Let's create a new task to fetch logs from CloudWatch.

Add notes to the playbook. A note is a custom guideline for the user, related to the playbook or a specific step. Add a note indicating that this task fetches logs.

Hover over a step to view the note. It's also possible to add multiple tasks in a step. For instance, fetch a Datadog metric. After adding the task, check the metric.

Add variables to the playbook. For example, add a service as a variable. To use the variable, enter a dollar sign followed by the variable name.

Add a step with conditions. A condition is a rule that determines whether a certain action should be taken. For example, check logs from CloudWatch and check if the row count equals six. Then add a task. Add a metric and run the task.

Save the playbook. Execute the playbook and observe the results with the condition. If the condition isn't met, the step isn't recommended and the next step is executed.

These are all the core components of a playbook.

]]>https://notes.drdroid.io/what-is-a-playbook-and-what-is-core-components-of-a-playbookhttps://notes.drdroid.io/what-is-a-playbook-and-what-is-core-components-of-a-playbook