Skip to main content

SLOs & alerts

Turn "everyone knows payments matters most" into configuration: classify services into criticality tiers, give each tier an error budget and a latency objective, and route down to the pager while degraded goes to a Slack channel. This guide is Helm-first — the production path.

Prerequisites​

  • avuru obs installed via Helm with the service-health and alerting modules on (they're on by default).
  • Traffic flowing, so services have RED data to judge.

Steps​

  1. Classify services into tiers. Name the groups that matter and let everything else auto-group by namespace at the default tier:

    serviceGroups:
      defaultTier: T2
      groups:
        - name: payments
          tier: T0
          selector: { namespaces: [payments] }
        - name: storefront
          tier: T1
          selector: { services: [web, catalog, search] }

  2. Set objectives per tier. Thresholds resolve by precedence — services > tiers > defaults — so tighten T0 without touching the long tail:

    serviceGroups:
      thresholds:
        defaults:
          errorRateWarn: 0.01        # 1% → degraded
          errorRateCrit: 0.05        # 5% → down
          latencyP95ObjectiveMs: 500
          minSampleCount: 5
        tiers:
          T0: { errorRateWarn: 0.005, errorRateCrit: 0.02, latencyP95ObjectiveMs: 300 }
        services:
          reports: { latencyP95ObjectiveMs: 2000 }   # slow by design

    Every judgement shows its work on the /health board: error rate 4.2% ≥ 1% budget, p95 780ms ≥ 500ms objective.

  3. Route by audience, not just severity. Two rules, two channels — the pager only hears about sustained T0 trouble:

    alerting:
      channels:
        - name: pager
          type: webhook
          url: https://events.pagerduty.com/integration/xxx/enqueue
          secret: "hmac-signing-secret"
        - name: team-slack
          type: webhook
          url: https://hooks.slack.com/services/xxx
      rules:
        - name: t0-down
          when: not-healthy          # degraded OR down
          for: 5m                    # sustained — a blip never pages
          selector: { tiers: [T0] }
          channel: pager
        - name: t1-early-warning
          when: degraded
          for: 10m
          selector: { tiers: [T1] }
          channel: team-slack

  4. Apply, or edit live. helm upgrade — or kubectl edit the rendered ConfigMaps: the hub hot-reloads both files within ~15 s, no restart. Bad config (unknown tier, undeclared channel, empty selector) is rejected loudly, not silently ignored.

Verify​

# Thresholds visibly applied — statuses carry their reasons:
curl -s 'http://<hub>/api/v1/health/groups' | jq '.groups[] | {name, status, reason}'

# Rules loaded (channel secrets are never serialized):
curl -s 'http://<hub>/api/v1/alerts/rules' | jq

Honest scope​

The rules above alert on sustained health status — a target staying down/degraded for a duration — not on multi-window error-budget burn rate. Burn-rate alerting is on the roadmap; until then, the tier objectives give you the SLO language while status transitions give you the mechanism.

Next​