promanomaly is an open-source anomaly-detection service for Prometheus-compatible metrics. It reads a YAML config, queries a long-term TSDB (VictoriaMetrics by default), runs robust statistical detectors over a rolling window of recent data, and exposes anomaly scores on /metrics — so an anomaly becomes a first-class metric: graphable in Grafana, alertable through Alertmanager, indistinguishable from anything else in your monitoring stack.

The idea is simple: you already collect the data. promanomaly tells you when that data is behaving unlike itself — a step after a deploy, a spike on a sparse signal, a series drifting away from its cohort — in the same language your stack already speaks.

It is the sibling project to promforecast. Where promforecast does forecast-based anomaly detection on signals with learnable seasonality, promanomaly handles everything else: change-points, distributional shifts, and statistical baselines on signals that aren’t reasonably forecastable. Install one, both, or neither — they share architectural DNA but no runtime dependency.

What it is

A small Python service that runs as a normal Kubernetes Deployment next to your existing Prometheus. It never touches production Prometheus — all reads go through the long-term TSDB (VictoriaMetrics, Mimir, Thanos all work). An internal scheduler reruns the detectors on an interval, typically once a minute; between runs, /metrics serves the most recent snapshot.

It’s stateless. There’s no database to operate. Baselines are recomputed from a rolling window on each run and the results live in the scraped /metrics endpoint, so a restart just means the next run reproduces everything.

It’s config-driven. Adding a thing to watch is a YAML change, not a code change — point a PromQL query at the metric you care about, attach one or more detectors, and the anomaly score shows up as a metric.

What it can do

  • Score any PromQL query. Whatever you can express in PromQL — CPU saturation, error rates, queue depth, filesystem fill, business KPIs — can be watched. PromQL in, Prometheus exposition format out.
  • Detectors matched to signal shape. Robust statistical baselines (MAD, Hampel), change-point detection, and distribution-shift detectors, each suited to a different kind of misbehaviour. It can auto-select a detector per series, or you can pin specific ones.
  • A score, not just a yes/no. Each series gets a continuous anomaly score, plus the baseline it was scored against (and an explicit upper/lower band for detectors that model one), so a “how weird is this, right now” panel is just a graph.
  • Ensemble fusion, kept simple. Run several detectors on one signal and fuse them with max (alert if any fire) or voting (alert when at least N agree). No black-box weighting — anything fancier you compose in recording rules.
  • Duration and severity built in. It tracks how long a signal has been over threshold and emits a severity, so “anomalous for 10 minutes” is alertable without a stopwatch in your alert rules.
  • Fleet rollups with bounded cardinality. Per-group counts of how many series are currently anomalous, and how dense the anomalies are, so you can alert on “a lot of things went weird at once” without enumerating them.

Built to behave in a real monitoring stack

  • Anomalies are just metrics. Same labels, same retention path, same dashboards. You compose with the recording rules and alerts you already know, rather than learning a new system. Labels (id, group, …) line up with promforecast on purpose, so you can join the two in PromQL.
  • The detector emits signal; it doesn’t decide. Flap suppression, silencing, deploy correlation and score damping belong in Alertmanager — promanomaly stays a clean source of truth.
  • Cardinality guardrails. Hard limits on series-per-query and total series, with overflow that drops lowest-priority groups first and counts what it dropped — anomaly detection is a cardinality multiplier and the safety controls are first-class.
  • Drop-in install. A Helm chart for the detector, and an umbrella chart that bundles VictoriaMetrics, dashboards and example alerts for a one-command start.

A few examples

Install it next to an existing Prometheus with Helm:

helm repo add promanomaly https://esops-dev.github.io/promanomaly
helm install promanomaly promanomaly/promanomaly-stack

Watching a metric is a few lines of YAML. Point a query at a node’s CPU and run two complementary detectors over a rolling window:

datasource:
  url: http://victoriametrics:8428/

defaults:
  window: 1h
  step: 15s
  alert_thresholds:
    score: 3.0

groups:
  - name: node_signals
    queries:
      - id: node_cpu_busy_pct
        promql: |
          100 - (avg by (instance)
            (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
        detectors:
          - name: MAD       # robust statistical baseline
          - name: Hampel    # spike-resistant outlier filter

That id becomes the metric name. The detector now exposes, on /metrics, a score and the baseline it was measured against:

anomaly_score{id="node_cpu_busy_pct", group="node_signals", detector="MAD"}              4.7
anomaly_baseline{id="node_cpu_busy_pct", group="node_signals", detector="MAD"}           31.2
anomaly_outside_threshold{id="node_cpu_busy_pct", group="node_signals", detector="MAD"}  1

Because it’s just a metric, “this signal is behaving abnormally” is an ordinary PromQL alert — and anomaly_duration_seconds means you can require it to persist:

- alert: MetricAnomalous
  expr: anomaly_outside_threshold == 1
  for: 10m
  labels:
    severity: warning

Fuse several detectors into one composite signal — here, only flag a series when at least two detectors agree:

groups:
  - name: api_latency
    ensemble:
      method: voting
      min_detectors: 2
    queries:
      - id: api_p99_seconds
        promql: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
        detectors:
          - name: MAD
          - name: Hampel
          - name: ChangePoint
- alert: LatencyAnomaly
  expr: anomaly_composite_outside_threshold{group="api_latency"} == 1
  for: 5m

Catch “a lot of things went weird at once” with a fleet rollup, instead of enumerating every series:

- alert: WidespreadAnomaly
  # more than 20 series in this group are anomalous right now
  expr: anomaly_active_series{group="node_signals"} > 20
  for: 5m
  labels:
    severity: critical

You can also probe a single query straight from the command line, without deploying anything:

promanomaly detect-once \
  --query 'rate(http_requests_total{job="api"}[5m])' \
  --window 1h --detector MAD

Operational metrics

Beyond the anomaly scores themselves, it exposes the metrics you’d want to run it in production — last run timestamp, run duration, series counts, drops, failures, detect duration, and leadership state when running highly available — so the detector monitors itself the same way everything else in your stack does:

anomaly_last_run_timestamp_seconds{group}
anomaly_run_duration_seconds{group}
anomaly_series_count{group}
anomaly_series_dropped_total{group, reason}
anomaly_failures_total{group, reason}
anomaly_group_ready{group}

In short

If you run Prometheus and keep eyeballing graphs to spot the moment something started misbehaving, promanomaly turns that hunch into a metric. Write a PromQL query, attach a detector, get a continuous anomaly score with a baseline, a duration and a severity — all in plain Prometheus exposition format, ready for the dashboards and alerts you already have.