Events & services dashboard

An admin page that answers three questions at a glance:

1. Is every topic carrying what it normally carries? Messages per second per topic, now vs the same hour yesterday and a week ago.
2. Is every consumer keeping up? Lag per consumer group and whether it grows.
3. Are the services healthy? CPU, memory, restarts and network per swarm service, next to per-action event counts and handler failures.

Everything comes from Prometheus. The page sends no PromQL: the backend owns a fixed catalogue of queries and the browser asks for series by name.

Data sources

Each source is optional: the overview endpoint reports a missing one in a missing list ("kafka-exporter (consumer lag)", "go-events ≥ v0.4.3 metrics") and the page shows it as Not available yet instead of an empty chart.

1. Emit event metrics from every service

• Services on go-events/module — nothing to do since v0.4.3 (on by default, EVENTS_METRICS=false disables). Make sure /metrics is served (go-modules/metrics).
• Standalone workers — wire the observer yourself and give the worker a /metrics listener (a worker with no HTTP server is otherwise invisible):
  goCopy

  bus := eventbus.New(tr, eventbus.WithName("default"), eventbus.WithObserver(prom.Default()))
  go http.ListenAndServe(":8080", promhttp.Handler())
  

• Consumers that bypass bus.Register (their own sarama loop, batch readers) — call obs.Consumed(...) after each message. Otherwise the topic fills up and the dashboard shows no consumer activity for it.
• Return eventbus.ErrSkipped for deliberately dropped messages so they count as skipped, not as failures.

Then add a scrape job per service. Swarm services with several replicas need dns_sd_configs on tasks.<service> so each replica is scraped on its own.

2. Run kafka-exporter

kafka-exporter:
  image: danielqsj/kafka-exporter:v1.9.0
  command:
    - --kafka.server=kafka-1:9092
    - --group.filter=^(nameniac|gymmer).*   # only your groups — see below
    - --no-offset.show-all                  # boolean flags are --no-x, never --x=false
    - --concurrent.enable
    - --topic.workers=20

3. The backend — a fixed query catalogue

Two endpoints, both admin-only:

Series names map to PromQL in Go — event_topics, event_actions, event_consumed, event_failures, event_handle_p95, consumer_lag, service_cpu, service_memory, service_restarts, service_network, plus your module's own. Unknown names 404. Never accept PromQL from the browser.

PromQL that is easy to get wrong

# Topic rate: rate() of the broker's cumulative counter …
sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesinpersec{topic=~`nameniac\.events(\..+)?`}[1h]))
# … NOT the *_oneminuterate / *_fiveminuterate gauges: they are decaying averages
# and stay inflated long after a burst (40 msg/s on a topic idle for 15 minutes).

# Same window a day / a week ago, for comparison
sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesinpersec{…}[1h] offset 7d))

# Lag only for groups that still have a member. A retired consumer's group keeps a
# growing lag forever and drowns the chart.
sum by (consumergroup) (kafka_consumergroup_lag{consumergroup=~`nameniac.*`})
  and on (consumergroup) (sum by (consumergroup) (kafka_consumergroup_members{consumergroup=~`nameniac.*`}) > 0)

# Restarts: containers started in the last hour (a crash loop is a staircase)
count by (container_label_com_docker_swarm_service_name) (container_start_time_seconds{…} > time() - 3600)

• Put regexes in backtick raw strings. "nameniac\.events" in double quotes is a PromQL parse error (400); `nameniac\.events` is not. Build them with a raw() helper and regexp.QuoteMeta the prefix.
• Lag is briefly negative when the committed offset is read after the log end — clamp it to 0.
• Run the overview's queries concurrently, each with its own timeout, and report per-query errors in errors rather than failing the page.

Anomaly rules

Computed in Go from the overview's samples — deterministic, testable, no ML:

The reference is last week's same hour, falling back to yesterday when last week was quiet (weekly patterns beat daily ones for scheduled imports). Critical first.

The misrouted rule pays for itself: a service still resolving topics from an old AsyncAPI document published to the bare nameniac.events topic for days, where no consumer listened. Mirror the rules as Prometheus alerts (nameniac_events_misrouted, nameniac_consumer_lag_growing) so they page even when nobody has the page open.

Scoping

A shared Prometheus scrapes every project on the cluster. Scope the page to one:

SYSTEMHEALTH_JOBS lists your own jobs plus the shared infrastructure you run on (Kafka, Mongo, Redis, Traefik, node-exporter, cAdvisor). Alerts are kept when their job matches, when their container name starts with the service prefix, or when they carry neither (node-level alerts — every project runs on the nodes). Another project's public_site_down or a replica set it alone uses stays off your page.

4. The page

• Order: anomalies → topics table → consumer-lag cards → charts grouped Events, Services, then your domain (e.g. Drop-catch).
• Range control (6 h / 24 h / 7 days) in one row above the charts, kept in the URL.
• Charts follow the dataviz rules: top 7 lines + "Other", colour assigned by series name (a hash into the fixed 8-slot palette) so a line keeps its colour when the set changes, crosshair tooltip, legend with the latest value, and a table toggle on every chart (the light palette needs it for contrast).
• Status is a badge with icon and label, never colour alone.
• Refresh the overview every 60 s; charts refresh on demand.

Checklist for a new project

1. Every service on go-events ≥ v0.4.3, /metrics served, a scrape job per service.
2. kafka-exporter with --group.filter for your groups.
3. cAdvisor on every node (usually already there).
4. The four scoping env vars on the admin API.
5. Open the page: anything under Not available yet is a source still missing.
6. Add alert rules for misrouted and lag_growing.

Further reading

• go-events — Metrics
• Observability — flow traces and live overlays
• Deploying to production