LetspingLetsPing
← Docs

Observability & Ops

Markov firewall and ingest metrics for Datadog, Prometheus, and custom scrapers

Overview

LetsPing emits structured JSON logs for the Markov behavioral firewall and the ingest pipeline. These are designed for log aggregation (Vercel, Datadog, etc.) and can be scraped into Prometheus, Grafana, or custom dashboards. In production, metrics are written to stdout with level: "metric" so you can filter and route them separately.

The built in Observability tab is optional. You can forward LetsPing metrics and firewall events into your existing observability stack and treat LetsPing as a control plane that emits signals rather than a new place you are forced to watch.

Markov firewall metrics

The behavioral firewall tracks transitions between nodes and scores each by a per edge edge anomaly score using running Welford statistics. Thresholds are local to each transition and enforcement only trips after a small burst of consecutive anomalies, so one noisy path does not mask or over amplify the rest. These metrics let you monitor baseline health and anomaly volume:

letsping_markov_edge_anomaly_score— Numeric score for each transition. Labels: project_id, service, priority, status. Use for SLOs and anomaly-rate dashboards.
letsping_markov_anomalies_total— Counter incremented when a transition exceeds the anomaly threshold. Labels: project_id, service, priority.
letsping_markov_mode— 0 = shadow, 1 = locked. Indicates whether the firewall is observing only or actively enforcing.

For more on how the Markov firewall works (edge anomaly score, Rich Markov, per-priority chains), see the behavioral firewall and LangGraph HITL + firewall guides.

Ingest metrics

The ingest route and process-request worker emit metrics for throughput, latency, and status distribution:

letsping_ingest_request_total— Counter per request. Labels: path (ingest, worker_process_request), status, project_id.
letsping_ingest_latency_ms— Duration in milliseconds. Labels: path, project_id.
letsping_ingest_status_count— Status counts from the worker (e.g. APPROVED, REJECTED, flagged). Labels: path, status.

Structured JSON format

Each metric is logged as a single JSON line, for example:

{
  "level": "metric",
  "metric": "letsping_markov_edge_anomaly_score",
  "project_id": "proj_...",
  "service": "billing-agent",
  "value": 2.1,
  "labels": { "priority": "high", "status": "ok" },
  "timestamp": "2026-02-28T12:00:00.000Z"
}

Scrape these from your log drain (e.g. Vercel Log Drains, Datadog log intake, CloudWatch Logs) and parse by level === "metric". Prometheus exporters can tail log files or subscribe to a log stream and convert to Prometheus exposition format. Datadog can create metrics from log attributes.

OpenTelemetry: firewall block events

When the LetsPing firewall or a guardrail blocks or pauses a request, we emit a structured log event (event: "letsping.firewall.block") with a stable code (e.g. PROMPT_INJECTION_DETECTED, BEHAVIORAL_ANOMALY) and trace_id. OTel log collectors (or LangWatch) can ingest this and show a span such as BLOCKED BY LETSPING FIREWALL: UNAUTHORIZED_FINANCIAL_ACTION on the same trace as your agent's tool call. See the repo doc docs/OPENTELEMETRY_FIREWALL_INTEGRATION.md for the full "Securing the Observable Agent" integration play.

Next steps

• See Trust anchor (security architecture) for identity, behavior, and settlement details.

• For agent-first observability (signup metrics, Redis keys), see docs/AGENT_FIRST_SECURITY_AND_SCALE.md.

• Guardrails (including sequence_entropy with min_markov_anomaly_score) are documented on the firewall page.