Clawker includes an optional monitoring stack that collects telemetry from Claude Code agents running in containers. It provides dashboards for logs and metrics — giving you visibility into what every agent is doing, how much it costs, what tools it’s calling, and whether it’s making progress.
What You Get
The stack ingests logs into OpenSearch and exposes them through OpenSearch Dashboards for ad-hoc querying and dashboarding. Prometheus backs metrics with its own UI. You can see per-agent and per-project activity for:
- Cost and token usage — API call costs, input/output tokens, rate limiting
- Coding activity — file edits, tool invocations, command executions
- Tool usage — which tools the agent calls, how often, and how long they take
- Session detail — full event timeline for individual agent sessions
- Egress traffic — Envoy access logs and CoreDNS query logs from the firewall stack (when enabled)
Architecture
The monitoring stack runs as four long-lived Docker Compose services on the clawker-net network, plus one short-lived bootstrap container that runs once per monitor up to preconfigure the cluster:
Claude Code (in container) Control plane (host)
| |
| OTLP (HTTP/gRPC) — agent lane | OTLP/gRPC + mTLS — trusted infra lane
| logs, metrics, traces | zerolog bridge, Envoy access logs,
| | CoreDNS query logs, netlogger ebpf-egress
v v
OTEL Collector ──┬──> OpenSearch (logs + traces) ──> OpenSearch Dashboards
└──> Prometheus (metrics + spanmetrics) ──> Prometheus UI
\\
\\── (also read by OpenSearch Dashboards
via a preconfigured datasource)
CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 flag (baked into the image alongside CLAUDE_CODE_ENABLE_TELEMETRY=1). The OTEL Collector’s spanmetrics connector also produces RED (rate/error/duration) metrics from incoming spans and feeds them into the same Prometheus pipeline.
Services
| Service | Image | Default Port | Purpose |
|---|
| OTEL Collector | otel/opentelemetry-collector-contrib | 4318 (HTTP), 4317 (gRPC) | Receives telemetry from agents and routes it to backends |
| OpenSearch | opensearchproject/opensearch | 9200 | Stores telemetry across the claude-code (Claude Code OTLP), clawker-cli (host CLI), clawker-cp (clawker-cp mTLS push), clawker-envoy (firewall HTTP/TLS/TCP access logs), clawker-coredns (firewall DNS query logs), clawker-ebpf-egress (eBPF per-decision egress events) log indices, plus the traces/clawker (SS4O) trace index |
| OpenSearch Dashboards | opensearchproject/opensearch-dashboards | 5601 | UI for querying logs, dashboards, and (via the preconfigured clawker_prometheus direct-query datasource) Prometheus metrics |
| Prometheus | prom/prometheus | 9090 | Time-series metrics storage and UI |
| clawker-opensearch-bootstrap | curlimages/curl | — | One-shot init container. Runs after OpenSearch + Prometheus are reachable, before the collector starts; applies templates, ISM, datasource, workspace, saved objects, then exits |
dev.clawker.purpose=monitoring) and attached to the clawker-net network. The OpenSearch security plugin is disabled by default for local development — Dashboards is reachable at http://localhost:5601 with no login required. Set OPENSEARCH_JAVA_OPTS via the opensearch_heap_mb setting if you need more heap.
Service Hostnames Are Constants
Individual service hostname constants (MonitoringServiceOtelCollector, MonitoringServicePrometheus, MonitoringServiceOpenSearchNode, MonitoringServiceOpenSearchDashboards) are defined in internal/consts/monitoring.go. The MonitoringServiceHostnames slice — consumed by CoreDNS internalHosts forward zones — intentionally contains only otel-collector and prometheus: agents push telemetry through the collector and never query OpenSearch directly, so opensearch-node and opensearch-dashboards are excluded from the forwarded set (containers on clawker-net that do need those reach them via Docker’s embedded resolver).
How Agents Connect
OTEL endpoints are baked into every Clawker image at build time. At container creation, Clawker checks whether the monitoring stack is running and conditionally enables telemetry:
OTEL_EXPORTER_OTLP_ENDPOINT — single base URL pointing at the collector’s OTLP/HTTP listener (baked into image). The Claude Code SDK appends /v1/metrics, /v1/logs, /v1/traces per signalOTEL_METRICS_EXPORTER / OTEL_LOGS_EXPORTER / OTEL_TRACES_EXPORTER — all set to otlpCLAUDE_CODE_ENABLE_TELEMETRY — set to 1 when the otel-collector container is detected as running on clawker-net, set to 0 otherwiseCLAUDE_CODE_ENHANCED_TELEMETRY_BETA — set to 1 (gates Claude Code’s span export; both this and CLAUDE_CODE_ENABLE_TELEMETRY must be 1 for traces to flow)OTEL_RESOURCE_ATTRIBUTES — tags with project= and agent= for filtering
Key behavior: Telemetry is only active when the monitoring stack is running at the time the container is created. If the stack isn’t up, Clawker explicitly disables telemetry so agents don’t waste time attempting exports to an unreachable collector. You can force-enable telemetry via agent.env in your .clawker.yaml even when the stack is down, but Claude Code won’t retry failed collector connections.
Start the monitoring stack before starting your agents to ensure telemetry is captured.
Egress Traffic Visibility
When the firewall is enabled, Envoy and CoreDNS access logs flow into OpenSearch alongside agent telemetry, plus a per-decision-point eBPF event stream that closes the bypass-mode forensic gap.
- Envoy access logs (
clawker-envoy) — server.address (host the client asked for), network.peer.address/network.peer.port (post-resolution upstream), network.transport/network.protocol.name/network.protocol.version, tls.established/tls.protocol.version/tls.cipher, action (clawker firewall verdict: allowed/denied), response_code/response_code_details (distinguishes Envoy direct_response from upstream via_upstream), duration breakdown (req_duration_ms/resp_duration_ms/resp_tx_duration_ms/duration_ms), and byte counters. Query action:denied for clawker firewall blocks; response_code:>=400 AND action:allowed for upstream errors.
- CoreDNS query logs (
clawker-coredns) — queried domain, qtype (A/AAAA), rcode (NOERROR / NXDOMAIN), resolution duration. rcode:NXDOMAIN is ambiguous: it covers both DNS-layer firewall blocks (non-allowlisted host) and legitimate misses inside allowed zones (typo, missing record). CoreDNS makes no per-query allow/deny decision — correlate domain against the allowlist, or use the eBPF egress stream below (explicit action:denied) for the authoritative block signal.
- eBPF egress events (
clawker-ebpf-egress) — one record per connect/sendmsg/sock_create decision carrying action (allowed / denied / bypassed), attribution (agent, project, container_id), the 4-tuple, and the resolved domain. The bypass case is the headline: bypassed traffic skips Envoy and CoreDNS entirely but still produces a netlogger record, so bypass windows leave a complete audit trail. See Egress Observability for the record shape and per-attribute reference.
Setup
1. Initialize Configuration
This scaffolds configuration files in your data directory (~/.local/share/clawker/monitor/):
compose.yaml — Docker Compose definition for all services (four long-lived + one-shot bootstrap)otel-config.yaml — OpenTelemetry Collector pipeline configurationprometheus.yaml — Prometheus scrape targets
Use --force to regenerate files if they already exist.
2. Start the Stack
The stack runs in the background. Service URLs are printed on startup:
OpenSearch Dashboards: http://localhost:5601
OpenSearch API: http://localhost:9200
Prometheus: http://localhost:9090
3. Run Agents
Start agents as usual — telemetry flows automatically:
clawker run -it --agent dev @
4. Explore in OpenSearch Dashboards
Open OpenSearch Dashboards at http://localhost:5601. From the splash / welcome screen, under the Analytics panel on the far right, click Clawker to enter the auto-created workspace.
Inside the workspace, the left navbar has an Explore section — click Logs or Metrics to browse. The preconfigured index patterns (claude-code, clawker-cli, clawker-cp, clawker-envoy, clawker-coredns, clawker-ebpf-egress) and the clawker_prometheus direct-query datasource are already wired, so logs and Prometheus metrics are both reachable from inside OSD. Raw Prometheus is also still available at http://localhost:9090 if you prefer it.
Dashboards. Three are preinstalled under the workspace’s Dashboards view:
- Claude Code Cost & Usage — KPI strip over Claude Code sessions, cost, input/output/cache tokens. Sourced from Prometheus counters.
- Claude Code Activity — Claude Code security activity audit: prompts, tooling, code editing, permissions, hooks, MCP, plugins — totals, distributions, and full event tables.
- Clawker Networking — per-source firewall telemetry: three event-stream panels (Envoy access logs, CoreDNS query log, eBPF egress decisions) plus three verdict pies (Envoy
action, CoreDNS rcode, eBPF action). Verdict pies use each source’s own field — values not normalized so each component’s truth stays visible (e.g. CoreDNS denies show as NXDOMAIN, not collapsed into denied). Red slices = denials. Sourced from OS log indices; no Prometheus counters for network telemetry today.
Build your own off the index patterns + Prometheus datasource for anything not covered.
Prometheus label naming: use kind instead of typeClaude Code’s metrics docs reference a type label on a few counters (claude_code_token_usage_tokens_total, claude_code_active_time_seconds_total, claude_code_lines_of_code_count_total). The monitoring stack stores these as kind instead — same values (input/output/cacheRead/cacheCreation, cli/user, added/removed), only the key is renamed.Why: a known bug in the current OpenSearch SQL plugin’s direct-query Prometheus connector causes the OSD Explore “Metrics” UI to error out (missing type id property 'type') on any Prom series that carries a label literally named type. The OTel collector renames the label at ingest so the UI works. The native Prometheus UI at http://localhost:9090 is unaffected either way — type would work there if it existed.When writing your own PromQL or saved searches against these metrics, query with kind= rather than type=.
Bootstrap
The stack is preconfigured every monitor up. A one-shot clawker-opensearch-bootstrap compose service runs after OpenSearch reports healthy, applies:
- Component templates with shared mappings (
@timestamp as date, service.name / ingest_source / project / agent as keyword)
- Index templates with per-source field mappings for each log index plus the SS4O
traces mapping (so dynamic mapping never locks the wrong type at first ingest). The claude-code template carries the full Claude Code log + span field set.
- ISM retention policy (7-day rollover-to-delete, auto-attached to all indices via
ism_template.index_patterns)
- Empty index pre-creation for the log indices so OSD Discover and dashboards don’t error on initial load before the first record arrives
clawker_prometheus direct-query datasource registered via the OpenSearch SQL plugin so OSD can read Prometheus metrics without a federated proxy- OSD
Clawker workspace with features: ["use-case-all"] so the explore-flavor metrics / logs / traces nav groups all mount
- Saved objects imported INTO the Clawker workspace — index patterns for every log index, the
Claude Code Cost & Usage KPI dashboard (sessions, cost, input/output/cache tokens), the Claude Code Activity dashboard (prompts, tooling, code editing, permissions, hooks), and the Clawker Networking dashboard (Envoy / CoreDNS / eBPF event streams + verdict pies). Build additional dashboards off the index patterns + Prometheus datasource as needed.
otel-collector gates on bootstrap completing successfully — it never starts until the cluster is preconfigured. Prometheus starts in parallel; bootstrap depends on Prometheus being up so the clawker_prometheus datasource registration can validate the configured URI. Bootstrap failure surfaces in docker logs clawker-opensearch-bootstrap and leaves the stack half-up by design (so wrong-mapped indices can’t be silently created). The throwaway-stack model means picking up template/policy edits requires monitor down --volumes && monitor up; templates only apply at index creation.
Telemetry Controls
Fine-tune what telemetry is collected via settings.yaml:
monitoring:
telemetry:
log_tool_details: true # Include tool invocation details
log_user_prompts: true # Include user prompts in logs
include_account_uuid: true # Include account identifier
include_session_id: true # Include session identifier
Port Configuration
Override default ports in settings.yaml if they conflict with other services:
monitoring:
otel_collector_port: 4318
otel_grpc_port: 4317
opensearch_port: 9200
opensearch_dashboards_port: 5601
opensearch_heap_mb: 512 # JVM -Xms/-Xmx for the OpenSearch node
prometheus_port: 9090
prometheus_metrics_port: 8889
clawker monitor init --force
clawker monitor down
clawker monitor up
Checking Status
Shows container status (running/stopped) and service URLs for running services.
Teardown
# Stop the stack (preserves data)
clawker monitor down
# Stop and remove all data volumes
clawker monitor down --volumes
--volumes, monitoring data persists across restarts (named volume clawker-opensearch for indices, clawker-prometheus for TSDB). The clawker-net network is preserved for other Clawker services (firewall, agents).