The Reference Application ships monitoring as two optional Docker Compose overlays that sit on top of the base docker-compose.yml. You pick one depending on where you want the data to end up.

Both paths share the same core: the OpenTelemetry Java agent in the backend, cAdvisor for container metrics, and Alloy as the single collector that everything flows through. The only real difference is what Alloy does with the data at the end.

If you knew the earlier setup, these are the big shifts:

• docker-compose.grafana.yml is now docker-compose.monitoring-bundled.yml, and there is a second overlay, docker-compose.monitoring-otlp.yml.

• JVM metrics come from the OpenTelemetry Java agent (bundled in the openmrs-core base image), not the JMX Prometheus exporter. The jar download, jmx_config.yml and backend port 9404 are gone.

• Prometheus no longer scrapes the backend or cAdvisor. Alloy pushes those metrics into Prometheus via remote-write. prometheus.yml only holds the blackbox job now.

• cAdvisor is pinned to v0.55.1, scraped by Alloy, and filtered down to an allowlist of metrics.

• The single config.alloy is split into several files under monitoring/alloy/, and monitoring-init picks which ones to install based on MONITORING_MODE.

• Grafana datasource UIDs are pinned to Prometheus and Loki (capitalised). Four new dashboards were added.

• Grafana alert rules are no longer provisioned by monitoring-init.

The agent is downloaded and checksum-verified in the openmrs-core base image, so nothing extra is fetched at runtime. Both overlays add the same environment block to the backend service:

Because the agent auto-instruments the whole web app, it exports HTTP server and JDBC client metrics on top of the jvm.* family. The JVM dashboard only plots JVM metrics, but everything else is still queryable.

If you run your own collector and want to skip Alloy entirely for backend metrics, point OTEL_EXPORTER_OTLP_ENDPOINT at it.

There are two pipelines, metrics and logs. Alloy sits in the middle of both.

Metrics pipeline

Open

Backend metrics. openmrs.alloy runs an otelcol.receiver.otlp listening on 0.0.0.0:4317 (gRPC) and 0.0.0.0:4318 (HTTP). Received metrics go through otelcol.processor.batch, then otelcol.exporter.prometheus (defined in prometheus.alloy), which translates OTel names to Prometheus names and hands them to prometheus.remote_write.local, pointed at http://prometheus:9090/api/v1/write.

OTLP resource attributes become Prometheus labels like this:

• service.name becomes job

• service.namespace, if set, prefixes it as job="<namespace>/<name>"

• service.instance.id, if set, becomes instance

• every other resource attribute lands on the target_info metric, reachable with a group_left join on (job, instance)

The stack only sets OTEL_SERVICE_NAME, so the JVM dashboard filters on job="openmrs-backend". If you run more than one backend replica, give each a unique service.instance.id through OTEL_RESOURCE_ATTRIBUTES. Otherwise every replica writes to the same series and Prometheus rejects the duplicates.

Metric names follow OpenTelemetry semantic conventions, translated by Alloy (for example jvm.memory.used becomes jvm_memory_used_bytes). Upgrading the agent in openmrs-core can rename series and break dashboard queries.

Container metrics. cadvisor-bundled.alloy scrapes cadvisor:8080 every 15 seconds and runs the result through prometheus.relabel "cadvisor" before remote-writing it to Prometheus:

1. Keeps an allowlist of metrics: up, machine_cpu_cores, machine_memory_bytes, and the container_* CPU, throttling, memory (working set, usage, RSS, cache, limit), CPU quota/period, network bytes/errors, fs reads/writes/usage/limit, oom_events_total, start_time_seconds and last_seen series. Everything else is dropped at the edge.

2. Copies the Compose service label (container_label_com_docker_compose_service) into a clean service_name label. Every container dashboard filters on this.

3. Drops all remaining container_label_* labels.

honor_timestamps = false is set on purpose. cAdvisor's machine_* metrics reuse a machine-info timestamp that only moves every few minutes, so honouring it makes those series go stale between refreshes. Stamping at scrape time keeps them fresh.

cAdvisor itself runs privileged with --docker_only=true, --store_container_labels=false, a whitelist of just the Compose service and project labels, --enable_metrics=cpu,memory,network,disk,diskIO,oom_event and a 15s housekeeping interval. It mounts the containerd socket in addition to the Docker socket. Without it, cAdvisor falls back to raw cgroups and emits no container names or labels, which means no service_name.

HTTP probes. This is the only thing Prometheus still scrapes itself. The blackbox-http-prod job in prometheus.yml asks blackbox to probe http://gateway:80/, http://gateway:80/openmrs and http://gateway:80/openmrs/ws/rest/v1/session. The relabel rules rewrite the target to blackbox:9115?target=<URL>&module=http_2xx_ignore_ssl and keep the URL as the instance label so the endpoint dashboard can group by it.

Logs pipeline

This now lives in monitoring/alloy/logs.alloy and works the same as before. Alloy mounts the Docker socket and uses discovery.docker to find every running container. discovery.relabel.docker_logs attaches container_name, service_name (from the Compose service label) and a static job = docker. loki.source.docker tails each container and forwards lines through loki.process.docker_process, which:

1. Drops noisy Loki self-reported messages (query-scheduler errors, ingester health-check errors, "entry too far behind" errors). See the stage.drop regex.

2. Joins multi-line entries (stage.multiline) using a first-line pattern that matches a leading date or a leading level keyword, so Java stack traces stay attached to their log line.

3. Extracts a level label (stage.regex + stage.labels).

4. Normalises the level to warn, info or error (stage.match blocks) so the log dashboard's filter has a small, predictable set of values.

Lines are pushed to http://loki:3100/loki/api/v1/push. Loki keeps 14 days (limits_config.retention_period: 336h) with the compactor's retention enabled, on the local filesystem at /loki/data inside a named volume.

Open

https://drawmaid.com/?utm_source=chatgpt.com#d=XZHNasMwEITvfopFpwTi_NCWmlACpYeWEkgovoke1vImDlUls5ad5u2LZSl1qssO2tGs9Omg7VlVyA62HwkAQIHqi0wpRRBPBS82k11OGt6xQ8AjGTcVn96ssOxOjWUpolpny2w5nFHWODwZ4qa3e3_TFkfGugLU2l6keO5LyOoXk-qk2OXbfS_p1BH7rPX93eoRFtDXbORvFGNNUgz1JkhjQVqKIEYt60jHOaSvc0aOAp2qegBOVaNt-qktO2IpogpNMuXwvLZuHBN-SxEV3FB8ZTygQXjRti1n0JEpLc98z54NKKs1KWf5yiuchjQFT2Xxlud7qNumEpCmG8_r5iMgnUOgUQJ1xBdYPTQC5ukmwEoiZx_g35j8oQypnpnXkdY_t1d-L7JIxoiuF4ZJ5VwNluHItZoOt45wfgE

Nothing is stored locally. There is no Prometheus, Grafana, Loki or blackbox, and logs are not shipped.

Alloy loads three files in this mode:

• openmrs.alloy — the same OTLP receiver on 4317/4318 as the bundled path.

• cadvisor.alloy (installed from cadvisor-otlp.alloy) — scrapes cAdvisor, then converts the Prometheus samples into OpenTelemetry with otelcol.receiver.prometheus.

• export.alloy (installed from external-otlp-http.alloy or external-otlp-grpc.alloy) — defines the otelcol.processor.batch "metrics" that both inputs feed into, and the exporter (otelcol.exporter.otlphttp or otelcol.exporter.otlp) that sends to the upstream.

The exporter reads its settings from the environment: endpoint from ALLOY_OTLP_ENDPOINT, headers parsed as JSON from ALLOY_OTLP_HEADERS (defaults to {}), and tls.insecure from ALLOY_OTLP_INSECURE.

The cAdvisor relabelling has two extra rules compared with the bundled path, because the data is leaving the host and often lands in a shared backend:

• Project filter. Only series whose container_label_com_docker_compose_project matches ALLOY_COMPOSE_PROJECT are kept, plus the machine_* series. The overlay sets ALLOY_COMPOSE_PROJECT from ${COMPOSE_PROJECT_NAME}, so other Compose projects on the same host don't leak into your metrics.

• Label trimming. The id and image labels are dropped to cut cardinality.

The scrape also sets honor_metadata = true, which preserves Prometheus metric types (counter, gauge) through the conversion to OpenTelemetry. This option is experimental in Alloy, so in OTLP mode Alloy runs with --stability.level=experimental. The bundled path writes straight to Prometheus, doesn't need it, and runs at the default stability level.

monitoring-init keeps monitoring config version-controlled in the repo without baking it into vendor images. Each overlay sets MONITORING_MODE, and the entrypoint installs only what that mode needs.

The image is published as openmrs/openmrs-reference-application-3-monitoring-init by .github/workflows/build-monitoring-init.yml on pushes to main that touch monitoring/**. Both overlays also declare build: ./monitoring, so --build uses your local source.

At startup, entrypoint.sh runs once:

Alloy config (both modes)

1. Deletes every *.alloy file in /etc/alloy (volume alloy-config), so files from a previous mode or version don't linger.

2. Copies the files for the selected mode:

Alloy is started with the directory /etc/alloy as its config path, so it loads every file there as one configuration. That's why components can reference each other across files, for example cadvisor-bundled.alloy forwarding to prometheus.remote_write.local, which is defined in prometheus.alloy.

Validation (OTLP mode only). The init container exits non-zero, and so blocks Alloy from starting, if:

• ALLOY_OTLP_ENDPOINT is empty.

• ALLOY_OTLP_PROTOCOL is anything other than http or grpc.

• With http: the endpoint doesn't start with http:// or https://. It also warns if the endpoint ends in /v1/metrics, because the exporter appends that itself.

• With grpc: the endpoint (after stripping any scheme) contains a path or isn't host:port.

An unknown MONITORING_MODE also fails the container.

Everything else (bundled mode only)

3. Copies loki-config.yaml → /etc/loki/local-config.yaml (volume loki-config).

4. Chowns /loki/data to 10001:10001 (the Loki container's uid).

5. Copies grafana-datasources.yaml → /etc/grafana/provisioning/datasources/datasources.yaml (volume grafana-provisioning).

6. Copies grafana-dashboards.yaml → /etc/grafana/provisioning/dashboards/dashboards.yaml, clears dashboards/json/*.json, then copies every JSON in monitoring/grafana/dashboards/ there.

7. Copies prometheus/prometheus.yml → /etc/prometheus/prometheus.yml (volume prometheus-config).

8. Copies blackbox.yml → /config/blackbox.yml (volume blackbox-config).

In OTLP mode only the alloy-config volume is mounted.

Every service that reads these files has depends_on: { monitoring-init: { condition: service_completed_successfully } }, so config is in place before it starts. Grafana additionally waits for Loki to start.

Because the targets are named volumes, a bare restart does not pick up config changes. Re-run the init container (docker compose -f docker-compose.yml -f docker-compose.monitoring-bundled.yml up -d --build monitoring-init) and then restart the services that read the changed files, or start clean with down -v.

Grafana reads two provisioning trees from /etc/grafana/provisioning:

• datasources/datasources.yaml — declares Loki (uid: Loki) and Prometheus (uid: Prometheus). The UIDs are pinned because dashboards reference datasources by UID; without pinning Grafana generates one and the references break. The file also has a deleteDatasources block that removes both datasources on every start before recreating them. That's needed because Grafana can't change the UID of a datasource it has already provisioned, and these UIDs were pinned after the fact. Don't change the UIDs.

• dashboards/dashboards.yaml — a single "Default" provider that loads every JSON file under dashboards/json/. Dropping a new JSON file there is enough to publish it.

No alert rules are provisioned at the moment.

Grafana runs with GF_SERVER_SERVE_FROM_SUB_PATH=true and a root URL ending in /grafana/, so it works behind the gateway. The admin password comes from GRAFANA_ADMIN_PASSWORD (default Admin123). The Logs dashboard is set as the home dashboard.

Provisioned dashboards