Monitoring Stack: Non-technical User Guide

Monitoring Stack: Non-technical User Guide

When you launch OpenMRS with monitoring turned on, you get a dashboard website called Grafana that lets you see, at a glance:

  • Whether OpenMRS is up and answering requests

  • How much CPU and memory the OpenMRS backend, the database and every other part of the system are using

  • What errors and warnings are being logged by any component

You do not need to install any extra software beyond Docker. Everything is included in the reference distribution.

Two ways to run monitoring

There are two options, and you only need one.

Bundled (most people want this). Everything runs on the same machine as OpenMRS, including Grafana with ready-made dashboards. The rest of this guide is about this option.

Send to your own monitoring system. If your organisation already has a monitoring platform that accepts OpenTelemetry (for example Grafana Cloud or a commercial monitoring service), OpenMRS can send its metrics there instead. Nothing is stored on the OpenMRS machine, there are no local dashboards, and logs are not collected. You'll need an address and login details from whoever runs that platform. See the Monitoring Stack: Technical architecture reference page for the setup steps.

Launching the stack

Open a terminal in the folder that contains docker-compose.yml and run:

docker compose -f docker-compose.yml -f docker-compose.monitoring-bundled.yml up

The first time you run this, Docker will download several images and set up storage. This can take a few minutes. After that, starts are quick.

To stop the stack, press Ctrl+C in the terminal, or run:

docker compose -f docker-compose.yml -f docker-compose.monitoring-bundled.yml down

Add -d to the up command (docker compose … up -d) if you want it to run in the background.

Opening Grafana

Once the stack is running, open your browser at:

http://localhost/grafana

Log in with:

  • Username: admin

  • Password: Admin123

The password can be changed by setting the GRAFANA_ADMIN_PASSWORD environment variable before running docker compose up. Please change it in any deployment that is reachable from outside your machine.

The first dashboard you land on is Logs. You can reach the other dashboards from the left sidebar: click Dashboards, then browse the list.

Dashboard tour

There are seven dashboards:

Dashboard

Answers the question

Dashboard

Answers the question

Logs

What is the system saying? Any errors?

JVM Runtime

Is the OpenMRS application itself healthy inside?

Backend Resources

How much of the server is the OpenMRS backend using?

Database Resources

How much of the server is the database using?

All Container Resources

Which part of the system is using the most?

Monitoring Stack Resources

How much are the monitoring tools themselves costing?

Endpoint health check

Can people actually reach OpenMRS, and how fast is it?

Logs

What it shows. Every message that any OpenMRS component prints (the backend, the frontend, the gateway, the database, the monitoring tools, and so on). You can filter by service, by log level (info / warn / error), by a free-text search, and by a time window.

Around the log viewer there are three counters (total logs, warnings, errors), a chart of log volume over time broken down by service and level, and a separate panel that shows only error lines.

image-20261002-120727.png

 

What decisions a manager can make from it.

  • Investigate a user-reported problem. If a clinician reports that "saving a patient failed at 10:35 today", set the time range to that window, filter by service = backend and level = error, and read the actual error messages.

  • Spot a component in trouble. If the log volume chart shows one service producing a sudden burst of warnings or errors, that service needs attention even if users have not complained yet.

  • Decide whether to redeploy. A steady, low background of warnings is normal. A rising rate of errors after a change usually means the change needs to be rolled back or fixed.

JVM Runtime

What it shows. The health of the Java process that runs the OpenMRS backend, seen from the inside. Panels cover:

  • Heap Used % — how full the application's memory is, as a percentage of the maximum it's allowed.

  • Heap Used — the same thing as an absolute amount.

  • Live Threads — how many concurrent tasks the process has running.

  • Classes Loaded — a technical counter that should flatten out after startup.

  • Heap vs Non-Heap Used — memory usage over time.

  • Memory Pools Used — memory broken down into internal categories (Eden, Survivor, Old Gen, Metaspace).

  • Heap After Last GC — how much memory is still in use right after Java has cleaned up. This is the most honest view of how much memory the application really needs.

  • GC Time Rate / GC Collections Rate — how much time the process spends cleaning up unused memory.

  • Threads by State — threads over time, split by what they're doing (running, waiting, blocked).

  • JVM CPU Utilization — CPU used by the backend process.

What decisions a manager can make from it.

  • Heap Used % sitting above about 90% for long periods means the backend is short on memory. Either give it more memory, or ask a developer to look for a memory leak.

  • Heap After Last GC creeping up over hours or days and never coming back down is the clearest sign of a memory leak. Open a ticket.

  • CPU pinned near 100% for long periods means the machine is undersized for the current workload, or something is stuck in a loop. Time to scale up or investigate.

  • Lots of time spent in GC usually goes hand in hand with memory pressure. Users will feel it as pauses and slow pages.

image-20260720-130420.png

Backend Resources

What it shows. The same kind of health information as the JVM dashboard, but from outside the process, at the Docker container level for the OpenMRS backend. Panels cover CPU usage, memory (Working Set, RSS and Cache), network traffic in and out, disk read/write, container uptime, current memory usage, and an "OOM Events (24 h)" counter.

image-20260720-130311.png

What decisions a manager can make from it.

  • Uptime resetting unexpectedly means the backend restarted or crashed. If this keeps happening without anyone deploying, escalate to a developer.

  • OOM Events > 0 means the operating system killed the container because it ran out of memory. This is a strong signal to raise the container's memory limit or fix a memory leak.

  • A rising memory Working Set that never comes down is another leak indicator. Schedule a restart and open a ticket.

  • Sustained high network or disk I/O at the same time as users report slowness means the disk or network is the bottleneck, not the CPU.

Database Resources

What it shows. The same panels as Backend Resources, but for the database container.

image-20261002-120556.png

 

What decisions a manager can make from it.

  • Database CPU high while the backend is calm usually points to heavy queries, such as large reports or searches. Worth asking a developer which ones.

  • Sustained high disk I/O on the database alongside slow pages means storage is the bottleneck. Faster disks or more memory for the database will help more than a bigger backend.

  • OOM Events > 0 or uptime resets on the database are serious. OpenMRS can't work without it, so treat this as urgent.

 

All Container Resources

What it shows. Every part of the system on one screen, with CPU, memory, network and disk broken down by service, plus totals.

image-20261002-120615.png

 

What decisions a manager can make from it.

  • Find the resource hog. When the server feels slow, this is the fastest way to see which component is using the most.

  • Plan capacity. Compare total memory use with what the machine has. If you're regularly using most of it, it's time for a bigger server before something gets killed.

Monitoring Stack Resources

What it shows. How much CPU, memory, network and disk the monitoring tools themselves are using (Grafana, Prometheus, Loki, Alloy, the HTTP prober and cAdvisor).

image-20261002-120632.png

 

What decisions a manager can make from it.

  • Monitoring taking a big share of a small server. If the tools are competing with OpenMRS for memory, ask an administrator to keep logs for a shorter period, or switch to sending data to an external monitoring system instead.

Endpoint health check

What it shows. Whether OpenMRS is reachable from an outside caller, measured by making real web requests to a few addresses (the home page, /openmrs, and the login session check) every 15 seconds.

image-20261002-120659.png

 

Top-of-page tiles show average uptime over the selected time range and current availability, followed by a daily uptime chart and a table of downtimes. Below that, each address gets its own section with status (up/down), response time, HTTP status code, HTTP version, whether HTTPS is used, certificate expiry, and DNS lookup time.

What decisions a manager can make from it.

  • Availability under 100% during business hours means the service dropped requests, even briefly. Check the downtimes table, then look at the Logs dashboard around the same time to see why.

  • Response time climbing over time means the site is getting slower. Look at the JVM, Backend and Database dashboards for the cause.

  • Certificate expiry heading toward zero means the HTTPS certificate is about to expire. Renew it well before then. Two weeks is a sensible buffer.

Alerts

The monitoring stack doesn't ship with automatic alerts at the moment. The dashboards show problems, but only when someone looks.

If you want to be told when something goes wrong, Grafana lets you add alert rules yourself under Alerting → Alert rules → New alert rule, and send them by email, Slack and so on under Alerting → Contact points. Rules you create this way are saved in Grafana's own storage and survive restarts.

Good first alerts to set up:

  • An address on the Endpoint health check dashboard is down for more than a couple of minutes

  • Heap Used % on the JVM Runtime dashboard stays above 90%

  • Any OOM event on the Backend or Database dashboards

  • The HTTPS certificate expires within 14 days

Customizing the stack

All monitoring configuration (dashboards, data collection rules, log settings) lives in the monitoring/ folder of the distro repository. A small helper container called monitoring-init runs at startup, copies those files into shared Docker storage, and then exits. Grafana and the other tools read the files from there when they start.

You don't need to write code to change most things. The workflow is:

  1. Edit the file in monitoring/ on disk (for example, add a new dashboard to monitoring/grafana/dashboards/).

  2. Rebuild and restart:

docker compose -f docker-compose.yml -f docker-compose.monitoring-bundled.yml up -d --build monitoring-init docker compose -f docker-compose.yml -f docker-compose.monitoring-bundled.yml restart grafana prometheus alloy loki blackbox

The first command puts your changed files in place. The second tells the other tools to reload them.

  1. Refresh Grafana in your browser.

Common customisations:

  • Add a new dashboard. Export it from Grafana as JSON (Dashboard settings → JSON model → Save to file), drop the file into monitoring/grafana/dashboards/, and rebuild as above. Files removed from that folder disappear from Grafana on the next restart.

  • Change how long logs are kept. Logs are kept for 14 days by default. The setting is retention_period in monitoring/loki-config.yaml. Longer means more disk space.

  • Collect logs from another container. Nothing to do. Logs are collected from every container on the machine automatically.

  • Collect metrics from another service. This needs a small configuration file under monitoring/alloy/. Ask a developer, or see the Monitoring Stack: Technical architecture reference page.

If something refuses to start after a change, check docker compose logs monitoring-init. A mistake in one of the config files usually stops the helper container, and the other tools wait for it to finish successfully before starting.