End-user focus

Do we see a stable pattern of use by expected end users?

Line graph of week on week average logins per user per week by role and Provider type

(Total number logins by role) / (Total users in a role)

With local user knowledge, we can look for concering patterns of discontinued use, or of use only by a fraction of expected end users, or patterns of use that are not consistent with point of care and timely data entry. We can investigate and resolve training, local prioritization, IT infrastructure etc. barriers to adoption.

• Source: For each request, the activity logs store the timestamp, session id, and user id, so we should be able to calculate user logins by this

• Could cross-reference user again role or provider_role table to group by role

Is O3 storing user logins? PIH team tracks & how can we leverage

What happens if many inactive users

Histogram of average logins per week per user by role for the last month, 6 months

Logins by hour and week day

Is there regular and prompt (encounter) data entry and by what type of user?

• Week on week average of new encounters, disaggregated by encounter type

• Median gap between encouter date/time and encouter submission.

New encounter forms 

Identify data timeliness issues that may indicate barriers to the data entry workflow and work with local colleagues to resolve them.

Encounter table (suggestion)

• Patient ID

• Encounter type

• Encounter location

• Provider

• User who entered the data

• Submit timestamp

• Encounter date or timestamp

• Number of non-blank obs

• Incremented count up per patient per encounter (1 = first encounter)

• Incremented count down per patient per encounter (1 is most recent encounter)

Value of this indicator depends on user accounts being accurately categoried to distinguish between data clerks and users who may enter data point of care, or other clinician data entry.

Is it possible to distinguish new encounters vs. updates?  Is the encounter date really the most recent update?  

Registrations

Is there a regular pattern of patient registration and visits that indicates regular use as the system of record for patient registration.

• Weekly new patients registered

• Weekly % of patient registration records modified

• Weekly patients marked as dead

Confirm that activity matches information we have about the current catchment area and clinical capacity.

• Weekly new patients registered could be calculated by looking at registration encounters

• Would need to research further how/if we could track patient registration records modified

• We don’t directly track when a patient is marked as dead, so we might need to go to the logs for this

Patient table with

• Patient ID

• Registration date

• Registration last modified date

• Death date

Can we tell whether a registration has been updated?

Patient Disposition

Where is it measured: program outcome, disposition, marked outcome in system

Clinical workflow focus

Is the check in feature being used to initiate the patient visit workflow?

• Check ins - average week on week

• Check ins with no encounters (or no obs?)

Inconsistent use indicates that there may be other systems (paper, Excel) used to manage patient visits

• Done by looking at check-in encounters in the database

• Encounters are group by visits, so a “check-in with no encounters” could be determine by a visit that has only one encounter, a check-in encounter

Is patient data being entered regularly and completely?

Encounter table

• Weekly encounters, weekly encounters per patient, by type

• Average obs per encounter that are not blank? by encounter type?

We can see if certain forms or form sections seem to be underused.

• Weekly encounters can be determined by looking at the encounter table

• FWIW, obs should never to “blank”, they are only “not there”, so we’d to manually categorize all the potential obs in an encounter (might be tedious, but shouldn’t be difficult)

Is patient data important for continuing care and clinical program being added regularly

Dispensing table

Diagnoses table

Statistics TBD on

• Prescriptions, dispensing

• Diagnoses

• Dispositions

• Labs

Likely averages per week or per visit, per patient

Determine if these features are being used to the level consistent with other system use, if not, this indicates gaps in implementation where other systems may be used (such as registers) as the system of record for these key aspects of the patient medical record

Are clinical programs being used to track patient status

Program table

• New program registrations per week, average, week on week

• Program status updates and program outcome updates, also per week, average, week on week

Determine if clinical program features are being used consistently and matching the scale expected for each program. Inconsistent or low patterns of use indicate that data entry is delayed, likely from a separate system of record.

• New program registrations can be determined just by looking at the patient_program table and the date_created

• Status updates would be harder to track via the DB because it only the most-recent changed date is recorded

• Activity log tracks when a POST is made to a particular program enrollment, but does not track what, if anything, is changed

Data completeness and quality

Is the data entered as completely as expected in order to:

• Indicate full adoption as the system of record for patient data

• Support data use needs

Field-by-field analysis dashboard (like PIH created)

This is specific to individual forms in the EMR.

% blank per form field (observation) per encounter type or form

Shows which clinical elements of the system are most in use or ignored

Review these indicators with the clinical leads to reduce fields/complexity and data entry, data prioritiziation

Examples:

• Determine if forms are entered in duplicate because all data is not available for one data entry session or via a single physical encounter with the patient

• Determine if data is complete enough so that indicators are an accurate representation of patients

This comes directly from encounter/observation data from the forms being monitored for completeness.

See Field-by-field analysis done by PIH in PowerBI 

Data entry values

Histogram about the frequencies of multiple choice or specific values

Are defulat values overrepresneted 

Considering required and coded fields, and checking to see if default values are overrepresented.

Is data in plausible range?

• Are patients in a normal age range? (ie. not 200 years old)

• Are there pregnant males? 

Data use focus

Note, this excludes use of data from the reporting DB such as for HIV and MCH.

Are people using the system data exports for reporting?

• Data exports per week or month, by user, user role, and export

Monthly (or less frequent) access to priority exports indicates possible missed potential for timely and internal use for regular monitoring and decision making.

• We might need to add more logging to properly track this…we track GET requests to the reports page, but it doesn’t look like we capture the date ranges (which are in the POST request)

Are people directly querying the db?

• Are SQL queries logged?

• I believe SQL queries are not logged by default, though we could turn this on, but I believe it might be difficult to distinguish direct SQL queries from those the applicant is providing

Are people accessing system data via API?

• Requests per week or month by user and request

• What do we mean by “via the API”? The client app accesses data via the API, so might be hard to distinguish from individual people accessing; though if there’s a particular request we are looking for we could likely track that

Data Quality  → added to completness & quality 

Is data in plausible range?

• Are patients in a normal age range? (ie. not 200 years old)

• Are there pregnant males? 

What do we want to know?

KPIs & analytics

Calculation

What can we do with this information?

Data Collection Requirements & Source

Open Questions

What is the most common use of the system?

Top 3/5/10 next “clicks” or actions, disaggregated by user role.

Determine what most people are actually doing in the system - searching for a patient? Adding a form?

• Source: Has to be from logs.

Remove to future ideas, data not available.

Is there frequent access to view patient summaries during clinic hours by clinicians?

What do clinicians look at (click on) on patient summaries?

• Average unique patient summary views per user per week by role, also by patient clincal program

• What are the “next click” when somene is on a patient summary  disaggregated by role.

Understand if the system is being used to inform clinicians about patient status, possibly at point of care

• Source: Activity logs track each load of patient dashboard (“/mirebalais/coreapps/clinicianfacing/patient.page”) and included query parameters could be used to determine the patient (patientId) or whether this is the generic dashboard or a program-specific dashboard (dashboard parameter)

Remove to future ideas and request technical analysis for capturing patient summary views.