End-user focus

Do we see a stable pattern of use by expected end users?

Line graph of week on week average logins per user per week by role and Provider type

(Total number logins by role) / (Total users in a role)

With local user knowledge, we can look for concering patterns of discontinued use, or of use only by a fraction of expected end users, or patterns of use that are not consistent with point of care and timely data entry. We can investigate and resolve training, local prioritization, IT infrastructure etc. barriers to adoption.

• Source: For each request, the activity logs store the timestamp, session id, and user id, so we should be able to calculate user logins by this

• Could cross-reference user again role or provider_role table to group by role

Is O3 storing user logins? PIH team tracks & how can we leverage

What happens if many inactive users

Histogram of average logins per week per user by role for the last month, 6 months

Logins by hour and week day

Is there regular and prompt (encounter) data entry and by what type of user?

• Week on week average of new encounters, disaggregated by encounter type

• Median gap between encouter date/time and encouter submission.

Identify data timeliness issues that may indicate barriers to the data entry workflow and work with local colleagues to resolve them.

Encounter table (suggestion)

• Patient ID

• Encounter type

• Encounter location

• Provider

• User who entered the data

• Submit timestamp

• Encounter date or timestamp

• Number of non-blank obs

• Incremented count up per patient per encounter (1 = first encounter)

• Incremented count down per patient per encounter (1 is most recent encounter)

Value of this indicator depends on user accounts being accurately categoried to distinguish between data clerks and users who may enter data point of care, or other clinician data entry.

Is it possible to distinguish new encounters vs. updates?  Is the encounter date really the most recent update?  

Registrations

Is there a regular pattern of patient registration and visits that indicates regular use as the system of record for patient registration.

• Weekly new patients registered

• Weekly % of patient registration records modified

• Weekly patients marked as dead

Confirm that activity matches information we have about the current catchment area and clinical capacity.

• Weekly new patients registered could be calculated by looking at registration encounters

• Would need to research further how/if we could track patient registration records modified

• We don’t directly track when a patient is marked as dead, so we might need to go to the logs for this

Patient table with

• Patient ID

• Registration date

• Registration last modified date

• Death date

Clinical workflow focus

Is the check in feature being used to initiate the patient visit workflow?

• Check ins - average week on week

• Check ins with no encounters (or no obs?)

Inconsistent use indicates that there may be other systems (paper, Excel) used to manage patient visits

• Done by looking at check-in encounters in the database

• Encounters are group by visits, so a “check-in with no encounters” could be determine by a visit that has only one encounter, a check-in encounter

Is patient data being entered regularly and completely?

Encounter table

• Weekly encounters, weekly encounters per patient, by type

• Average obs per encounter that are not blank? by encounter type?

We can see if certain forms or form sections seem to be underused.

• Weekly encounters can be determined by looking at the encounter table

• FWIW, obs should never to “blank”, they are only “not there”, so we’d to manually categorize all the potential obs in an encounter (might be tedious, but shouldn’t be difficult)

Is patient data important for continuing care and clinical program being added regularly

Dispensing table

Diagnoses table

Statistics TBD on

• Prescriptions, dispensing

• Diagnoses

• Dispositions

• Labs

Likely averages per week or per visit, per patient

Determine if these features are being used to the level consistent with other system use, if not, this indicates gaps in implementation where other systems may be used (such as registers) as the system of record for these key aspects of the patient medical record

Are clinical programs being used to track patient status

Program table

• New program registrations per week, average, week on week

• Program status updates and program outcome updates, also per week, average, week on week

Determine if clinical program features are being used consistently and matching the scale expected for each program. Inconsistent or low patterns of use indicate that data entry is delayed, likely from a separate system of record.

• New program registrations can be determined just by looking at the patient_program table and the date_created

• Status updates would be harder to track via the DB because it only the most-recent changed date is recorded

• Activity log tracks when a POST is made to a particular program enrollment, but does not track what, if anything, is changed

Data completeness and quality

Is the data entered as completely as expected in order to:

• Indicate full adoption as the system of record for patient data
• Support data use needs

Field-by-field analysis dashboard (like PIH created)

This is specific to individual forms in the EMR.

Shows which clinical elements of the system are most in use or ignored

Review these indicators with the clinical leads to reduce fields/complexity and data entry, data prioritiziation

Examples:

• Determine if forms are entered in duplicate because all data is not available for one data entry session or via a single physical encounter with the patient
• Determine if data is complete enough so that indicators are an accurate representation of patients

• Are patients in a normal age range? (ie. not 200 years old)
• Are there pregnant males? 

Data use focus

Note, this excludes use of data from the reporting DB such as for HIV and MCH.

Are people using the system data exports for reporting?

• Data exports per week or month, by user, user role, and export

Monthly (or less frequent) access to priority exports indicates possible missed potential for timely and internal use for regular monitoring and decision making.

• We might need to add more logging to properly track this…we track GET requests to the reports page, but it doesn’t look like we capture the date ranges (which are in the POST request)

Are people directly querying the db?

• Are SQL queries logged?

• I believe SQL queries are not logged by default, though we could turn this on, but I believe it might be difficult to distinguish direct SQL queries from those the applicant is providing

Are people accessing system data via API?

• Requests per week or month by user and request

• What do we mean by “via the API”? The client app accesses data via the API, so might be hard to distinguish from individual people accessing; though if there’s a particular request we are looking for we could likely track that

Data Quality  → added to completness & quality 

• Are patients in a normal age range? (ie. not 200 years old)
• Are there pregnant males?