Community KPIs and Activity Analytics

Owned by Darius Jazayeri
Last updated: Jan 30, 2018 by Daniel Futerman
3 min read

 

Primary mentor

N/A

Backup mentor

N/A

Assigned to

N/A

Abstract

 

As an open-source community we are driven and sustained by the contributions that people make every day. We always encourage people to be more involved, and exhibit various behaviors, and we have ideas about what makes our community successful. But we do a really bad job of measuring our overall community health, and the behaviors of individual community members, and thus we don't really know whether things are going well or poorly, except anecdotally.

The goal of this project is to gather the bits of community activity that are taking place every day on the web everyday, consolidate these into a single database for analytics, and generate KPIs that can help drive our organizational behavior.

 

Project Champions

  • @Darius Jazayeri

  • @Theresa Cullen

  • @Burke Mamlin

Objectives

Business Analysis:

  • Define 5 KPIs that we would like to use to measure OpenMRS's community health

  • Define a list of activities that contributors should get "points" for carrying out

  • Mock up a public "OpenMRS Community Status" dashboard 

Technical:

  • Create a single database where we can store activity

  • Connectors to pull activity data from GitHub, JIRA, IRC, etc. 

  • Database to store our calculated KPI scores, and contributor scores 

  • Create the "OpenMRS Community Status" Dashboard

Brainstormed Ideas

OpenMRS Community KPI

  • total monthly number of code commits (year on year)

  • # of people scoring "good" or above on an individual contribution metric

    • review the specific people who were "good" for the prior two months but not this month, for potential individual followup

  • % of dev activity that is happening in priority areas (priority = identified by the Implementation team as priorities? or is part of an active sprint?)

  • % of posts to "Ask OpenMRS" that receive a reply within 3 days; % of posts to "Ask OpenMRS" that receive an accepted answer within 7 days

  • # of tweets with #openmrs in the last week

  • # of people (or people-days) who participated in an IRC scrum in the last week

 

Individual Contributor Scoring

  • said something on IRC between !scrumon and !scrumoff (+10)

  • said something on IRC (+1)

  • Any Talk post (+1)

  • Reply to the Ask OpenMRS category on Talk (+5)

  • Reply to Ask OpenMRS with a link to the wiki (+10)

  • commented on an open github PR (+1)

  • merged someone else's PR (+10)

  • had your PR merged (+10)

  • contributed a translation on transifex (+1)

  • github commit to any /openmrs repo (+1)

  • github commit to any repo with an active sprint (+5)

  • created a new wiki page (+5)

  • edited a wiki page (+5)

  • openmrs twitter interactions (+0.5 per tweet)

  • commented on a wiki page (+0.5)

  • created a site on openmrs atlas (+5)

  • added unit tests (+2 per commit)

  • pull request broke the build on CI ( -2)

  • pull request merged with no CI failure (+1)

  • creates an openmrs id (+5)

  • forks an OpenMRS repository (+5)

  • attends an OpenMRS Design call (+5)

  • writes notes for an OpenMRS meeting in notes.openmrs.org (+2)

  • # of reported JIRA issues (claimed and closed issues are probably better reflected in created and merged pull requests)

  • # of conducted reviews (# of all comments in github). At the moment we don't have a way to accredit reviewers, thus only those who commit or merge get the credit. It would be nice to track reviewers in JIRA issues in a dedicated field and a state in workflow

  • # of comments in JIRA

 

Tech Notes

These days it is really easy to consolidate and analyze data, and there are great modern technologies for doing this. I did a quick spike of this idea using ElasticSearch, NodeJS, and AngularJS, and we'll likely use similar technologies for this project (unless we find an off-the-shelf suitable solution to this problem).

Broadly, we want:

  • a NoSQL DB to store our "community activity", used in an append-only style

  • a few feed readers +/- an IRC bot to harvest data from github, JIRA, Talk, etc.

  • a server-side app with scheduled processes to calculate and store our metrics (should use js or groovy for scripting the calculations)

  • a small html+js app that we run at some url like om.rs/communitystatus which publicly displays our KPI trends and highlights top contributors (recent, and all-time)

See my spike at: https://github.com/djazayeri/openmrs-contrib-analyzecontributions

It's possible there is an off-the-shelf solution for what we want to do. We should look into: