Detailed Description of Project Workflow

Owned by Sri Maurya Kummamuru
Last updated: Sep 06, 2013 by Roger FriedmanVersion comment
10 min read

  1. Dhis2 Server is registered to the our module. In this details such as Name,URL ,Description, User name and Password are registered.

  2. From the Dhis2 server, a a template is imported. This template might in turn contain smaller templates.

  3. This template can be updated whenever the user is about to make a mapping.(this option is provided in the admin page)

  4. The template is broken down based on the smaller templates and a new file is created for each.

  5. The smaller template now has to be associated with a report in OpenMRS module.

  6. The associated OpenMRS module would provide options such as dimensions and indicators which are cohort queries, which can be linked with the Dhis2 options.This is the first mapping that happens and this is done by the administrator.

  7. After the mapping has been completed. Before running the report if the renderer is selected as the  Integration Renderer(Tentative name) and the report is run.

  8. Now a temporary file is created which will have variables such as #var# that will be replaced by the values of the the report generated.The "var" replacements are calculated based on the initial mapping and are associated with the respective results. Which can be sent to the Dhis2 server.

Test case

  1. OpenMRS large dataset is basis

  2. Data will be obs_datetime in [2006/02,2006/06]

  3. Data will be obs of concepts 5356 CURRENT WHO STAGE and 5497 CD4 COUNT.  The answers to concept 5356 are concepts 1204-1207, WHO STAGE 1-4 ADULT.

  4. Monthly indicators:

    1. NUMBER OF PATIENTS OBSERVED WITH WHO STAGE 1-4 (4 indicators), potentially disaggregated by age (0-5,5-14,14-21,21-49,50+) and sex;

    2. NUMBER OF PATIENTS OBSERVED WITH ANY WHO STAGE, sum of indicators in A, potentially disaggregated by age (0-5,5-14,14-21,21-49,50+) and sex;

    3. NUMBER OF PATIENTS WITH A CD4 COUNT, potentially disaggregated by CD4 Count<=250 and CD4Count>250; and

    4. NUMBER OF NEW PATIENTS BY CD4 COUNT, where a new patient is one whose first CD4 count is in the reporting month, disaggregated by CD4 Count<= or >250 and sex and concept 5272 PREGNANCY STATUS whose values are 1055 YES and 1066 NO.

  5. The test data has only one location.  It will be modified on the following basis: new locations will be created 2 REGION A, 3 HOSPITAL X (child of 2), 4 FACILITY Y (child of 2), 5 IN-PATIENT (child of 3), 6 OUTPATIENT (child of 3).  Encounters will be assigned randomly to locations 4-6.  DHIS2 will have org units for REGION A, HOSPITAL X and FACILITY Y.  (See ticket REPORT-288)

What is mapping?

  • A DHIS Option is mapped to a reporting CohortDefinition

  • A DHIS OptionSet is mapped to a reporting Dimension; the dimensions are the filter specifications of a reporting dataset

  • A DHIS CategoryCombo is represented by the filter specification of a reporting dataset

  • A DHIS CategoryOption is mapped to the right side of a reporting dataset Column name

  • A DHIS DataElement is mapped to a reporting CohortDefinition, a reporting Indicator and the left side of a reporting dataset column name

  • All DHIS DataElements using the same CategoryCombo are mapped to the same reporting Cohort Indicator with Dimension DataSet.  DataElements using the (default) DHIS disaggregation are mapped to a Simple Indicator DataSet.

  • DHIS Frequency and Date attributes are mapped to reporting startDate and endDate parameters

  • DHIS Location is mapped to an OpenMRS LocationTag value and to a reporting Location parameter consisting of one or more OpenMRS locations and is used in a CohortQueryDefinition

  • A DHIS ReportTemplate is mapped to a reporting ReportDefinition. The reporting report definition contains a base query which is the Location CohortQueryDefinition and datasets corresponding to each DHIS CategoryCombo. 

Workflow for Locations

  • On the admin location mapping page,

    • (1) page is divided into 3 vertical sections -- the left for the DHIS2 org tree, the center for the orgunit table, the right for the OpenMRS location widget

    • (2) the left side is populated only when the org unit table is empty for that server, instead a button will be there allowing the user to load it ... this should be a parameter in the url to the controller

    • (3) the org unit table will contain only those org units that the user has selected from the DHIS tree.  each org unit is 1:many with the omrs location table -- this means a new link table

    • (4) the middle section will have 2 columns, one for the org unit name and the other for a list of locations for that org.  each will have a delete button or icon next to them to remove the org unit from the table or the location from the association with that org unit

    • (5) the right part will have the omrs org unit widget.  Selecting an org unit there will add it to the association for the currently selected row of the org unit table

    • (6) The DHIS2 tree will be produced from a set of DHIS org unit objects – these will be persisted in XML, only the ones the user selects will be persisted in the DB

  • The set of locations will be passed as a parameter to the report as a filter definition.  This needs more clarification.

  • DHIS org units come from /api/organisationUnits.xml?viewClass=export

Screens Revisited

  1. Server Selection (creation icon, edit icon) – leads to Report Selection or Location Mapping

  2. Report Selection (creation icon, edit icon) – for selected Server – base cohort selection here – bottom table is data elements by categorycombo with option sets – leads to Data Element Mapping or Option Set Mapping or back to Server Selection

  3. Data Element Mapping has table of data elements at top – selected data element can be mapped to a Reporting Cohort Definition – stay here until user selects back to Report Mapping

  4. Option Set Mapping has table of option sets at top – middle table is options for selected option set – selected option can be mapped to a Reporting Cohort Definition – stay here until user selects back to Report Mapping

  5. Location Mapping – for selected Server – leads to Server Selection

  6. Run Report

  7.  

    1. Select Server

    2. Select Report – make sure report is fully defined, make sure that all parameters have been set, include frequency in list of information shown for report

    3. Select Date (compute parameters startDate, endDate from date and period type)

    4. Select Location (as base cohort from list of DHIS org unit cohorts for server)

    5. Run the report using the DHIS2IntegrationRenderer, which takes the output, does the conversion from column names to data value templates and saves the report in Integration/Reports/<server>.<report>.<location>.<start date>.xml

  8. View/Send Report – based on report queue status; categories are scheduled, completed, completed & sent, failed; columns are server, report, start-end, status, actions; actions are view error, view report, send, delete

    1. Select from available reports, choose View or Send

    2. Viewed reports use xslt to render xml

    3. Sent reports are renamed to have .<sent date> added to filename before .xml.

    4. If possible, and if we are using API to send, a portlet will display the results of the send (added, merged, transmission error)

 Permissions

High – CRUD on server, R on others, can view all admin pages

Medium – R on server, CRUD on others, can view all admin pages

Low – whatever is necessary to queue and view a report, can view Run Report page 

Creating XSDs for use with Spring-WS, JAXB, etc.

Retrieve a set of resources from the DHIS API using http://apps.dhis2.org/demo/api/[resource].xml?viewClass=[viewclass]&pageSize=10 (for the master template, use the curl command given elsewhere and open the xml file).  Copy the xml (you must put the cursor at the first character and then do ctrl-shift-end to select, ctrl-A does not work).

In Eclipse, create a new XML file in api/resources named [resource]-[viewClass].xml.   Paste the copied text into it.  In the navigator pane, right-click the file, select Source>Format.  Delete the entire <paging> tag and its contents.

Go through the file and replace < and > in quoted strings with &lt; and &gt; respectively (it may be necessary to make similar substitutions for other XML metacharacters).  Copy the edited file.

Go to http://www.freeformatter.com/xsd-generator.html and paste your XML into the first text box.  Select force new window = true and click Generate XSD.  Either an error message will be shown in this window or a new window will open with the xsd text in it.  Copy the xml as with the first step.

In Eclipse, create a new XML Schema file in api/resources named [resource]-[view].xsd.  Paste the copied text into it so as to replace the entire <schema> tag.  In the navigator pane, right-click the file, select Source>Format.

Workflow for Detecting Changes in Templates and Dimensions

  1.  

    1. Download the master template from the server.  Compare the new file to the old file.  If they are the same, no further action.

    2. For each report template in the file system:

      1. Check the master template to see if the report template exists.  If not, the report has been deleted

      2. From the report template and the master template for the report, create collections of data elements.  If these collections do not contain the same elements, the elements have been changed.

      3. From the report template and the master template for the report, determine the category combo being used for each data element.  If any data element has unmatched category combos, the category combo has been changed.

    3. Build a collection of OptionSet objects and a collection of CategoryCombo objects from the master template.

    4. For each OptionSet object in the collection

      1. Check the OptionSet table to see if the OptionSet exists.  If not, the OptionSet has been added.

      2. For each OptionSet value in the OptionSet object, check the OptionSet table to see if all the OptionSet values exist.  If not, the OptionSet value has been added 

    5. For each OptionSet in the table

      1. Check the OptionSet collection to see if the OptionSet exists.  If not, the OptionSet has been deleted

      2. For each OptionSet value in the OptionSet table, check the OptionSet collection to see if all the OptionSet values exist.  If not, the OptionSet value has been deleted

    6. For each CategoryCombo object in the collection

      1. Check the CategoryCombo table to see if the CategoryCombo exists.  If not, the CategoryCombo has been added.

      2. For each CategoryCombo value in the CategoryCombo object, check the CategoryCombo table to see if all the CategoryCombo values exist.  If not, the CategoryCombo value has been added 

    7. For each CategoryCombo in the table

      1. Check the CategoryCombo collection to see if the CategoryCombo exists.  If not, the CategoryCombo has been deleted

      2. For each CategoryCombo value in the CategoryCombo table, check the CategoryCombo collection to see if all the CategoryCombo values exist.  If not, the CategoryCombo value has been deleted

    8. Actions to be taken:

      1. Report template has been deleted: remove the report template

      2. Report template has been added: no action; report will appear as unmapped in the admin pages

      3. Data element has been added or removed: remove the report template, report will appear as unmapped in the admin pages

      4. Category combo has changed for a data element: remove the report template, report will appear as unmapped in the admin pages

      5. OptionSet has been added: Add new OptionSet to OptionSet table

      6. OptionSet has been deleted: Delete OptionSet from OptionSet table

      7. OptionSet value has been added: Add new OptionSet value to OptionSet table

      8. OptionSet value has been deleted: Delete OptionSet value from OptionSet table

      9. CategoryCombo has been added: Add new CategoryCombo to CategoryCombo table

      10. CategoryCombo has been deleted: Delete CategoryCombo from CategoryCombo table

      11. CategoryCombo value has been added: Add new CategoryCombo value to CategoryCombo table

      12. CategoryCombo value has been deleted: Delete CategoryCombo value from CategoryCombo table

Data Model

 

 

Workflow for Building In-Memory Representation of DB structure from DHIS

The following data structures will be needed:

a.  HashSet<OptionSet> optionSets -- holds all OptionSet objects needed
b.  HashMap<String,OptionSet> optionSetIndex -- maps from option_set_value_id to OptionSet object in optionSets
c.  HashSet<CategoryCombo> catCombos -- holds all CategoryCombo objects needed
d.  HashMap<String,OptionSet> comboOptionLink -- maps from catCombos.Uuid to OptionSet object in optionSets to indicate linkage
e.  HashSet<ReportData> dataElements -- holds all ReportData objects needed
f.  HashMap<String> dataElementIndex -- maps from element_id to ReportData object in dataElements

  1.  

    1. Get all category combo values from the DHIS server:

      [server]/api/categoryOptionCombos.xml?viewClass=detailed&links=false&paging=false
      Example:
      <metaData>
      <categoryOptionCombos>
      <categoryOptionCombo name="(0-11m)" created="2011-12-24T11:24:25.319+0000" lastUpdated="2011-12-24T11:24:25.319+0000" id="S34ULMcHMca">
      <categoryCombo name="Morbidity Cases" created="2011-12-24T11:24:25.203+0000" lastUpdated="2011-12-24T11:24:25.203+0000" id="t3aNCvHsoSn"/>
      <categoryOptions>
      <categoryOption name="0-11m" created="2011-12-24T11:24:24.149+0000" lastUpdated="2011-12-24T11:24:24.149+0000" id="FbLZS3ueWbQ"/>
      </categoryOptions>
      </categoryOptionCombo>
      <categoryOptionCombo name="(0-28d, At PHU)" created="2011-12-24T11:24:25.319+0000" lastUpdated="2011-12-24T11:24:25.319+0000" id="PP09MbybbwK">
      <categoryCombo name="Location and Mortality age group" created="2011-12-24T11:24:25.203+0000" lastUpdated="2011-12-24T11:24:25.203+0000" id="MOWykxm5agr"/>
      <categoryOptions>
      <categoryOption name="0-28d" created="2011-12-24T11:24:24.149+0000" lastUpdated="2013-03-19T19:44:14.046+0000" id="BMi0oxm2mYH"/>
      <categoryOption name="At PHU" created="2011-12-24T11:24:24.149+0000" lastUpdated="2011-12-24T11:24:24.149+0000" id="Fp4gVHbRvEV"/>
      </categoryOptions>
      </categoryOptionCombo>
      </categoryOptionCombos>
      </metadata>

    A.  For each categoryOptionCombo in categoryOptionCombos
        i.  Create a new CategoryCombo object
        ii.  Populate server and updated from [server] and today()
        iii.  Populate category_combo_value_name and category_combo_value_id from categoryOptionCombo:name and categoryOptionCombo:id
        iv.  Populate category_combo_name and category_combo_id from categoryOptionCombo.categoryCombo:name and categoryOptionCombo.categoryCombo:id
        v.  For each categoryOption in categoryOptionCombo.categoryOptions
            a.  If categoryOption:id with option_set_value_id==categoryOption:id exists in OptionSet, use it
            b.  If not, create a new OptionSet and populate option_set_value_name and option_set_value_id from categoryOption:name and categoryOption:id
            c.  Create a new link from the CategoryOptionCombo to the OptionSet

3.  Get all categories from the DHIS server:
      [server]/api/categories.xml?viewClass=export&links=false&paging=false
Example:
<metaData>
  <categories>
     <category name="EPI/nutrition age" created="2011-12-24T11:24:25.155+0000" lastUpdated="2013-05-28T07:32:43.922+0000" id="YNZyaJHiHYq">
       <concept name="default" created="2013-03-04T10:41:07.349+0000" lastUpdated="2013-03-04T10:41:07.349+0000" id="oo43k7FXQ2g"/>
       <dataDimension>true</dataDimension>
       <categoryOptions>
         <categoryOption name="< 12 mths" created="2011-12-24T11:24:24.149+0000" lastUpdated="2011-12-24T11:24:24.149+0000" id="btOyqprQ9e8"/>
         <categoryOption name="12 to 59 mths" created="2011-12-24T11:24:24.149+0000" lastUpdated="2011-12-24T11:24:24.149+0000" id="GEqzEKCHoGA"/>
      </categoryOptions>
     </category>
     <category name="Gender" created="2011-12-24T11:24:25.155+0000" lastUpdated="2013-05-28T07:27:46.728+0000" id="cX5k9anHEHd">
       <concept name="default" created="2013-03-04T10:41:07.349+0000" lastUpdated="2013-03-04T10:41:07.349+0000" id="oo43k7FXQ2g"/>
       <dataDimension>true</dataDimension>
      <categoryOptions>
         <categoryOption name="Female" created="2011-12-24T11:24:24.149+0000" lastUpdated="2011-12-24T11:24:24.149+0000" id="apsOixVZlf1"/>
        <categoryOption name="Male" created="2011-12-24T11:24:24.149+0000" lastUpdated="2011-12-24T11:24:24.149+0000" id="jRbMi0aBjYn"/>
      </categoryOptions>
     </category>
  </categories>
</metadata>

A.  For each category in categories
    i.  For categoryOption in category.categoryOptions
         a.  If OptionSet with OptionSet.option_value_id==categoryOption:id exists, populate OptionSet.option_set_name and Option.option_set_id
from category:name and category:id

4.  Get the master template:
        curl -u user:userpassword -H "Accept: application/dsd+xml" [server]/api/dataSets
Example: see attachment
A.  For each dataElement in dataElements
i.  Create a new ReportData object
ii. Populate [server] and updated from [server] and today()
iii.  Populate element_name, element_code and element_id from dataElement:name, dataElement:code, dateElement:id
B.  For each reportTemplate in reportTemplates
i.  For each dataValueTemplate in dataValueTemplates
a.  Locate ReportData with element_code==dataValue:code.  If report_id is already populated,
make a new ReportData copying element_name, element_code, element_id
b.  If ReportData.report_id is null, populate report_name and report_id from reportTemplate:name and reportTemplate
c.  If ReportData.category_combo_id is null, lookup dataValueTemplate:id as category_combo_value_id in catCombos. 
Populate ReportData.category_combo_name and .category_combo_id from catCombos.category_combo_name and .category_combo_id