Sync Module
- Former user (Deleted)
- Fawwaz Yusran
- Kaweesi Joseph
Overview
Synchronization is needed when you have multiple sites using OpenMRS with separate databases and you want them to copy data to each other. In this situation, one site has to act as the "Parent", and one acts as the "Child". The Parent is always the computer or site that is most central. Multiple child sites send their data up to the parent site for dissemination. Children/parent sites can be set up in a multi hierarchy way. (eg. a parent server can have a parent server). The parent/child servers can be connected by a network continuously, connected occasionally, or not connected at all.
The Sync Module works by keeping track of all changes going through the API into the database. These changes are then replayed on the other server(s). The changes are stripped of all primary key references. Instead, UUIDs are used to compare and reference other data. Because of this, the parent/child servers must be exact duplicates when sync is first started.
This module uses code from the Data Synchronization Project and should be installed on OpenMRS v1.8.3 or greater.
Download and Installation
Install the omod with the Manage Modules admin page
See parent server setup and child server setup pages for more on how to configure the parent and child servers.
Updating The Sync Module to a New Version
The sync module is a Mandatory Module and cannot be stopped or unloaded without stopping OpenMRS. This prevents any data loss for connected parent/child servers from occurring. However, it means that there is no way to remove the module to install a new one in the traditional fashion.
Download the latest version of the omod
Option #1:
Visit the Manage Modules admin page and upload the module with the "Upgrade An Existing Module" form on the right side of the page
Option #2:
Visit the Module Properties admin page (/openmrs/admin/modules/moduleProperties.form) to find where your modules are stored on the server.
Delete the old sync-0.1.omod file
Copy in the new sync-0.2.omod file
Restart openmrs and/or tomcat
Uninstalling the Sync Module
The sync module is a Mandatory Module and cannot be stopped or unloaded without stopping OpenMRS. This prevents any data loss for connected parent/child servers from occurring. However, it means that there is no way to delete or uninstall the module from the web interface.
Set the sync.mandatory module to false.
Visit the Module Properties admin page (/openmrs/admin/modules/moduleProperties.form) to find where your modules are stored on the server filesystem.
Delete the sync-*.omod file
Restart openmrs and/or tomcat
To clean up the database, these statements need to be executed. Be careful as you are also deleting all the sync history and can't resume your work with the Sync Module later on.
drop table sync_server_record;
drop table sync_server_class;
drop table sync_server;
drop table sync_class;
drop table sync_record;
drop table sync_import;
delete from global_property where property like 'sync.%';
Additionally you should stop and remove the Scheduled Tasks "Cleanup Old Sync Records" and "master Data Synchronization" (if the automatic sync via web is configured).
Upgrading OpenMRS when using the Sync Module
When upgrading OpenMRS, occasionally data is modified that is key to sync (a uuid column). This is done outside the API, so care must be taken to update the child sites correctly.
This is the general process you should follow whenever upgrading openmrs
Make sure all sites are sync'd to their parents and no data entry is happening on any parent/child server
Go through the upgrade process on the parent server. (Download+install the new war file, view the webapp and run all database updates)
Immediately visit "Sync --> Upgrade Scripts" on the administration screen
On that form, if upgrading from 1.5, choose 1.5.* and click the Download button (if your old version is not there, no updates are needed)
Run the sql script on each child database (mysql -u -p -e"source upgrade.sql" openmrs)
Go through the upgrade process on each child server (similar to the second step)
WARNINGS!
Direct Database Updates
Once the sync module is installed, all database updates should be done through the openmrs api. Direct sql on the database should not be used because those changes will not cascade to the parent/child servers.
Instead use the groovy module to make calls on the OpenMRS API to change the database.
(TODO: Add examples here)
Conflict Handling/Resolution
The initial version of the sync module provides no means for conflict handling or resolution: it simply operates on a last-in-wins model. The assumption is that patient-level conflicts are rare in practical use, and therefore conflict handling is not a "must-have" feature for a version 1.0 of the system. It is recommended that the Sync module, as currently designed, NOT be used in situations where frequent conflicts are expected (aka concurrent patient editing).
Using Sync with Infopath
It is strongly recommended that you don't depend on sync to propagate changes to your infopath forms.
Infopath XSNs are .cab files that contain a whole bunch of files inside of them. Some of these files have the hard-coded paths of the server that one was working on when modifying the form. The problem is that these hard-coded strings then get propagated to the other servers through sync (when it's working). In the past, we've seen forms from one site being submitted to another site, which caused all kinds of problems.
It is recommended to block synchronization of org.openmrs.module.formentry, and do all XSN manipulation by hand on each server.
Alert and AlertRecipient
These objects are supported in Sync Module Beta 2 + and with OpenMRS v1.5.3+, v.1.6.2+, and 1.7+. Any other version and you will have to "ignore" the class "org.openmrs.notification.AlertRecipient" from both sending and receiving in order to avoid errors.
Compatibility with other Modules
Formentry Module
If using the Formentry Module 5.x and below with the Sync Module you must take care with how the concepts are developed. Formentry still depends on internal concept.concept_id values being the same on all servers. The sync module cannot guarantee that to be the case. So all concept/form development must happen on one and only one server in your sync network.
See full note on Formentry Module wiki page.
History of the Sync Project
The sync project was the brain child of Christian Allen in 2007 while he was working for Partner's in Health. PIH had multiple sites that were not continuously connected by internet across the Rwandan countryside. Transporting paper forms 4-5 hours on a daily or even weekly basis was not working. His vision of a bidirectional asynchronous database synchronization technique lived and developed in a branch or two for several years. In mid 2009 it was clear that sync was not going to be a part of core code going forward. We made changes to core that were incorporated into the OpenMRS 1.5 release to enable sync to be a module. We then ported the code from the branch into the sync module to allow for maximum flexibility and ease of use.
Release Notes
1.1
[SYNC-54] - Add Javascript Refresh to Overview page in Sync Module
[SYNC-65] - Add link to Scheduler in Sync Module
[SYNC-194] - null pointer hit when onFlushDirty is exectued before afterTransactionBegin in interceptor
[SYNC-242] - Fix failing sync unit tests
[SYNC-245] - From History of Changes page, unable to reset a record as a new sync record if any of the child server records are in the "waiting to synchronize" state
[SYNC-247] - Configure sync poms to allow testing against multiple versions of Openmrs
[SYNC-253] - failing unit test against 1.9.0-alpha: shouldAddObsWithEncounter(org.openmrs.module.sync.SyncEncounterTest)
[SYNC-254] - failing unit test against 1.9.0-alpha: shouldMergePatients(org.openmrs.module.sync.SyncPatientTest)
[SYNC-255] - failing unit test against 1.9.0-alpha: shouldNotGenerateSystemIdIfGPNotDefined(org.openmrs.module.sync.SyncUserTest)
[SYNC-272] - History of Changes: Records per page should not set "firstRecordId" if not previously set
[SYNC-274] - When adding classes to exclude/include, trim off any trailing spaces
[SYNC-276] - SyncIngestService should also not update records with state COMMITTED_AND_CONFIRMATION_SENT
[SYNC-278] - Add index to original_uuid column in sync_record table
[SYNC-279] - Index sync_record.timestamp, sync_server_record.server_id & .state
[SYNC-280] - Voided PersonAttribute get deleted when syncing
[SYNC-282] - Problem adding existing concept to an existing concept set
[SYNC-284] - failing unit test against 1.9.1: shouldFindUuidsOnConcepts(org.openmrs.module.sync.SyncORUR01HandlerTest)
[SYNC-287] - Sync module should prevent OpenMRS Core from persisting changes if no sync record is created
[SYNC-302] - Fix failing unit test: shouldDeletePersonAttributeType(org.openmrs.module.sync.SyncOnDeleteTest)
[SYNC-304] - Sync module fails to load on 1.9.3 standalone
[SYNC-306] - If the only changes that occur to an object are within a collection property, sync fails to detect and record this change
[SYNC-307] - Sync fails to save sets and maps of non-openmrs objects if the owning object has no updates
[SYNC-308] - History of changes page does not correctly use configured setting for number of records per page
[SYNC-311] - History page throws ELException on Tomcat7
[SYNC-323] - Improve performance of "lastRecord" and "previousRecord"
[SYNC-324] - Stacktrace from SyncInterceptor when trying to save user properties
[SYNC-325] - Sync Records are lost if multiple transactions run on the same thread
[SYNC-265] - liquibase changeset 2011121-1030k needs special handling in sync context
[SYNC-277] - Create SyncUtil.isFinalState(state) method
[SYNC-288] - Add sync record id to the history of changes page
[SYNC-289] - Show current time on the overview page
[SYNC-293] - Add quick link on history page to "Most recent all committed"
[SYNC-294] - Ensure outgoing sync error details are properly logged and available to the administrator
[SYNC-305] - Merge sprint201303 branch into master
[SYNC-313] - Add ability to test email configuration
1.0.7
1.0.6
SYNC-286 - If you click on the "Maintenance" tab for Sync Administration, the repeat interval on the task to clean the records gets set to sec
1.0.5
SYNC-250 - Sync module should be able to handle properties that are enums
SYNC-257 - Sync module doesn't build against 1.10-SNAPSHOT
SYNC-267 - Sync should explicitly set character encoding to UTF-8 when creating and reading sync transmissions
SYNC-283 - Problem syncing changes to concept names
SYNC-266 - Add unique constraint to sync_import.uuid column
1.0.4
1.0.3
SYNC-280: Voided PersonAttribute gets deleted while syncing
1.0.2
Fixed SyncIngest bug SYNC-276
1.0.1
Fixed saving redirect bug SYNC-268
1.0
Fixed many bugs and added a few more features
Only known sync'ing issue is with formentry xsn objects. See note above.
0.972
Fixed SQL LEVEL ACCESS privilege requirement SYNC-179
Fixed records marked as Do Not Process from holding up the queue SYNC-193
Sync'ing concepts first in the list SYNC-180
Removing the special-cased hard copy of Concept.conceptId and depending solely on uuid SYNC-160
Added a socket timeout setting SYNC-169
Fixed PersonAttribute sync'ing SYNC-177
Fixed hibernate error with new records SYNC-175
Made configuration timeout configurable SYNC-180
Fixed sync'ing of concept answers SYNC-174
0.94
Fixed failing unit tests and flaky logic when creating new patients
Fixed sync'ing of class objects - SYNC-172
0.93
Fixed Sync statistics page causing OOM when there are a lot of sync records - SYNC-165
0.92
Fixed sync'ing of object attributes marked as 'text' SYNC-171
0.91
Fixed patient editing bug - SYNC-170
0.9 (official "RC 1" release)
Fixed syncing of concepts so that it is compatible with metadata sharing module (i.e. concept_ids are preserved) SYNC-160, SYNC-166
Added ability to sync persistent maps (this enables sync-ing of person attributes) - SYNC-59, SYNC-164
Added Sync In Progress indicator - SYNC-23
Better logging of errors for FAILED records - SYNC-58
0.8 (official "beta 2" release)
Exclude certain types (Scheduler, hl7, global property) from sync'ing. Adding package exclusions - #2352
PersonAttirbutes with empty values fixed - #2360
Indeterminate PatientIdentifier errors - #1535
AlertRecipient can be sync'd now - #2233
Cohort members can be sync'd - #2362
Sync setup on 1.5.x was failing - #2389
0.7 (official "beta" release)
Added Antoine's new sync admin functionality and ability to generate/send/receive database backup over the web - #547
Show number of sync incoming sync items - #2014
Fixed receiving of LoginCredential objects - #2236
Made rebuild of xsn conditional on xsn existence - #1440
Fixed display of error translations - #612
Added more description around received items on status page - #602
KNOWN ISSUE: AlertRecipient cannot be received - #2233
0.6.1 (official "alpha" release)
Changed the default logging so that there isn't as much noise in the server log files
0.6
Fixed concept word rebuilding when new concepts are added. This allows new concepts to be searched/found at remote sites after a new concept was created on central
0.5
Added ability to generate/download upgrade scripts between OpenMRS 1.5 & 1.6
Minor Sync API cleanup - archive:TRAC-2029
0.4 (Initial public pre-release)
Fixed saving of non foreign keyed links to primary key columns in voided obs, person attribute and attribute types
Changed all classloader calls to go through the openmrs classloader to enable sync'ing of module objects
0.3
Fixed offline file transfer to work in bidirectional fashion
Fixed cohort sync'ing
0.2
Changed classes to shorter sync name
Added ability for module classes to be sync'ed
Fixed user id generation
Reorganized administration pages
0.1
Initial release
Contains all functionality from sync bidir branch
Resources
For development purposes, see the Active Sync Module tickets