Synchronization Project Plan

Owned by Former user (Deleted)
Last updated: Aug 23, 2010 by Michael DowneyVersion comment
4 min read

Project Plan [archive:DRAFT]

The following are the larger pieces of the project that need to be completed in order for synchronization to work properly:

[edit]

Change Journaling

  • Resources: Julie, Christian

  • Description: In order to safely guarantee correct synchronization in most cases, we need to implement the idea of "history" for entities in the system. Since almost every entity is synchronized, we will ultimately need this for almost every item in the system. History also gives the benefit of retracing anything that has happened to any object in the system, and could possibly improved the performance, since auditing information will be moved from primary tables to "history tables".

  • Breakdown

    • Catch changes in Hibernate Interceptor (done)

    • Inspect objects and serialize them using reflection (in progress)

      • Issue with Patient-Person-User having 2 GUIDs - resolved, Hibernate is force syncing these values

    • Create folder based on timestamp for changes (done)

    • Save XML of changes in files, in those folders (done)

  • For v2:

    • Grab objects in Hibernate Interceptor at session level (so changes can be packaged together)

    • Hook up so that saves go to DB instead of file system

[edit]

Sync Request, Client

  • Resources: Maros, Anders

  • Description: One of the first steps in the process is to figure out the algorithm for locating data that needs to be synched and packaging it together. Overlaps with synchronization.

  • Current Progress:

    • Maros building up code packages that are involved.

      • initial code checked into api/org.openmrs.synchronization

    • We think that a call such as getPatientsToSync() belongs in the service layer

  • Tasks for Next 7-14 days:

    • 1st finish up the sync abstractions:

      • clean up SyncSource interface and sample implementation (i.e. the NoHistorySync)

      • Add sync controller: add class(es) that would operate on syncsource(s) -- tentatively ala sync4j this may be a combination of SyncStrategy (something that will bind a specific sync source(s) with specific conflict resolution method) and SyncEngine -- the primary interface to be called by 'clients' (i.e. rest of openmrs api)

      • Add some conflict resolution placeholder: ConflictResolver; that given syncItem and access to target sync source will implement logic to detect and resolve conflict

      • Add container(s) for holding changeset (i.e. something more than just List<SyncItem>)

[edit]

Package and Transport

  • Resources: Julie, Anders

  • Description: We need a way to get object data that needs to be sync'ed into XML and saved to disk, ready to transfer via disk or 'net.

  • Breakdown

    • Package all relevant history objects into single, transportable object (could contain error/status and other metadata)

      • Implement a selializeMyself() method

    • Post data to an HTTP page

      • Allow posting raw data

      • Show form by default (for testing)

      • Send structured reply (same as sneakerNet reply)

    • Allow upload from file

      • Write response back to disk (xml)

  • For v2:

    • Formalize - maybe use REST so HTTP gives back a structured answer

  • Notes

    • We could hold off on worrying about gzipping so we can get prototype done faster

[edit]

Sync Request, Server

  • Resources: Maros, Christian, Julie

  • Description: These are the steps the server takes when it receives data, in order to insert into its own records

  • Breakdown:

    • Deserialize SyncTransmission into an object in memory

    • Insert journal object data into primary tables

    • Log and send reply

[edit]

GUIDs and PKs

  • Resources: Maros, Christian

  • Description: In order for sync to work well, objects that are sync'ed between machines need a universal identifier that is NOT the primary key of the object.

  • Breakdown

    • Add GUID column to every table in DB (done: openmrs_1.1.0-add-guids.sql)

    • Update services to allow load/fetch by guid

    • Convert all tables to having a single PK

    • Update DAO code around single PK changes

  • List of Tables/objects being updated:

    • Patient

    • PatientIdentifier

    • PatientIdentifierType

    • PatientProgram

    • PatientState

    • Person

    • PersonAddress

    • PersonAttribute

    • PersonAttributeType

    • PersonName

  • Notes:

    • Updated based on conversation from 07/20 call: we will use GUIDs for uniquely identifying records for sync purposes. We will keep existing keys for ref integrity et al., but use GUIDS to match records when sync-ing.

    • On related note, we also discussed a need to have simple single key PK for *every* table, this is because a) it simplifies DB schema where we need to keep track of multiple table PKs (as in possibly single history table) b) unrelated to sync, Ruby apparently has the same requirement c) it is a good ER modeling practice

    • Tasks:

      • 1. Add guid( char(36) column) to every table that needs to be sync-ed:

        • Description: Create schema upgrade script to add columns and as a part of initial upgrade also come up with a way to initially populate values (i.e in Java this is easy but mysql up to 5.5xx does not have guid function

        • Duration: 5 days

        • Dependencies: <none>

      • 2. Convert all tables to single PKs: proposed to use 'unsigned big int'

        • Description: decide on the data type (i.e. do we want to/need to change the existing integers to unsigned big int? for sake of consistency?)

        • 2.1 drive the decision on type to use - investigate and document pros/cons:

          • Duration: 3 days

          • Dependencies: <none>

        • 2.2 schema upgrade script

          • Duration: 5 days

          • Dependencies: 2.1

        • 2.3 code updates

          • Duration: 5+ days - unsure about the extent of impact if changing existing types

          • Dependencies: 2.1

    • Other things of interest:

      • Christian confirmed that Ben and the OCC project will not need GUIDs, so we are free to come up with our own method of making them

[edit]

Conflict resolution (optional for phase I)

  • Lead: Christian, Maros, Julie

  • Description: The algorithm for detecting conflicts and the UI to let the user resolve them.

  • Current Progress: This be the last piece to start with since it depends on History/auditing