History

Synchronization

Overview

Sync Algorithm
Conversion to polling only - Issues
Hashmap synchronization - Issues

Local changes are propagated immediatelly to the server, through a set of agents (actors). Remote changes are detected through polling.

Local Change Propagation

The client immediatelly propagates changes to the local file system to the server through a set of agents:

  • The FileAgent detects creation, deletion, modification and move of local files. For each modified file it updates the local state and checksum before posting a message to the WorkflowAgent.
    It will also generate move events for every child object of a moved dictionary.
    Folders that are permanently deleted will generate a delete event for each of their children. Folders that are moved to the recycle bin generate only a single event. WorkflowAgent is responsible for generating delete events for the child objects of this folder.
  • The WorkflowAgent determines the action to take for each file event: skip, upload,move or delete. For each action, it posts a message to the NetworkAgent.
    The agent also restarts processing of any interrupted uploads or downloads when the client starts by checking the transfer status in the local state database.
    Folders that are moved to the Trash Can will generate a single delete event for the root folder. The WorkflowAgent retrieves the folder's child objects from the state database and generate delete events for them.
  • The NetworkAgent communicates with the Pithos server. It performs uploads/deletions/moves in response to messages.
    Deletion messages are redirected to the DeleteAgent and processing of network messages stops until all deletion messages have been processed.

Mapping of File Events to Actions

http://yuml.me/diagram/scruffy;/activity/(start)->[FileAgent]-Events>[WorkflowAgent]-Actions><d1>Normal->[NetworkAgent]->|a|->(end),<d1>Delete->[DeleteAgent]->|a%7C

Polling

Server changes are detected by the Poll Agent. The Poll Agent performs a periodic poll asynchronously to retrieve all server objects. This listing is compared to the last cached listing to detect changes.
When a poll finishes it schedules the next poll in a recursive manner.

Polling pauses while NetworkAgent has messages to process. Specifically,

  1. The Poll Agent polls Pithos Server
  2. The results are compared to the results of last poll.
    • For all moved objects, a check is made that the corresponding local file already exists. If it does, a RenameLocal message is posted to the Network agent. Otherwise, a Download message is posted
    • For all new objects ( Current.Except(Old) ), a Download message is posted to the Network Agent
    • For all missing objects ( Old.Except(Current) ), the file is deleted.
    • For all Common objects ( Old.Intersect(Current) ) with differing, a MustSynch operation is posted.
  3. The MustSynch handler compares the hashes of the local and remote objects and decides whether to do a download or mark a conflict.
    • If the local and remote hashes match, the file is skipped
    • Otherwise, we need to detect conflicts by checking the previous server hash
      • If the local hash matches the previous hash, the file has changed on the server and must be downloaded
      • If the local file is different from the server hash, there is a conflict.

Change Detection Rules

  • Moved objects are those that detected in both the Previous and Current listings by matching their UUIDs but have different names.
  • Changed objects are those that are detected both in the Previous and Current listings by matching their URLs but have different hashes.
  • Created objects are those that are detected only in the Current listings, using their URLs as keys
  • Deleted objects are those that are detected only in the Previous listings, using their URLs as key

Conflict Prevention

To prevent conflicts when uploading/deleting files, a special NetworkGate object is used in a fashion similar to a lock.
A separate gate is acquired for a specific object (keyed by its URL) each time a network operation starts and is released when it completes.
If gate aquisition fails, it means that an operation is already in progress concerning this file and the new operation is skipped.