XpertMart User's Manual: Synchronization

Synchronization

Chapter Topics
Introduction to Synchronization
Date Last Modified
Using an FTP Server
Configure Synchronization
Overview of the Synchronization Cycle
Synchronize Documents
Upload Document Synchronization
Download Document Synchronization
Process Document Synchronization
Synchronize Catalogs
Upload Catalog Synchronization
Download Catalog Synchronization
Process Catalog Synchronization

Introduction to Synchronization
When you are operating in a multi-location environment your database can be said to be a "distributed database" since the data is distributed among the different locations. Users at any station (be it the Main, a store or a warehouse) see their part of the database live, with every change immediately updated, but the other parts of the database are snapshots at a given point in time and do not reflect the latest information. The longer it's been since the last snapshot of the database was taken, the less relevant it becomes.

Therefore, the many parts of the distributed database need to be regularly "synchronized" to be kept current. The Synchronization process in XpertMart™ (often called polling by other systems) is used to transmit information between the Main station and the Remote stations so that they are all kept as up to date as possible.

This chapter will provide an overview of the Synchronization process as well as explain how to do a synchronization cycle manually. A subsequent chapter will discuss how to Automate the Synchronization Process Using the Scheduler . Other chapters will explain how to Configure the Synchronization , how to Add a New Store to the Database and how to Monitor and Troubleshoot the Synchronization . There is also a parallel Synchronization Audit process you can use as a quality check to make sure all data is successfully making it from the Main to the Remotes and vice-versa.

Date Last Modified
It would be unnecessarily burdensome for the Remotes to send al of their Documents and for the Main to send over every Catalog every time there was a Syncrhonization. The large size of the files being trasnmitted would slow the process down. To be more efficient, XpertMart ™ only synchronizes information that is new or that has changed.

Every table in the database has a Date Last Modified or DLM field attached to it. (The DLM is not visible to the end-user but is read automatically by the system). The date of the last change made to the table. If you add a new vendor to the Vendors Catalog on May 3, 2003 then the DLM for the Vendors table in the database would be 20030503. Likewise if you change the price of one of your styles in styles catalog on May 1, 2003 then the DLM for the Styles table in the database would be 20030501.

Every time XpertMart™ succesfully completes a synchronization cycle it records the date of the synchronization. The next time it does a synchronization XpertMart™ looks for any file whose DLM comes after the date of the last synchronization. Suppose a synchronization was done on May 2, 2003. When a synchronization takes place the following day, the system knows it needs to synchronize the Vendors Catalog because the DLM (May 3, 2003) comes after the last synchornization (May 2, 2003) so the information is new. However, the system knows it can safely skip sending the Styles Catalog because the DLM (May 1, 2003) comes before the date of the last synchronization (May 2, 2003) so the information has already been updated at every location.

Using an FTP Server
XpertMart™ uses an FTP Server to handle the sending and receiving of files between the Main station and the Remote stations. This is a vast improvement over older systems that used one-to-one polling to send and receive data. The FTP Server (FTP stands File Transfer Protocol) can be thought of functioning as a virtual mailbox. The Main can connect to the server and deposit its files there without any of the Remote stations having to be connected to the server. Each Remote can then access the server (at different times), download the files left there by the Main and upload their own files. The Main does not have to be connected to the server when the Remote stations are.

Using an FTP server affords us several important advantages:

1) The communication cycle is more efficient. If there are 30 Remote stations, the Main does not have to go through 30 individual polling sessions with each Remote but instead "broadcasts" its data once to the server.

2) Because the communication cycle is more efficient, it is less prone to error. Under one-to-one polling the Main and the Remot have to be both connected at the same time. If there is a problem on either side, the communication cycle is broken and data is not updated. So in a 30 store chain there are 60 polling pairings that need to be successful for all data to be updated. That multiplies the probability of something going wrong.

3) As a result of the communication process being more efficient, computers are online less frequently, minimizing their exposure to viruses and hacker attacks. The less exposed your system and data are to the outside world, the better.

4) Each store retains its own data. Remote stations are not dependent on a continuous live connection to the Internet for their data. If the connection to the server is lost, stores have their own database and can continue selling offline. There is no risk of catastrophic downtime.

5) On the other hand, the synchronization can be configured to take place as often as once every hour. So you have the best of both world: data can be kept up-to-date (only 1 hour old) but you are not dependent on a continuous connection to the Internet.

Configure Synchronization
Before you get started syncrhonizing data, the basic configuration needs to be setup. This is done by going to the Synchronize>Configure>Configure Synchronization menu.

To begin, Synchronization Directories need to be specified. These are the folders where the system places files that need to be sent out and files that are coming in and need to be processed. These directories are intermediary clearinghouses for data. By default, XpertMart™ uses the "Out" folder in the C:/Main directory for outgoing fiels and the "In" folder for incoming files.

If you select the "Use compresed file" checkbox the file size will be reduced and the process sped-up. IMPORTANT NOTE: If you choose to use compressed files at the Main then you must use compressed files at the Remote (and vice-versa). Otherwise the Main will upload compressed files and the Remote will not be looking to download compressed files and the synchronization will fail.

The next step is to specify the FTP Server that is going to be used for the synchronization. Enter the address of the FTP, the user name and the Password. The address of the FTP Server can be a URL, a DNS address or, if you are using a local server, the name of the server. If youare not sure if the FTP Server is configured correctly, there are two tests you can do to verify whether it is working correctly.

If your computer has a broadband or network connection to the Internet, XpertMart ™ will automatically access the FTP Server you specify during the synchronization cycle. If you do not have a dedicated Internet connection you will need to specify the Phone Dialer you will use to access the FTP Server.

To use the Phone Dialer select the checkbox and specify the name of the dial-up connection you want to use. The name must match exactly the name of a dial-up connection you have previously setup on that computer. You can modify the alloted time to dial and connect before the system hangs up and tries again.

The Configure Synchronization menu also lets you decide whether Minimums and Maximums as well as Customer data will be synchronized back and forth in addition to Documents and Catalogs.

Overview of the Synchronization Cycle
There are eight steps to a complete synchronization cycle. These are summarized in the diagram below.

The first step is for the Remote stations to Synchronize Documents . During this step the system looks for all Documents (Invoices, Receipts, etc.) that have been modified or created since the last synchronization by comparing the DLMs of every file with the date of last synchronization. The files that need to be sent to the Main are placed in the C:/Main/Out folder as specified in the configuration .

The second step is Upload Document Synchronization . During this step the system takes all files in the C:/Main/Out folder and uploads them to the FTP Server . The files are now sitting on the server where they can be downloaded at any time.

The third step is for the Main to Download Document Synchronization . During this step the system downloads all document files on the FTP Server and puts them in the C:/Main/In folder.

The fourth step is for the Main to Process Document Synchronization . During this step the system compares every Document file to the files in the C:/Main/In folder and updates all Documents files accordingly. All new Documents (Invoices, Receipts, etc.) are imported into the respective Documents Catalog.

Note that the Documents residing at the Main are merely copies of the real Documents. From a philosopy of design perspective, the valid Documents are considered to reside at the Remote. Should there ever be a problem during the synchronization process that requires resending Documents, the point of reference will always be the Remote.

At this point the Synchronization cycle is half complete. The Main now has up-to-date transaction data from every store, but the stores have not yet received the latest changes to Catalogs (e.g. price changes, new styles, etc.).

Step Five is for the Main to Synchronize Catalogs . During this step the system looks for all Catalogs that have been modified or created since the last synchronization by comparing the DLMs of every file with the date of the last synchronization. The files that need to be sent to the Remotes are placed in the C:/Main/Out folder.

During the next step, Upload Catalog Synchronization , the files in the C:/Main/Out folder are uploaded to the FTP Server.

The seventh step is for the Remotes to Download Catalog Synchronization . During this step the system imports all catalog files and places them in the C:/Main/In folder.

The final step, Process Catalog Synchronization , is when the system compares every Catalog file to the files in the C:/Main/In folder and updates all Catalogs accordingly.

The synchronization cycle is now complete. The Main has received all Document updates and the Remotes have received all Catalog updates. The rest of this chapter explains how to carry out each of these eight steps in XpertMart ™ manually. The next chapter explains how to use the Scheduler to have all eight steps take place automatically at scheduled times.

Synchronize Documents
To manually synchronize documents at a Remote, go to Synchronize>Documents>Synchronize Documents as shown below.

XpertMart™ will begin an automatic process that will create files for every document type for every date since the last synchronizaiton date. As this process is underway you will the the system go through every document type. In the example below, the system is creating the synchronization files for Invoices.

Once the system is done, you can open the C:/Main/Out folder and view the contents. In the example below, the last synchronization was done on May 2, 2003 and the current synchronization is being done on May 5, 2003:

A file is created for each date since the last synchronization even if there were no changes or modifications (the file will be 1kb in size and is essentially empty). For most Document types two files are actually created, one with general information about the transaction and the other with the actual details of which items are affected by the transaction. For invoices there is a third file created that has all of the payment information.

So in the case of Invoices of May 5, 2003 the following three files were created:
    INVOICES00120030503.xfn = data from the Invoice header including employee logged in, invoice number, and invoice type
    INVDET00120030503.xfn = data from the items area including quantity, price and discounts
    INVPAY00120030503.xfn = data form the payments footer including tender types

The numbers in the file name tell the system the store number the file is originating from and the date. Invoices00120030503.xfn indicates the file corresponds to data from Store 1, May 5, 2003. Invoices01720021123.xfn indicates the file contains data from Store 17's sales on November 23, 2002.

The .xfn extension indicates that the data being sent for that particular date is final. A .xtm extension indicates that the data is temporary. The file created for the day the synchronization is taking place will always be a temporary (.xtm) file because it's possible that changes might be made to the at the Remote even after the synchronization takes place. For example, the synchronization might take place at 8:00 p.m. but an hour later the store manager notices a pile of receipts that were not inputted into the system. Those changes take place after the synchronization has taken place. A user at the Main will get a snapshot of the days sales but the system knows that the final data for the day will not be send until the next synchronization when it will replace the .xtm file with the .xfn file.

In the example above, the synchronization is taking place on May 5, 2003 so the Invoices00120030505.xtm is a temporary file.

You do not need to concern yourself with how XpertMart ™ takes these synchronization files and then assembles transaction data from them. It is useful, however, to understand the files that are created during the synchronization process so that if there is ever a communication failure (e.g. a power outage or computer mistakenly left offline) you know enough to do some detective work and figure out which files were sent or not sent. A later chapter will explain how to manipulate synchronization dates to troubleshoot these typs of problem.

Upload Document Synchronization
To manually upload the synchronized document files to the FTP Server go to Synchrnoize>Documents>Upload Document Synchronization:

Download Document Synchronization
To download the files on the FTP Server the remotes have uploaded, go to Synchronize>Documents>Download Document Synchronization at the Main:

XpertMart™ will access the FTP Server and download all of the files into the C:/Main/In directory. When the process is done, you will be able to see the files in your hard drive:

Process Document Synchronization
To process the document synchronization files that are in the C:/Main/In directory, go to Synchronize>Documents>Process Document Synchronization:

XpertMart™ will begin an automatic process that will check every Documents Catalog against the synchronization files in the In folder looking for any new data. As this process is underway you will the the system go through every document type. You may also notice that the system is also processing the Client Catalog. Since customer data is obtained at the Remotes, it is the one catalog that is also synchronized along with documents form the Remote stations to the Main.

When the process is finished, all transaction data in the Documents catalogs should be up to date.

Synchronize Catalogs
To manually synchronize catalogs at the Main, go to Synchronize>Catalogs>Synchronize Catalogs as shown below.

XpertMart™ will begin an automatic process that will create a file for every catalog for every date since the last synchronizaiton date. As this process is underway you will the the system go through every catalog. In the example below, the system is creating the synchronization files for the Stores Catalog.

Once the system is done, you can open the C:/Main/Out folder and view the contents. The result is similar to that of the Synchronize Documents process.

Once again a file is created for every day since the last synchronization. The BRANDS00020030504.xfn contains data modified or created in the Brands Catalog on May 4, 2003. The 000 before the date indicates that the file was created by the Main station (which is internally assigned store number 000) and the .xfn extension indicates that it is the final file created for that date. The CLASSES00020030505.xtm file contains preliminary data that was modified or created in the Classes Catalog at the Main and will be replaced by an .xfn file during the next synchroniztion.

You might notice that in addition to catalog files there are also synchronization files for Purchase Orders as these are the one transaction that can also be made at the Main station.

Upload Catalog Synchronization
To manually upload the synchronized catalog files to the FTP Server go to Synchrnoize>Catalogs>Upload Catalog Synchronization:

XpertMart™ will take all of the files in the C:/Main/Out folder and upload them to the FTP Server you specified in the configuration . When the process is done, you will be able to see that the catalog synchronization files are now residing on the FTP Server.

Download Catalog Synchronization
To download the files on the FTP Server the remotes have uploaded, go to Synchronize>Catalogs>Download Catalog Synchronization at each Remote:

XpertMart™ will access the FTP Server and download all of the files into the C:/Main/In directory. When the process is done, you will be able to see the catalog synchronization files in your hard drive.

Process Catalog Synchronization
To process the catalog synchronization files that are in the C:/Main/In directory, go to Synchronize>Catalogs>Synchronize Catalogs:

XpertMart™ will begin an automatic process that will check every Catalog against the synchronization files in the In folder looking for any new data. As this process is underway you will the the system go through every Catalog on screen. When the process is finished, all data in the Catalogs should be up to date. The synchronization process is now complete.

* Return to Table of Contents *