Design Guide: Replication
If you wish to maintain replicated copy of data held in Fieldpine then you can
- Periodically request a complete table refresh. Simple, but high load on servers, network and your application
- Poll for changes since "last change" date and apply only those changes.
- Open a SSE socket (server sent events) to receive a realtime feed of changes
Maintaining a replicated copy of the table is quite common for static information such as products, although we advise caution when replicating sensitive data such as customer information.
Detecting Changes using Polling
Fieldpine provides an API end point that provides a summary of all dynamically changing information for a retailer. /OpenApi/RetailConfig/Dynamic This small API is designed to permit frequent polling and avoid repeated polling of other APIs simply to check if a change is present. Information for this API is specially updated by various components internally to indicate a change is present.
To detect changes to Products:
- Query /OpenApi/RetailConfig/Dynamic as often as you would like to receive change notifications
- From the response check the value of RetailConfigDynamic.RVE[1] If this value is higher than the value you have, then issue a query for changes since the the RVE you currently have
- When processing the response containing the changes, save the highest RVE returned on the data rows. Storing the value from RetailConfigDynamic.RVE[1] instead may mean you skip changes if not all changed rows where returned.
Detecting Removals using Polling
If you are maintaining a subset of data, such as only those products that have been marked as published to the website, then if a product is un-published, it will no longer appear in the response packet. If you are only polling for changes though, then you will not see this change as the record is no longer visible to you. Stated another way, you asked for all changes to published products changed since 11am. If a product was un-published at 11:10am, this will not be in the change list as it no longer meets the "published" criteria.
The easist way to solve this is simply not too. Have a periodic full refresh run and leave this un-publish entries to that process. If you have a button that users can click to "full refresh now", then users can control this. For many environments, this may be enough.
If you do want to automate it though, and still have the benefit of small polling updates, use the following logic
- Query /OpenApi/RetailConfig/Dynamic as often as you would like to receive change notifications
- From the response check the value of RetailConfigDynamic.RVE[1] If this value is higher than the value you have, then issue a query for changes to all products (even non published) since the the RVE you currently have
- Process each row in turn, if the publish flag is set, then insert/edit your copy. If the publish flag is not set, check and remove from your copy if it is present
- When processing the response containing the changes, save the highest RVE returned on all the data rows, not just the publised rows.