Bulk Data Downloading

Should you wish to download all available records, and possibly keep your copy "up to date" with minimal overall load, then several options exist

This page will reference "customers" in all examples as that is often the first set of data that grows and becomes too large to handle easily. The methods explained work equally as well for other areas, products, sales, accounts, logs etc

FD1 imposes maximum response sizes for a single response packet. This maximum is large enough for typical use, but may not be for bulk downloads. The limits can vary between server, and cannot simply be increased.

Simple Poll and Download

The obvious first choice is to simply request the data periodically as a single call.

{
  a: "fd1.customers.get_customers",
  rq: 12345
}

{
  r: "fd1.customers.get_customers",
  rp: 12345,
  data: {
    rows: [ {  ... },
            {  ... },
            {  ... },
            ...
    ]
  }
}

This approach is completly fine for small tables (say less than 1000 rows) or where you are infrequently requesting data

You can use "qo" in the request to list exactly what fields you require. This can be faster for the server and reduce network traffic considerably.

{
  a: "fd1.customers.get_customers",
  rq: 12345,
  qo: {
    cid: true,
    name: true,
    phone: true,
    email: true
  }
}

Simple Poll and Download using Chunked Responses

This method extends the simple poll and the server can respond with multiple response packets

{
  a: "fd1.customers.get_customers",
  rq: 12345,
  mo: "chunk"
}

{
  r: "fd1.customers.get_customers",
  rp: 12345,
  ch: 1,
  data: {
    rows: [ {  ... },
            {  ... },
            ...
    ]
  }
}

{
  r: "fd1.customers.get_customers",
  rp: 12345,
  ch: 2,
  data: {
    rows: [ {  ... },
            {  ... },
            ...
    ]
  }
}

// repeats until done  (ch:0)

This technique can be used for any sized table, and can result in a large amount of network traffic.

If the network fails mid transfer, you will need to restart the whole operation. The chunk numnbers (ch) in the response are not able to be used to restart or request retransmission

Polling for Changes and Download using Chunked Responses

Most data in Fieldpine has a semi hidden field "rve" which is managed by Fieldpine (you cannot set it directly) and is increased whenever the record changes. Think of it like a "last edit date". We use "rve" so that the field name is the same over all tables, and there is no confusion around timezones. From your point of view, you can simply consider it an increasing number.

Sidenote. Fieldpine uses two rve formats, one stored in a double (YYYYMMDD.HHMMSSccc) the other in a 64 bit integer (YYYYMMDDHHMMSScccNN). Within FD1 we are trying to only show the 64 bit integer version, but you may periodically see the double format.

Once you have the current highest RVE in your system, (which can be determined in your system with something like select max(rve) from table) your poll includes that RVE value

{
  a: "fd1.customers.get_customers",
  rq: 12345,
  mo: "chunk",
  q: {
    "_rve(gt)": 2024010113245600023 // Highest value you have
  }
}

The response is identical to the standard poll, but only includes rows where the RVE is "greater than" your current value

We still recommened using chunked response format, as something may happen server side and change the RVE values on millions of rows in a short space of time.

Subscribing to Firehose of changes

A firehose provides almost instant advice when something changes in Fieldpine. When an edit is made, we scan for active firehoses and send details immediately. This is different to a webhook, which causes a HTTP request to be generated for each edit. Webhooks are great for low volume random events, but would be overwhelmed for something like individual stock level changes in a large chain store.

{
  a: "fd1.firehose.open_firehose",
  rq: "myfirstfirehose",
  v: {
    "event": "pos.products.edit"
  }
}

Restarting a Firehose

Many firehoses provide an RVE value, which is a timestamp of the change. If you record these RVE values as you receive them, you can reconnect and request all changes since that RVE value.

Notes:

• If you request a restart firehose and explicity set the RVE to "all", then you will be sent all existing records, complete table extract, and changes made from now forward
• Not all firehoses can be restarted. Some events are transient and gone if you didn't catch them
• When restarting a firehose there are essentially two parts sending to you. (a) The new changes and (b) Historic records in the database. There is a chance that you might receive a new change and then a historic change for the same record afterwards. This is called an un-sequenced restart. You can tell the records involved by examining the RVE values but you might not be able to fully reconstruct current data. You have a few options
  • Check the documentation for the firehose endpoint. It may clarify how to restart. Example stock level changes tell you to ignore historic changes
  • You can request a "sequenced restart" where the server sends data in strict order. To do this though, it holds back changes until you have received all historic data, which might be a non-zero amount of time
  • You can capture affected records and request a single "current view" for just that record using other fd1 endpoints. eg. If you detect customer 12345 has an unsequenced change, simply issue a single call to read only customer 12345 current data
• If you are working with high volume firehoses and storing to your own database then you can use the RVE and restarts to store multiple records in one database transaction. If you open/save/commit each record individually then most databases will run quite slow, but if you buffer say 100 records then open/save(100)/commit you will probably store many more records per second.
  In the rare cases where the commit fails, close the firehose and restart from the lowest RVE in that transaction.