FD1 Webhooks & Firehoses

Fieldpine provides options to capture events in near realtime. When an event is fired, a message is sent to your endpoint containing some level of detail

• Webhook Static registered URL Fieldpine sends to when events happen.
  • Fieldpine provides limited retry if your server is down
  • Suitable for low volume changes. As a general indication, Fieldpine dont permit webhooks for events that occur more than once every 30 seconds
  • Not currently generally available.
• Firehose Receives objects over a websocket as events happen
  • Suitable for high volume. As an indication, up to around 10,000 events/second
  • Retry depends on the event. Data style events (eg edit customer) can be restarted, but memory style events (freezer temperature changed) cannot
• Doorbell A doorbell URL is one that can be pinged, but does not explicitly require authorisation or Api Keys. Typically an external server 'rings a dorrbell' to say 'I have something for you', without passing any information of a sensitive nature. Doorbells do not respond with any data, and in fact can simply return 404 errors if they wish.

  For example, Fieldpine servers may ring a retailers doorbell when new data arrives. Fieldpine can call the retailer, but does not require any centralised Api Keys. Typically the receiver of a doorbell will schedule a poll operation.

Registering a Firehose

A firehose instructs the server to send all events or changes to you so long as the websocket is open. Firehose messages are only sent over websockets. You may open multiple firehoses over the same websocket. One websocket can have multiple different firehoses active. Avoid opening a seperate websocket for individual firehoses as websocket network connections are a finite resource.

Send a registration packet as shown over your websocket

{
    "a": "fd1.firehose.open_firehose",
    "rq": ... your request id ...
    "v": {
        "event": ... event name to subscribe ...
    }
}

Arguments

• a Endpoint Address. Constant, required.
• rq Request Id. Your request id. This value is returned in every firehose message sent to you in the "rp" field
• v.event A single event to subscribe too. See list below

Messages Generic Format

{
    "rp": ....      // Original "rq" that requested the webhook or firehose event
    "sc": ... data content type
    "rve": NNNN  // timestamp for this record
    "data": {
        // Event specific data
    }
}

Example

Request a firehose of any customer changes

{
    "a": "fd1.firehose.open_firehose",
    "rq": "MY-customer-changed",
    "v": {
        "event": "pos.customers.change"
    }
}

Server will start sending customer changes as they occur

{"rp":"MY-customer-changed","data": { "id": "abc13432", "action", "E"}}

{"rp":"MY-customer-changed","data": { "id": "JSH82232", "action", "I"}}

{"rp":"MY-customer-changed","data": { "id": "K9wh394s", "action", "E"}}

Lets request product changes too

{
    "a": "fd1.firehose.open_firehose",
    "rq": "MY-product-1234",
    "v": {
        "event": "pos.product.change"
    }
}

Server will start sending both as they occur. Note how the "rp" field is your request value, and also the "sc" field changes

{"rp":"MY-customer-changed","v": { "id": "acv82s", "action", "E"}}

{"rp":"MY-product-1234","v": { "id": "kso92", "action", "E"}}

{"rp":"MY-customer-changed","v": { "id": "znzj2738", "action", "E"}}

Tips for Receiving

Firehoses are technically capable of sending many thousands of events per second and how you handle those messages is important. If the server detects you are too slow handling the communication; it can and will revoke sending to you.

• Don't assume "my computers big enough". Size is not important, it is how you use it
• Don't try and process a message inside your message receive loop.
• Do create a "recevied queue" and put message in there

  // Thread reading websocket
  function FirehoseRead(x) {
      message x;
      while (WaitForNewMessage(x)) {
          LockReceivedQueue();
          AddToReceivedQueue(x);
          ReleaseReceivedLock();
  
          SetEvent(MessageReceived);      <<-- Done after lock released
      }
  }
  
  // Worker thread
  function ProcessFirehose() {
      while (forever) {
          LockReceivedQueue();
          if (GetFromReceiveQueue(...)) {
              ReleaseReceivedLock();      <<-- Important
              // Handle message, write to your database etc
          } else {
              ReleaseReceivedLock();
              WaitForEvent(MessageReceived);
          }
      }
  }
  

• The upper bound of messages is the total bandwidth of your network connecton to the firehose server. If you have a 1Gbit network connecion, then you can roughly expect to receive a maximum of 100Mbytes/second. If a firehose packet averages 500 bytes, then you will receive a maximum of 200,000 packets per second. If you are traversing the internet though, then all components between must be capable of this sustained load.
  Firehose servers do not require confirmation of messages. They also pump messages as soon as practical (unless you have specifically requested slowing)

Restartable Firehoses

A restartable firehose is one that can restart from a given point and resend all¹ messages.

To request a firehose restart, supply a "dv" value in your initial request

{
    "a": "fd1.firehose.open_firehose",
    "rq": "some-id",
    "dv": 2024031213004590422,      // <-- Restart timestamp
    "v": {
        "event": "pos.customers.change"
    }
}
                

¹ "all" for restartable firehoses means all possible. Some events are transient in nature and cannot be resent if missed.

Event Names

Name	Description	Where
pos.customer.change	A customer record is editted, created, deleted. A direct change to a customer record	Gds
pos.customer.change.any	Anything relating to a single customer changes. This can be record edit (pos.customers.change) or related items such as email/text message received, new sale.	Gds
salesbuilder.change	An active sale has changed. An active sale is one still being created, or worked with	PosGreen
debugscope.XYZ	Diagnostic channels to Fieldpine applications. These are advanced firehoses and not available to most callers.
debugscope.siteissue	Sent as site level configuration issues detected	PosGreen
debugscope.exception	Sent for code level exceptions. Can be serious or trivial exceptions	PosGreen
debugscope.sitetrace	Sends a brief trace of more user level descriptions without heavy technical detail	PosGreen
debugscope.poscommand	Sent as PosCommands evaluated	PosGreen
debugscope.dbstorage	Sent as database layer interacted with	PosGreen
debugscope.funccall	Sends data about individual function being invoked. Requires additional setup	PosGreen
debugscope.sale	Sent as sale type objects are altered	PosGreen
debugscope.settings	Sent for each setting read/write	PosGreen
debugscope.everything	Send everything. Reserved for automated test tools	PosGreen
debugscope.trace	Sends classic "trace lines" consisting of unstructured messages from developers	PosGreen
debugscope.active_pos	Historic	PosGreen
debugscope.p41	Historic	PosGreen
debugscope.p1792	Historic	PosGreen
debugscope.developer	Reserved for low level developers to send/receive random data while testing	Any