Webhooks & Firehoses
FD1 Client Protocol Definition
FD1 is a bidirectional protocol. Requests/Commands are sent and responses are received. Multiple responses are possible depending on the request.
At the top level, FD1 field names are short and terse, this is to reduce network traffic and parsing overhead.
Command or Request Packets
Name | Descripton |
a | Address or name of the endpoint this packet is intended for. |
q | "query" arguments used to select or subset an endpoint. The endpoint "list products" would use the "q" argument to select only those matching "abc" |
qk |
"query keywords" provides an array or comma seperated list of keywords that influence a specific endpoint.
These are typically used to quickly restrict or expand an endpoint. For example, a "supplier list" is generally
defined as "all suppliers marked active", however you might want to only include "used recently", so the keyword "onlyrecent"
can be provided.
Keywords are endpoint specific. Keywords are quite similar to providing selection query parameters, but are easier to work with |
k | "key" provides a single value for functions that deal with single records. |
v | "values". Endpoint specific data to exchange. These are the parameters or values for the specific endpoint handler |
rq | Request Id. A optional unique value you supply that is returned in responses. While optional, it is typically required to correctly route responses. The value you supply and be any value that can be converted to a string. It must be less than 200 characters |
rt | Requesting tab/thread. Any value sent in the request packet is reflected in the response. This allows a single web socket to be shared over multiple browser tabs or application threads. |
pt |
Protocol Type. Indicates the protocol this packet contains. If not present, or empty, this means "fd1" packets.
The intention is that onmessage handlers should check the pt field and divert those packets elsewhere. The common case is that packets with "pt=socket" can be sent in both directions and are used to control the overall socket connection itself. |
mo |
Mode. Sets specific mode options. The value can either an a comma seperated string or a JSON array.
|
dv | Timestamp. Provides a request timestamp as indication to the server to restart from this timestamp. This is used with Restartable Firehoses |
Response Packets
Name | Descripton |
Packet Addressing (who it is for) | |
r | Reply from an endpoint. Basically your "a" reflected back. This field may not be present if you sent an "rq" in your request. |
rp | Request id that was present in the command/request packet |
rt | Request tab/thread that was present in the command/request packet. The request "rt" value supplied is returned in the response |
dv | Fieldpine internal sequence id or RVE. Basically a timestamp that always increases for a single record. Not all responses include dv at this level, some requests also have RVE inside data packets. RVE is very useful to websockets as it permits restarting after failure |
ch |
Chunk number. When a request allows chunked response mode ( mo:'ch' ), this response field contains a number, starting at 1 for
the first packet and increasing by one (1,2,3...) for subsequent packets. The final packet is numbered 0. If the response only requies a single
packet it will be numbered 0. Should the server get it wrong and realise that the final packet has already been sent as a numbered packet, it will
send a final empty style response numbered 0.
Chunk numbers are indications only, you cannot request a single chunk to be retransmitted. |
Packet Data Payload (the data to deliver) | |
data | The object containing the requested data |
data.rows | An array of result rows. The result is always an array, even if zero or one result rows are returned. |
ds | Schema. What type of data is contained in "data". This value is not always present, but typically will be sent if a request can return multiple different pieces of data. This can either be a string or a number. |
error |
q "Query" Object
Some endpoints use a "q" object to query, select or restrict the data to be accessed. In SQL terminology this would be the "where" clause
- Any Field name that starts with an underscore has special meaning
- Any field that starts with a letter is a reference to column in the data
- If a field name is used in isolation (eg just "id" ) then this means id=value
-
If a field name includes a bracket reference at the end, (eg "id(gt)" ) then this defines the requested predicate check. Endpoints can block predicate types,
do not assume that you can use different predicates on every field
Q Field Server Operation field
field(=)
field(eq)
field(equal)Field = value field(>)
field(gt)
field(greaterthan)Field > value field(>=)
field(ge)
field(greaterthanequal)Field >= value field(<)
field(gt)
field(lessthan)Field < value field(<=)
field(le)
field(lessthanequal)Field <= value field(!=)
field(<>)
field(ne)
field(notequal)Field <> value field(like) Field like value
Example selecting a subset of products
{ a: "FD1 endpoint to read products" q: { "description(like)": "en", "depid(in)": [3,4,5], "price(ge)": 100, "price(lt)": 2500 } }The server might run the equivalent of this SQL
select ... where description like '%en%' and depid in (3,4,5) and price >= 100 and price < 2500
Name | Descripton |
Reply from an endpoint. Basically your "a" reflected back. This field may not be present if you sent an "rq" in your request. |
Examples
The minimum packet consists of a "a" value targeting an endpoint.
{ a: "FD1 endpoint" }
A packet requesting a subset of products
{ a: "FD1 endpoint to read products" q: { description: "Pencil" } }
A malformed packet
{ a: "FD1 endpoint to read products", q: { theDescription: "Pencil" } }
{ r: "FD1 endpoint to read products", error: { message: "Query field theDescription is unknown" } }
A malformed packet which uses a unique request id.
{ a: "FD1 endpoint to read products", rq: 823, q: { theDescription: "Pencil" } }
{ rp: 823, error: { message: "Query field theDescription is unknown" } }
A packet submitting an edit
{ a: "FD1 endpoint to edit products" v: { pid: 1234, description: "Big pencil" } }
A packet requesting all products
{ a: "FD1 endpoint to read products" rq: 1234, mo: "chunk" }
First response packet, indicated by ch=1
{ rp: 1234, ch: 1, data { rows: [ { pid: 123, description: "Pencil"}, { pid: 1234, description: "Big Pencil"}, ... ] } }
Subsequent response packets, indicated by ch=2. Each subsequent response increments this value
{ rp: 1234, ch: 2, data { rows: [ { pid: 5678, description: "Paper A4"}, { pid: 5679, description: "Paper A4 gold"}, ... ] } }
Final response packet. Field ch=0 to indicate final response
{ rp: 1234, ch: 0 data { rows: [ { pid: 441123, description: "Plastic plant pot"} ] } }
A tab specific request and response
{ a: "FD1 endpoint to read products", rt: "iframe.48484.bksl2", rq: "nb87374gb", q: { pid: 456 } }
Response
{ rt: "iframe.48484.bksl2", rp: "nb87374gb", data: { rows: [ { pid:456, description: "Roast beef sandwich" } ] } }
Binary Packets
Websockets can send either text packets or binary packets. Binary packets are used for things such as images or external files, and also to transfer very large amounts of data in pieces.
When binary data is received, the first byte contains a packet content descriptor
Byte Value | Description |
---|---|
0x01 | BinJson. Contains binary data chunks followed by a JSON packet using a specialised naming convention |
0x02 | Fragment. A piece of a large data transfer. If sending a 400Mb response, this will generally be sent as several hundred fragments that need to be reassemabled. |
0x47 'G' | GNAP. Internal Fieldpine protocol. Will not be seen unless specifically invoked |
0x01 - BinJson
This format is used to transfer relatively simple binary data, such as an image. The format is
1 Byte | 4 Bytes (little endian) | N Bytes (unstructured) | JSON Packet (UTF-8) |
0x01 | Offset to Json | Binary data chunks. Can be 0, 1 or several | JSON packet starts at "offset" and continues to end of packet |
Example | |||
0x01 | 0x07 0x01 0x00 0x00 | ...binary data... | {"x|bin":"5|258"} |
JSON offset = 263 | Contains 263 - 5 = 258 bytes total len = JsonOffset - 5 (header length) | Starting at offset 263 from the beginning of the packet. Uses all remaining data until end of packet |
Within the contained JSON, field names may have a |bin suffix to indicate they reference binary data. The struture is
"Field-name|bin": "binary-offset|length"
- binary-offset is relative to the start of the packet
- length is required. You will not receive binary-offset without a length
- Offset/Length may overlap with other fields using binary data, although this should be rare
- Binary data can only reference the binary section and not over or underflow
- binary-offset must be >= 5
- binary-offset must be < JsonOffset
- length must be >= 0
- (binary-offset + length) must be < JsonOffset
Example
{ "r": "fd1.images.get_image", "rq": "1234", "data": { "attribution":"Photo by ....", "copyright":"Public domain", "format":"jpeg", "content|bin":"5|189354" } }The value of "content" is in the binary data, starting at offset 5 for a length of 189354. That it is in bytes[5] through bytes[5+189354]. Notes
- No binary data chunks being present is specifically allowed.
- Endpoints will document if they might respond with BinJson, you will not suddenly starting receiving these.
Protocol Type: "socket"
Packets can be sent and received over the websocket with a "pt=socket" present. This indicates the packet is not an end user FD1 packet, but rather a system link level control packet.
{ pt: "socket", a: "fd1.socket.socket_option", ... }
Reconnecting a Socket
Send this command first when a socket is opened to reuse a previous session. All logins and definitions are restored.
{ pt: "socket", a: "fd1.socket.socket_option", reconnect: "...socket uniqueid provided to original socket..." }
- Only one network connection can be connected to a socket-unique-id at a time. You cannot reconnect to a socket that is still in use.
- Reconnections can time out, dont expect to reconnect 3 days later. The actual time frame available depends on server load and volume of traffic.
- Reconnects can only be made to the same server, you cannot reconnect to a fail over server.
- You don't have to reconnect, you can simply send your login information again if you need to reopen a socket that disconnected.
- When you reconnect, and data queued will be delivered
Operational Flow
When a socket is originally opened
GET /fd1/open_websocketThe first packet returned is a hello style response packet
{ pt: "socket", data: { socketid: "abcdef12345", pid: 165348 } }If you save the socketid
var SavedSocketId = event.data.data.socketid;Then if the socket is closed and you want to reopen it, open the websocket and immediately send a reconnect packet
GET /fd1/open_websocket { pt: "socket", a: "fd1.socket.socket_option", reconnect: SavedSocketId }
Requesting Debug Mode
Instructs the server to place the socket into debug mode. The field "setdebug" contains keywords to select various debug modes.
{ pt: "socket", a: "fd1.socket.socket_option", setdebug: "go-slow" }