On this page· 5
Records and kinds
- file format 6
- Rust SDK 1.0 preview
Plain words: the cover note, cards, links and where-from tags
The body of a pack is JSONL: one JSON object per line, encoded as UTF-8, each line ending in LF. Each object is a record.
Two members belong to the format, not to any kind.
| Member | Type | Required | Meaning |
|---|---|---|---|
k | string | yes | the kind. A line with no string k is rejected with invalid_record. |
v | non-negative integer | no | the kind's own version. Absent means 1. |
The other members belong to the kind.
A record is typed when its kind is registered and its v is no higher than the version the reader supports. In revision 6.0 every registered kind is at version 1. Any other record is opaque.
A typed record is checked against its kind's members. A missing required member or a wrong JSON type is rejected with json. Members the kind does not name are kept and written back after the named ones.
An opaque record is kept whole, not rejected. A writer emits it with the same members, the same values, and the members in the order they were read. That is how a record from a newer writer passes through an older reader.
| Read | Written | |
|---|---|---|
| typed | {"k":"ent","id":"e2","t":"Person"} | {"k":"ent","v":1,"id":"e2","t":"Person","stub":false} |
| opaque | {"k":"x-future","v":7,"id":"n1","z":{"b":1,"a":2}} | the same line |
A typed record keeps its values and may change its bytes: k comes first, then v, then the kind's members in their declared order. An opaque record keeps its member order; the format promises equal values, not equal bytes, so escapes and number spellings follow the format's own rules.
A 32-bit float member of a typed record, such as salience, is written as the shortest decimal that reads back to the same 32-bit value, not by the binary64 rule of the identity page: 0.123456789 is written 0.12345679. In an opaque record the same member keeps its binary64 spelling. Spec §4.3
Revision 6.0 registers 19 typed kinds, ranked 0 to 18. A kind's rank is its position in the order records are written. Kinds 0 to 7 describe the pack itself, entities and the relations between them, binary data bound to entities, and where statements came from; their members are listed here. ? marks an optional member; types not stated are strings.
| Rank | Kind | What it is | Primary id | Members |
|---|---|---|---|---|
| 0 | meta | when the pack was made and from what | created_at, |, source | created_at, source, tool?, description?, ngdb_generation? (u64) |
| 1 | payload | a summary of the appendix | the empty string | embedded (bool), csdt_file_checksum? (u32), sections? (array), refs? (array of strings) |
| 2 | csdt_ref | a pointer to binary data in another file | ref_id | ref_id, path, shard_set_id? (u32), file_checksum (u32), source_hash, section_type (u16), section_index (u32), record_range? |
| 3 | ent | an entity: one thing | id | id, t, name?, salience? (32-bit float), stub? (bool), properties?, prov? |
| 4 | rel | a relation from one entity to another | src, tgt, kind, joined by | | src, tgt, kind, properties?, prov? |
| 5 | hyper | a relation among several entities | id | id, members (array of strings), kind, properties? |
| 6 | embref | a binding from an entity to a row of binary data | entity_id, |, the target key | entity_id, target, annotations? |
| 7 | prov | where a fact came from | subject, method, the time padded to 20 digits, joined by | | subject, agent, method, unix_secs (u64), inputs?, notes? |
Optional members are left out when absent, with four exceptions a writer always emits: payload.csdt_file_checksum (0), payload.sections ([]), payload.refs ([]) and ent.stub (false).
A stub is an ent record with "stub":true: an entity known by its id only, standing in for a full record kept elsewhere.
Eleven more kinds are registered for applications built on PLXI. They are typed, like kinds 0 to 7. Section 5.1 of the specification gives each one's name, rank and primary id, which is all that sorting, merging and comparing use. Their member lists are specified separately; the Rust SDK reads and writes all eleven kinds as typed records.
| Rank | Kind | Primary id |
|---|---|---|
| 8 | grounded_answer | answer_id |
| 9 | citation | source_answer_id, entity_id, span.start padded to 20 digits, joined by | |
| 10 | clause | clause_id |
| 11 | knob_set | scope, key, unix_secs padded to 20 digits, joined by | |
| 12 | discovery | gid |
| 13 | promo | content_hash, order padded to 20 digits, joined by | |
| 14 | codebook_manifest | codebook_id |
| 15 | session | session_id |
| 16 | sev | step padded to 20 digits, |, t |
| 17 | session_end | end_unix padded to 20 digits |
| 18 | outcome | the identity form of its outcome member |
"Padded to 20 digits" means the number in decimal with zeros in front, 20 characters long, so that primary ids sort as text in number order.
All 19 kinds: Spec §5.1
Eleven kind names are reserved. Each has a fixed version-1 shape and no typed reader in revision 6.0, so a reader keeps them opaque.
One is mut, the change record. Change records
The other ten names are specified separately. A reader keeps any record whose kind it does not know as an opaque record, so it needs none of the ten names to read a pack.
Kind names that begin with x- belong to applications: no revision of this specification adds one to the typed kinds (§5.1) or the reserved kinds (§5.2). Spec §5.2
- A writer emits no empty lines, no carriage-return byte outside a JSON string, no raw LF inside a value, and no repeated member name inside one object.
- A reader skips empty lines and strips one trailing carriage return. Skipped lines are not counted as records.
- A body that is not valid UTF-8 is rejected with
invalid_utf8.