On this page· 4
The appendix
- file format 6
- Rust SDK 1.0 preview
Plain words: the finder
Binary data, such as embedding rows and tensors, does not go in the JSON lines. It goes in one optional block between the text section and the footer: the appendix.
The appendix is one complete CSDT container, embedded byte for byte. CSDT is a binary container format with numbered sections, defined by the CSDT container specification; PLXI defines no binary encoding of its own.
Offsets inside the container count from its own first byte, csdt_offset, not from the start of the pack.
csdt_offset is a multiple of 64, and the container aligns its own contents to 64 bytes, so data aligned inside the container is aligned in the pack too.
The container's header is 128 bytes, and its first four bytes are the ASCII text CSDT.
When a pack has an appendix, a marker line comes after the last record, then zero bytes up to the next multiple of 64, then the container. The footer's csdt_offset and csdt_size say where it is. File layout
An embref record binds an entity to one row of binary data. Its target has one of two shapes.
| Shape | Points at |
|---|---|
{"local":{"section_index":S,"record_index":R}} | row R of section S of this pack's own appendix |
{"ext":{"ref_id":F,"record_index":R}} | row R of the section named by the csdt_ref record whose ref_id is F, in another file; R counts from the start of that section |
Verify counts a target that does not resolve as dangling. A caller that binds a dangling target gets dangling_embref.
Note
Pack::bind_embrefs resolves local targets; given an ext target it returns unsupported.
The binding profile the Rust SDK reads is the Compact profile: a local target names a section of type 0x0020, Compact, whose rows are 320 bytes each. A target of another type is counted as a type mismatch, and a caller that binds one gets section_type_mismatch.
The profile belongs to the reader, not to the format: other kinds of section are legal binding targets.
Verify step 9 checks the container: its own structure, the CRC32C of every section, and the footer's csdt_file_checksum against the container's. These run in addition to the digest. Verify
Example 4 writes two embedding rows and a tensor, reads them back, and binds each row to its entity. Example 4: Appendix and embeddings