On this page· 4

Errors and limits

  • file format 6
  • Rust SDK 1.0 preview

Goal: open packs from files and from gzip input, cap input of unknown size, and act on what went wrong.

Errors

Every failing call returns a plxi_sdk::Error with three parts.

Table 1. Parts of an error: 3
PartRustUse
kindError::kindone of 22 values. Code branches on this.
messageError::messagea sentence for people. It can change between releases.
detailsError::detailsa JSON object with the numbers behind the failure
  1. 1. Match on err.kind().
  2. 2. Add a _ arm. ErrorKind can gain values in a later release, and Rust requires the arm.
  3. 3. Log err.to_json(). It holds all three parts on one line.
  • Display prints the kind, a colon and the message, as in checksum_mismatch: the file's SHA-256 digest does not match the one recorded in its footer.
  • Each kind has a fixed string. ErrorKind::as_str returns it, and ErrorKind::from_key reads it back.
  • A fault inside the SDK is returned as an error of kind internal, and the calling process keeps running, when your program is built with panic = "unwind", which is Cargo's default. A program built with panic = "abort" stops at the fault instead.
  • Some caller mistakes are not errors: AppendixSpec::compact and with_tensor_f32 return None, and MergeStrategy::from_name returns None for an unknown name.

Files and gzip

  1. 1. Write to a path with write_to_path, and open by path with Pack::open.
  2. 2. Pass gzip bytes to Pack::from_bytes, or a .plxi.gz path to Pack::open. Input that starts with the bytes 1F 8B is inflated first.
  3. 3. Set a cap with OpenOptions::default().with_max_input_bytes(n) when input comes from elsewhere.

Needs the Rust SDK

This program also needs flate2 = "1.1" under [dependencies] in its Cargo.toml.

files_limits_errors.rsrust
// Files, gzip, a size limit, and handling errors by kind.
use std::io::Write as _;

use flate2::{write::GzEncoder, Compression};
use plxi_sdk::{write, write_to_path, ErrorKind, OpenOptions, Pack, Record};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let records = vec![
        Record::from_json_line(r#"{"k":"ent","id":"e1","t":"Person"}"#)?,
        Record::from_json_line(r#"{"k":"ent","id":"e2","t":"Person"}"#)?,
    ];
    let id = std::process::id();
    let dir = std::env::temp_dir().join(format!("plxi-example-{id}"));
    std::fs::create_dir_all(&dir)?;

    // Write to a file and open it by path.
    let path = dir.join("people.plxi");
    write_to_path(&path, &records, None)?;
    let pack = Pack::open(&path, OpenOptions::default())?;
    assert_eq!(pack.record_count(), 2);

    // A gzip-compressed pack opens the same way: input that starts
    // with the gzip magic bytes is inflated first.
    let plain = write(&records, None)?;
    let mut enc = GzEncoder::new(Vec::new(), Compression::default());
    enc.write_all(&plain)?;
    let gz = enc.finish()?;
    let from_gz = Pack::from_bytes(gz.clone(), OpenOptions::default())?;
    assert_eq!(from_gz.to_bytes(), plain.as_slice());

    // No size limit applies unless one is set. With a limit, both the
    // input and the inflated output are capped.
    let cap = plain.len() as u64 - 1;
    let capped = OpenOptions::default().with_max_input_bytes(cap);
    let err = Pack::from_bytes(gz, capped).unwrap_err();
    assert_eq!(err.kind(), ErrorKind::LimitExceeded);
    assert_eq!(err.details()["max_input_bytes"], cap);

    // Errors carry a stable kind, a message for people, and JSON
    // details.
    let missing = dir.join("nope.plxi");
    let missing = Pack::open(missing, OpenOptions::default()).unwrap_err();
    let what = match missing.kind() {
        ErrorKind::Io => "could not read the file",
        ErrorKind::ChecksumMismatch => "the file is damaged",
        _ => "something else",
    };
    assert_eq!(what, "could not read the file");
    let not_object = Record::from_json_line("[1,2]").unwrap_err();
    assert_eq!(not_object.kind(), ErrorKind::InvalidRecord);
    let cut_short = r#"{"k":"ent","id":"e1""#;
    let cut_short = Record::from_json_line(cut_short).unwrap_err();
    assert_eq!(cut_short.kind(), ErrorKind::Json);
    let key = ErrorKind::from_key("limit_exceeded");
    assert_eq!(key, Some(ErrorKind::LimitExceeded));
    println!("{}", err.to_json());

    std::fs::remove_dir_all(&dir)?;
    Ok(())
}
Output
{"kind":"limit_exceeded","message":"the inflated input is larger than the caller's limit of 460 bytes; inflating stopped after 461 bytes","details":{"inflated_at_least":461,"max_input_bytes":460}}

Ran with cargo run --example 08_files_limits_errors · exit status 0

Limits

Table 2. Limits: 6
LimitValueWhen it is passed
Input sizenone unless setlimit_exceeded, with size and max_input_bytes in the details
Inflated size of gzip inputthe same capinflating stops one byte past the cap: limit_exceeded, with inflated_at_least and max_input_bytes
Gzip layersonegzip inside gzip is not unwrapped twice
Address widththe platform'sa footer offset or size that does not fit: limit_exceeded
JSON nesting127 levels128 nested arrays fail with json
Memorythe whole pack, in one bufferthere is no streaming reader
  • No cap is applied by default. The SDK does not guess one.
  • For a path, the file's size is checked against the cap before the file is read.
  • Damaged gzip input under the cap fails with io.
  • Pack::to_bytes returns the inflated bytes when the input was gzip.

Reference

Sections