On this page· 4
Errors and limits
- file format 6
- Rust SDK 1.0 preview
Goal: open packs from files and from gzip input, cap input of unknown size, and act on what went wrong.
Every failing call returns a plxi_sdk::Error with three parts.
| Part | Rust | Use |
|---|---|---|
| kind | Error::kind | one of 22 values. Code branches on this. |
| message | Error::message | a sentence for people. It can change between releases. |
| details | Error::details | a JSON object with the numbers behind the failure |
- 1. Match on
err.kind(). - 2. Add a
_arm.ErrorKindcan gain values in a later release, and Rust requires the arm. - 3. Log
err.to_json(). It holds all three parts on one line.
Displayprints the kind, a colon and the message, as inchecksum_mismatch: the file's SHA-256 digest does not match the one recorded in its footer.- Each kind has a fixed string.
ErrorKind::as_strreturns it, andErrorKind::from_keyreads it back. - A fault inside the SDK is returned as an error of kind
internal, and the calling process keeps running, when your program is built withpanic = "unwind", which is Cargo's default. A program built withpanic = "abort"stops at the fault instead. - Some caller mistakes are not errors:
AppendixSpec::compactandwith_tensor_f32returnNone, andMergeStrategy::from_namereturnsNonefor an unknown name.
- 1. Write to a path with
write_to_path, and open by path withPack::open. - 2. Pass gzip bytes to
Pack::from_bytes, or a.plxi.gzpath toPack::open. Input that starts with the bytes1F 8Bis inflated first. - 3. Set a cap with
OpenOptions::default().with_max_input_bytes(n)when input comes from elsewhere.
Needs the Rust SDK
This program also needs flate2 = "1.1" under [dependencies] in its Cargo.toml.
files_limits_errors.rsrust
// Files, gzip, a size limit, and handling errors by kind.
use std::io::Write as _;
use flate2::{write::GzEncoder, Compression};
use plxi_sdk::{write, write_to_path, ErrorKind, OpenOptions, Pack, Record};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let records = vec![
Record::from_json_line(r#"{"k":"ent","id":"e1","t":"Person"}"#)?,
Record::from_json_line(r#"{"k":"ent","id":"e2","t":"Person"}"#)?,
];
let id = std::process::id();
let dir = std::env::temp_dir().join(format!("plxi-example-{id}"));
std::fs::create_dir_all(&dir)?;
// Write to a file and open it by path.
let path = dir.join("people.plxi");
write_to_path(&path, &records, None)?;
let pack = Pack::open(&path, OpenOptions::default())?;
assert_eq!(pack.record_count(), 2);
// A gzip-compressed pack opens the same way: input that starts
// with the gzip magic bytes is inflated first.
let plain = write(&records, None)?;
let mut enc = GzEncoder::new(Vec::new(), Compression::default());
enc.write_all(&plain)?;
let gz = enc.finish()?;
let from_gz = Pack::from_bytes(gz.clone(), OpenOptions::default())?;
assert_eq!(from_gz.to_bytes(), plain.as_slice());
// No size limit applies unless one is set. With a limit, both the
// input and the inflated output are capped.
let cap = plain.len() as u64 - 1;
let capped = OpenOptions::default().with_max_input_bytes(cap);
let err = Pack::from_bytes(gz, capped).unwrap_err();
assert_eq!(err.kind(), ErrorKind::LimitExceeded);
assert_eq!(err.details()["max_input_bytes"], cap);
// Errors carry a stable kind, a message for people, and JSON
// details.
let missing = dir.join("nope.plxi");
let missing = Pack::open(missing, OpenOptions::default()).unwrap_err();
let what = match missing.kind() {
ErrorKind::Io => "could not read the file",
ErrorKind::ChecksumMismatch => "the file is damaged",
_ => "something else",
};
assert_eq!(what, "could not read the file");
let not_object = Record::from_json_line("[1,2]").unwrap_err();
assert_eq!(not_object.kind(), ErrorKind::InvalidRecord);
let cut_short = r#"{"k":"ent","id":"e1""#;
let cut_short = Record::from_json_line(cut_short).unwrap_err();
assert_eq!(cut_short.kind(), ErrorKind::Json);
let key = ErrorKind::from_key("limit_exceeded");
assert_eq!(key, Some(ErrorKind::LimitExceeded));
println!("{}", err.to_json());
std::fs::remove_dir_all(&dir)?;
Ok(())
}
Output
{"kind":"limit_exceeded","message":"the inflated input is larger than the caller's limit of 460 bytes; inflating stopped after 461 bytes","details":{"inflated_at_least":461,"max_input_bytes":460}}Ran with cargo run --example 08_files_limits_errors · exit status 0
| Limit | Value | When it is passed |
|---|---|---|
| Input size | none unless set | limit_exceeded, with size and max_input_bytes in the details |
| Inflated size of gzip input | the same cap | inflating stops one byte past the cap: limit_exceeded, with inflated_at_least and max_input_bytes |
| Gzip layers | one | gzip inside gzip is not unwrapped twice |
| Address width | the platform's | a footer offset or size that does not fit: limit_exceeded |
| JSON nesting | 127 levels | 128 nested arrays fail with json |
| Memory | the whole pack, in one buffer | there is no streaming reader |
- No cap is applied by default. The SDK does not guess one.
- For a path, the file's size is checked against the cap before the file is read.
- Damaged gzip input under the cap fails with
io. Pack::to_bytesreturns the inflated bytes when the input was gzip.