Run a Complex CLI Tool from Scripts and Check Success in Its JSON Report

## Judge CLI runs by structured data, never by progress text When a script, pipeline, or coding agent drives a complex command-line tool, treat the CLI as a versioned machine interface rather than an interactive shell. Separate planning, execution, and verification so each stage consumes structured data and produces evidence that the next stage can validate. - Do not scrape human-readable progress text to decide whether a run succeeded. - Do not infer supported tasks, output classes, or license requirements from memory. - Record the tool version, resolved runtime settings, requested scope, and produced artifacts. - Fail closed when required structured evidence is absent or inconsistent. ## List the tool's tasks and classes before planning a run Use the tool's lightweight information or registry command before constructing an expensive job. Prefer a JSON view generated from the same registry that validates the main CLI, because source-copied lists and hard-coded names can drift from the installed version. Discovery should expose: - available tasks and their input modality; - whether each task requires a license or entitlement; - the exact index-to-class-name mapping accepted by class filters; - the installed tool version and any task-specific constraints. Prefer a discovery path that does not load heavy inference dependencies, initialize an accelerator, download model weights, or mutate local state. Scope the query to one task or class view when the full registry is unnecessary. ## Spell out every input, output, and setting so a run can be repeated Construct every run from explicit values: input, output, task, device policy, optional class subset, output representation, and report path. Accept only documented input forms, such as a volume file, a directory of ordered slices, or an archive of slices, and validate the chosen output destination against the selected output mode. Before execution: 1. Confirm that the requested task and classes exist in the current capability registry. 2. Confirm that the input form is supported and the output path is unambiguous. 3. Resolve licensing before starting a gated task. 4. Select the device and performance mode deliberately. 5. Reserve a unique path for the machine-readable run report. Do not reuse an output directory when stale artifacts could be mistaken for current results. Preserve the resolved command parameters with the run evidence. ## Choose the device and fast mode, and check the license before a gated task Treat device selection and speed options as part of the reproducibility contract. Record the resolved device rather than only the requested one, because unavailable acceleration may cause a CPU fallback with materially different runtime. When resources are constrained, reduce cost through documented controls: - use a lower-resolution fast mode when its accuracy tradeoff is acceptable; - request only the classes needed by downstream work; - use a specific accelerator index when multiple devices exist; - select the supported platform backend explicitly when required. Never silently substitute a faster or narrower mode when it changes output quality or scope. If a task is license-gated, stop before inference unless valid authorization is already configured. ## Check the JSON report and every file it lists Require the CLI to write a structured report after successful completion. The report is the authoritative handoff artifact for automation and should include: - tool, runtime, framework, and dependency versions; - task, modality, and license requirement; - resolved device and performance flags; - normalized input and output locations; - requested class subset and the final index-to-class mapping; - wall-clock runtime and the exact output files produced. Verify that the report exists, parses as JSON, identifies the expected task, and lists every artifact needed by the next stage. Then verify that each listed file exists. Use dedicated statistics output for per-class volume, intensity, voxel counts, dispersion, extrema, centroids, or bounding boxes; do not infer these values from filenames or progress text. ## Sort failures by exit code and retry into a fresh folder Interpret process exit status before reading any optional artifacts. Distinguish successful completion, runtime failure, and argument or usage failure so retries do not repeat invalid commands. - On success, validate the report and declared output files. - On runtime failure, capture diagnostic stderr and classify causes such as missing authorization, unavailable resources, corrupt input, or an unhandled tool error. - On argument failure, correct the task, input, output, or option set before retrying. - Use quiet mode to reduce routine noise and debug mode only when additional diagnostic context is needed. Do not treat partial files, a progress message, or an empty output directory as success. A retry must use a new or cleaned destination so artifacts from the failed attempt cannot satisfy verification. ## Call the public API and chain later steps from the report's file list Use the supported high-level programming API when in-process execution is required, but keep the same explicit task, input, output, and verification contract as the CLI workflow. For planning and validation, prefer a lightweight registry module that can list tasks and classes without importing the full inference runtime. Keep orchestration independent from implementation internals: discover from the public registry, execute through the documented CLI or API, and chain downstream processing from the structured report's output list. This preserves automation across version changes while avoiding brittle stdout parsing and duplicated capability tables.