ASQI CLI Reference¶
This page documents the ASQI command-line interface.
Overview¶
Documentation¶
asqi¶
A test executor for AI systems.
asqi [OPTIONS] COMMAND [ARGS]...
Options
- --install-completion¶
Install completion for the current shell.
- --show-completion¶
Show completion for the current shell, to copy it or customize the installation.
evaluate-score-cards¶
Evaluate score cards against existing test results from JSON file.
asqi evaluate-score-cards [OPTIONS]
Options
- --input-file <input_file>¶
Required Path to JSON file with existing test results.
- -r, --score-card-config <score_card_config>¶
Required Path to grading score card YAML file.
- -o, --output-file <output_file>¶
Path to save evaluation results JSON file.
- Default:
'output_scorecard.json'
execute¶
Execute the complete end-to-end workflow: tests + score cards (requires Docker).
asqi execute [OPTIONS]
Options
- -t, --test-suite-config <test_suite_config>¶
Required Path to the test suite YAML file.
- -s, --systems-config <systems_config>¶
Required Path to the systems YAML file.
- -r, --score-card-config <score_card_config>¶
Required Path to grading score card YAML file.
- -o, --output-file <output_file>¶
Path to save execution results JSON file.
- Default:
'output_scorecard.json'
- -c, --concurrent-tests <concurrent_tests>¶
Number of tests to run concurrently (must be between 1 and 20, default: 3)
- Default:
3
- -m, --max-failures <max_failures>¶
Maximum number of failures to display (must be between 1 and 10, default: 3).
- Default:
3
- -p, --progress-interval <progress_interval>¶
Progress update interval (must be between 1 and 10, default: 4).
- Default:
4
- --container-config <container_config_file>¶
Optional path to container configuration YAML. If not provided, built-in defaults are used.
execute-tests¶
Execute only the test suite, skip score card evaluation (requires Docker).
asqi execute-tests [OPTIONS]
Options
- -t, --test-suite-config <test_suite_config>¶
Required Path to the test suite YAML file.
- -s, --systems-config <systems_config>¶
Required Path to the systems YAML file.
- -o, --output-file <output_file>¶
Path to save execution results JSON file.
- Default:
'output.json'
- -tn, --test-names <test_names>¶
Comma-separated list of test names to run (matches suite test names).
- -c, --concurrent-tests <concurrent_tests>¶
Number of tests to run concurrently (must be between 1 and 20, default: 3)
- Default:
3
- -m, --max-failures <max_failures>¶
Maximum number of failures to display (must be between 1 and 10, default: 3).
- Default:
3
- -p, --progress-interval <progress_interval>¶
Progress update interval (must be between 1 and 10, default: 4).
- Default:
4
- --container-config <container_config_file>¶
Optional path to container configuration YAML. If not provided, built-in defaults are used.
validate¶
Validate test plan configuration without execution.
asqi validate [OPTIONS]
Options
- -t, --test-suite-config <test_suite_config>¶
Required Path to the test suite YAML file.
- -s, --systems-config <systems_config>¶
Required Path to the systems YAML file.
- --manifests-dir <manifests_dir>¶
Required Path to dir with test container manifests.