src.asqi.main¶

Attributes¶

`console`
`app`
`typer_click_object`

Functions¶

`load_yaml_file`(→ dict[str, Any])	Loads a YAML file with environment variable interpolation.
`load_score_card_file`(→ dict[str, Any])	Load and validate grading score card configuration.
`load_audit_responses_file`(→ dict[str, Any])	Load and validate audit responses YAML file.
`resolve_audit_options`(→ tuple[dict[str, Any], ...)	Handles all validation and preparation for audit indicators.
`load_and_validate_plan`(→ dict[str, Any])	Performs all validation and returns a structured result.
`version_callback`(value)	Display version information and exit.
`validate`(test_suite_config, systems_config, manifests_dir)	Validate test plan configuration without execution.
`execute`(test_suite_config, typer.Option], ...)	Execute the complete end-to-end workflow: tests + score cards (requires Docker).
`execute_tests`(test_suite_config, typer.Option], ...)	Execute only the test suite, skip score card evaluation (requires Docker).
`generate_dataset`(generation_config, typer.Option], ...)	Generate synthetic data using data generation containers.
`evaluate_score_cards`(input_file, score_card_config[, ...])	Evaluate score cards against existing test results from JSON file.

Module Contents¶

src.asqi.main.console¶

src.asqi.main.load_yaml_file(file_path: str) → dict[str, Any]¶

Loads a YAML file with environment variable interpolation.

Args:: file_path: Path to the YAML file to load
Returns:: Dictionary containing the parsed YAML data with environment variables interpolated
Raises:: FileNotFoundError: If the specified file does not exist ValueError: If the YAML file contains invalid syntax or cannot be parsed PermissionError: If the file cannot be read due to permissions

src.asqi.main.load_score_card_file(score_card_path: str) → dict[str, Any]¶

Load and validate grading score card configuration.

Args:: score_card_path: Path to the score card YAML file
Returns:: Dictionary containing the validated score card configuration
Raises:: FileNotFoundError: If the score card file does not exist ValueError: If the YAML is invalid or score card schema validation fails PermissionError: If the file cannot be read due to permissions

src.asqi.main.load_audit_responses_file(audit_responses_path: str) → dict[str, Any]¶: Load and validate audit responses YAML file.

src.asqi.main.resolve_audit_options(score_card_data: dict[str, Any], audit_responses_path: str | None, skip_audit_flag: bool) → tuple[dict[str, Any], dict[str, Any] | None]¶

Handles all validation and preparation for audit indicators.

Returns:

updated_score_card_data
audit_responses_data (or None)

Raises:

AuditResponsesRequiredError: If audit indicators exist but no responses provided

src.asqi.main.load_and_validate_plan(suite_path: str, systems_path: str, manifests_path: str) → dict[str, Any]¶

Performs all validation and returns a structured result. This function is pure and does not print or exit.

Returns:: A dictionary, e.g., {“status”: “success”, “errors”: []} or {“status”: “failure”, “errors”: [“error message”]}.

src.asqi.main.app¶

src.asqi.main.version_callback(value: bool)¶: Display version information and exit.

src.asqi.main.validate(test_suite_config: Annotated[str, typer.Option('--test-suite-config', '-t', help='Path to the test suite YAML file.')], systems_config: Annotated[str, typer.Option('--systems-config', '-s', help='Path to the systems YAML file.')], manifests_dir: Annotated[str, typer.Option(help='Path to dir with test container manifests.')])¶: Validate test plan configuration without execution.

src.asqi.main.execute(test_suite_config: Annotated[str, typer.Option('--test-suite-config', '-t', help='Path to the test suite YAML file.')], systems_config: Annotated[str, typer.Option('--systems-config', '-s', help='Path to the systems YAML file.')], score_card_config: Annotated[str, typer.Option('--score-card-config', '-r', help='Path to grading score card YAML file.')], output_file: Annotated[str | None, typer.Option('--output-file', '-o', help='Path to save execution results JSON file.')] = 'output_scorecard.json', audit_responses: Annotated[str | None, typer.Option('--audit-responses', '-a', help='Path to YAML file with manual audit indicator responses.')] = None, skip_audit: Annotated[bool, typer.Option('--skip-audit', help="Skip 'audit' type indicators if no audit responses are provided.")] = False, concurrent_tests: Annotated[int, typer.Option('--concurrent-tests', '-c', min=1, max=20, help=f'Number of tests to run concurrently (default: {ExecutorConfig.DEFAULT_CONCURRENT_TESTS}).')] = ExecutorConfig.DEFAULT_CONCURRENT_TESTS, max_failures: Annotated[int, typer.Option('--max-failures', '-m', min=1, max=10, help=f'Maximum number of failures to display (default: {ExecutorConfig.MAX_FAILURES_DISPLAYED}).')] = ExecutorConfig.MAX_FAILURES_DISPLAYED, progress_interval: Annotated[int, typer.Option('--progress-interval', '-p', min=1, max=10, help=f'Progress update interval (default: {ExecutorConfig.PROGRESS_UPDATE_INTERVAL}).')] = ExecutorConfig.PROGRESS_UPDATE_INTERVAL, container_config_file: Annotated[str | None, typer.Option('--container-config', help='Optional path to container configuration YAML. If not provided, built-in defaults are used.')] = None, datasets_config: Annotated[str | None, typer.Option('--datasets-config', '-d', help='Path to the datasets registry YAML file containing reusable dataset definitions.')] = None)¶: Execute the complete end-to-end workflow: tests + score cards (requires Docker).

src.asqi.main.execute_tests(test_suite_config: Annotated[str, typer.Option('--test-suite-config', '-t', help='Path to the test suite YAML file.')], systems_config: Annotated[str, typer.Option('--systems-config', '-s', help='Path to the systems YAML file.')], output_file: Annotated[str | None, typer.Option('--output-file', '-o', help='Path to save execution results JSON file.')] = 'output.json', test_ids: Annotated[list[str] | None, typer.Option('--test-ids', '-tids', help='Comma-separated list of test ids to run (matches suite test ids).')] = None, concurrent_tests: Annotated[int, typer.Option('--concurrent-tests', '-c', min=1, max=20, help=f'Number of tests to run concurrently (default: {ExecutorConfig.DEFAULT_CONCURRENT_TESTS}).')] = ExecutorConfig.DEFAULT_CONCURRENT_TESTS, max_failures: Annotated[int, typer.Option('--max-failures', '-m', min=1, max=10, help=f'Maximum number of failures to display (default: {ExecutorConfig.MAX_FAILURES_DISPLAYED}).')] = ExecutorConfig.MAX_FAILURES_DISPLAYED, progress_interval: Annotated[int, typer.Option('--progress-interval', '-p', min=1, max=10, help=f'Progress update interval (default: {ExecutorConfig.PROGRESS_UPDATE_INTERVAL}).')] = ExecutorConfig.PROGRESS_UPDATE_INTERVAL, container_config_file: Annotated[str | None, typer.Option('--container-config', help='Optional path to container configuration YAML. If not provided, built-in defaults are used.')] = None, datasets_config: Annotated[str | None, typer.Option('--datasets-config', '-d', help='Path to the datasets registry YAML file containing reusable dataset definitions.')] = None)¶: Execute only the test suite, skip score card evaluation (requires Docker).

src.asqi.main.generate_dataset(generation_config: Annotated[str, typer.Option('--generation-config', '-t', help='Path to the Generation YAML file.')], systems_config: Annotated[str | None, typer.Option('--systems-config', '-s', help='Path to the systems YAML file.')] = None, output_file: Annotated[str | None, typer.Option('--output-file', '-o', help='Path to save execution results JSON file.')] = 'output.json', concurrent_tests: Annotated[int, typer.Option('--concurrent-tests', '-c', min=1, max=20, help=f'Number of tests to run concurrently (default: {ExecutorConfig.DEFAULT_CONCURRENT_TESTS}).')] = ExecutorConfig.DEFAULT_CONCURRENT_TESTS, max_failures: Annotated[int, typer.Option('--max-failures', '-m', min=1, max=10, help=f'Maximum number of failures to display (default: {ExecutorConfig.MAX_FAILURES_DISPLAYED}).')] = ExecutorConfig.MAX_FAILURES_DISPLAYED, progress_interval: Annotated[int, typer.Option('--progress-interval', '-p', min=1, max=10, help=f'Progress update interval (default: {ExecutorConfig.PROGRESS_UPDATE_INTERVAL}).')] = ExecutorConfig.PROGRESS_UPDATE_INTERVAL, container_config_file: Annotated[str | None, typer.Option('--container-config', help='Optional path to container configuration YAML. If not provided, built-in defaults are used.')] = None, datasets_config: Annotated[str | None, typer.Option('--datasets-config', '-d', help='Path to the datasets registry YAML file containing reusable dataset definitions.')] = None)¶: Generate synthetic data using data generation containers.

src.asqi.main.evaluate_score_cards(input_file: Annotated[str, typer.Option(help='Path to JSON file with existing test results.')], score_card_config: Annotated[str, typer.Option('--score-card-config', '-r', help='Path to grading score card YAML file.')], output_file: Annotated[str | None, typer.Option('--output-file', '-o', help='Path to save evaluation results JSON file.')] = 'output_scorecard.json', audit_responses: Annotated[str | None, typer.Option('--audit-responses', '-a', help='Path to YAML file with manual audit indicator responses.')] = None, skip_audit: Annotated[bool, typer.Option('--skip-audit', help="Skip 'audit' type indicators if no audit responses are provided.")] = False)¶: Evaluate score cards against existing test results from JSON file.

src.asqi.main.typer_click_object¶