src.asqi.main¶

Attributes¶

`console`
`app`
`typer_click_object`

Functions¶

`load_yaml_file`(→ Dict[str, Any])	Loads a YAML file with environment variable interpolation.
`load_score_card_file`(→ Dict[str, Any])	Load and validate grading score card configuration.
`load_audit_responses_file`(→ Dict[str, Any])	Load and validate audit responses YAML file.
`resolve_audit_options`(→ tuple[Dict[str, Any], ...)	Handles all validation and preparation for audit indicators.
`load_and_validate_plan`(→ Dict[str, Any])	Performs all validation and returns a structured result.
`version_callback`(value)	Display version information and exit.
`validate`([test_suite_config, systems_config, ...])	Validate test plan configuration without execution.
`execute`(test_suite_config, systems_config, ...)	Execute the complete end-to-end workflow: tests + score cards (requires Docker).
`execute_tests`(test_suite_config, systems_config, ...)	Execute only the test suite, skip score card evaluation (requires Docker).
`evaluate_score_cards`([input_file, score_card_config, ...])	Evaluate score cards against existing test results from JSON file.

Module Contents¶

src.asqi.main.console¶

src.asqi.main.load_yaml_file(file_path: str) → Dict[str, Any]¶

Loads a YAML file with environment variable interpolation.

Args:: file_path: Path to the YAML file to load
Returns:: Dictionary containing the parsed YAML data with environment variables interpolated
Raises:: FileNotFoundError: If the specified file does not exist ValueError: If the YAML file contains invalid syntax or cannot be parsed PermissionError: If the file cannot be read due to permissions

src.asqi.main.load_score_card_file(score_card_path: str) → Dict[str, Any]¶

Load and validate grading score card configuration.

Args:: score_card_path: Path to the score card YAML file
Returns:: Dictionary containing the validated score card configuration
Raises:: FileNotFoundError: If the score card file does not exist ValueError: If the YAML is invalid or score card schema validation fails PermissionError: If the file cannot be read due to permissions

src.asqi.main.load_audit_responses_file(audit_responses_path: str) → Dict[str, Any]¶: Load and validate audit responses YAML file.

src.asqi.main.resolve_audit_options(score_card_data: Dict[str, Any], audit_responses_path: str | None, skip_audit_flag: bool) → tuple[Dict[str, Any], Dict[str, Any] | None]¶

Handles all validation and preparation for audit indicators.

Returns:

updated_score_card_data
audit_responses_data (or None)

Raises:

AuditResponsesRequiredError: If audit indicators exist but no responses provided

src.asqi.main.load_and_validate_plan(suite_path: str, systems_path: str, manifests_path: str) → Dict[str, Any]¶

Performs all validation and returns a structured result. This function is pure and does not print or exit.

Returns:: A dictionary, e.g., {“status”: “success”, “errors”: []} or {“status”: “failure”, “errors”: [“error message”]}.

src.asqi.main.app¶

src.asqi.main.version_callback(value: bool)¶: Display version information and exit.

src.asqi.main.validate(test_suite_config: str = typer.Option(..., '--test-suite-config', '-t', help='Path to the test suite YAML file.'), systems_config: str = typer.Option(..., '--systems-config', '-s', help='Path to the systems YAML file.'), manifests_dir: str = typer.Option(..., help='Path to dir with test container manifests.'))¶: Validate test plan configuration without execution.

src.asqi.main.execute(test_suite_config: str = typer.Option(..., '--test-suite-config', '-t', help='Path to the test suite YAML file.'), systems_config: str = typer.Option(..., '--systems-config', '-s', help='Path to the systems YAML file.'), score_card_config: str = typer.Option(..., '--score-card-config', '-r', help='Path to grading score card YAML file.'), output_file: Optional[str] = typer.Option('output_scorecard.json', '--output-file', '-o', help='Path to save execution results JSON file.'), audit_responses: Optional[str] = typer.Option(None, '--audit-responses', '-a', help='Path to YAML file with manual audit indicator responses.'), skip_audit: bool = typer.Option(False, '--skip-audit', help="Skip 'audit' type indicators if no audit responses are provided."), concurrent_tests: int = typer.Option(ExecutorConfig.DEFAULT_CONCURRENT_TESTS, '--concurrent-tests', '-c', min=1, max=20, help=f'Number of tests to run concurrently (must be between 1 and 20, default: {ExecutorConfig.DEFAULT_CONCURRENT_TESTS})'), max_failures: int = typer.Option(ExecutorConfig.MAX_FAILURES_DISPLAYED, '--max-failures', '-m', min=1, max=10, help=f'Maximum number of failures to display (must be between 1 and 10, default: {ExecutorConfig.MAX_FAILURES_DISPLAYED}).'), progress_interval: int = typer.Option(ExecutorConfig.PROGRESS_UPDATE_INTERVAL, '--progress-interval', '-p', min=1, max=10, help=f'Progress update interval (must be between 1 and 10, default: {ExecutorConfig.PROGRESS_UPDATE_INTERVAL}).'), container_config_file: Optional[str] = typer.Option(None, '--container-config', help='Optional path to container configuration YAML. If not provided, built-in defaults are used.'))¶: Execute the complete end-to-end workflow: tests + score cards (requires Docker).

src.asqi.main.execute_tests(test_suite_config: str = typer.Option(..., '--test-suite-config', '-t', help='Path to the test suite YAML file.'), systems_config: str = typer.Option(..., '--systems-config', '-s', help='Path to the systems YAML file.'), output_file: Optional[str] = typer.Option('output.json', '--output-file', '-o', help='Path to save execution results JSON file.'), test_ids: Optional[List[str]] = typer.Option(None, '--test-ids', '-tids', help='Comma-separated list of test ids to run (matches suite test ids).'), concurrent_tests: int = typer.Option(ExecutorConfig.DEFAULT_CONCURRENT_TESTS, '--concurrent-tests', '-c', min=1, max=20, help=f'Number of tests to run concurrently (must be between 1 and 20, default: {ExecutorConfig.DEFAULT_CONCURRENT_TESTS})'), max_failures: int = typer.Option(ExecutorConfig.MAX_FAILURES_DISPLAYED, '--max-failures', '-m', min=1, max=10, help=f'Maximum number of failures to display (must be between 1 and 10, default: {ExecutorConfig.MAX_FAILURES_DISPLAYED}).'), progress_interval: int = typer.Option(ExecutorConfig.PROGRESS_UPDATE_INTERVAL, '--progress-interval', '-p', min=1, max=10, help=f'Progress update interval (must be between 1 and 10, default: {ExecutorConfig.PROGRESS_UPDATE_INTERVAL}).'), container_config_file: Optional[str] = typer.Option(None, '--container-config', help='Optional path to container configuration YAML. If not provided, built-in defaults are used.'))¶: Execute only the test suite, skip score card evaluation (requires Docker).

src.asqi.main.evaluate_score_cards(input_file: str = typer.Option(..., help='Path to JSON file with existing test results.'), score_card_config: str = typer.Option(..., '--score-card-config', '-r', help='Path to grading score card YAML file.'), output_file: str | None = typer.Option('output_scorecard.json', '--output-file', '-o', help='Path to save evaluation results JSON file.'), audit_responses: str | None = typer.Option(None, '--audit-responses', '-a', help='Path to YAML file with manual audit indicator responses.'), skip_audit: bool = typer.Option(False, '--skip-audit', help="Skip 'audit' type indicators if no audit responses are provided."))¶: Evaluate score cards against existing test results from JSON file.

src.asqi.main.typer_click_object¶