src.asqi.main

Attributes

console
app
typer_click_object

Functions

load_yaml_file(→ Dict[str, Any])	Loads a YAML file with environment variable interpolation.
load_score_card_file(→ Dict[str, Any])	Load and validate grading score card configuration.
load_and_validate_plan(→ Dict[str, Any])	Performs all validation and returns a structured result.
validate([test_suite_config, systems_config, ...])	Validate test plan configuration without execution.
execute(test_suite_config, systems_config, ...)	Execute the complete end-to-end workflow: tests + score cards (requires Docker).
execute_tests(test_suite_config, systems_config, ...)	Execute only the test suite, skip score card evaluation (requires Docker).
evaluate_score_cards([input_file, score_card_config, ...])	Evaluate score cards against existing test results from JSON file.

Module Contents

src.asqi.main.console

src.asqi.main.load_yaml_file(file_path: str) → Dict[str, Any]

Loads a YAML file with environment variable interpolation.

Args:

file_path: Path to the YAML file to load

Returns:

Dictionary containing the parsed YAML data with environment variables interpolated

Raises:

FileNotFoundError: If the specified file does not exist ValueError: If the YAML file contains invalid syntax or cannot be parsed PermissionError: If the file cannot be read due to permissions

src.asqi.main.load_score_card_file(score_card_path: str) → Dict[str, Any]

Load and validate grading score card configuration.

Args:

score_card_path: Path to the score card YAML file

Returns:

Dictionary containing the validated score card configuration

Raises:

FileNotFoundError: If the score card file does not exist ValueError: If the YAML is invalid or score card schema validation fails PermissionError: If the file cannot be read due to permissions

src.asqi.main.load_and_validate_plan(suite_path: str, systems_path: str, manifests_path: str) → Dict[str, Any]

Performs all validation and returns a structured result. This function is pure and does not print or exit.

Returns:

A dictionary, e.g., {“status”: “success”, “errors”: []} or {“status”: “failure”, “errors”: [“error message”]}.

src.asqi.main.app

src.asqi.main.validate(test_suite_config: str = typer.Option(..., '--test-suite-config', '-t', help='Path to the test suite YAML file.'), systems_config: str = typer.Option(..., '--systems-config', '-s', help='Path to the systems YAML file.'), manifests_dir: str = typer.Option(..., help='Path to dir with test container manifests.'))

Validate test plan configuration without execution.

src.asqi.main.execute(test_suite_config: str = typer.Option(..., '--test-suite-config', '-t', help='Path to the test suite YAML file.'), systems_config: str = typer.Option(..., '--systems-config', '-s', help='Path to the systems YAML file.'), score_card_config: str = typer.Option(..., '--score-card-config', '-r', help='Path to grading score card YAML file.'), output_file: Optional[str] = typer.Option('output_scorecard.json', '--output-file', '-o', help='Path to save execution results JSON file.'), concurrent_tests: int = typer.Option(ExecutorConfig.DEFAULT_CONCURRENT_TESTS, '--concurrent-tests', '-c', min=1, max=20, help=f'Number of tests to run concurrently (must be between 1 and 20, default: {ExecutorConfig.DEFAULT_CONCURRENT_TESTS})'), max_failures: int = typer.Option(ExecutorConfig.MAX_FAILURES_DISPLAYED, '--max-failures', '-m', min=1, max=10, help=f'Maximum number of failures to display (must be between 1 and 10, default: {ExecutorConfig.MAX_FAILURES_DISPLAYED}).'), progress_interval: int = typer.Option(ExecutorConfig.PROGRESS_UPDATE_INTERVAL, '--progress-interval', '-p', min=1, max=10, help=f'Progress update interval (must be between 1 and 10, default: {ExecutorConfig.PROGRESS_UPDATE_INTERVAL}).'), container_config_file: Optional[str] = typer.Option(None, '--container-config', help='Optional path to container configuration YAML. If not provided, built-in defaults are used.'))

Execute the complete end-to-end workflow: tests + score cards (requires Docker).

src.asqi.main.execute_tests(test_suite_config: str = typer.Option(..., '--test-suite-config', '-t', help='Path to the test suite YAML file.'), systems_config: str = typer.Option(..., '--systems-config', '-s', help='Path to the systems YAML file.'), output_file: Optional[str] = typer.Option('output.json', '--output-file', '-o', help='Path to save execution results JSON file.'), test_names: Optional[List[str]] = typer.Option(None, '--test-names', '-tn', help='Comma-separated list of test names to run (matches suite test names).'), concurrent_tests: int = typer.Option(ExecutorConfig.DEFAULT_CONCURRENT_TESTS, '--concurrent-tests', '-c', min=1, max=20, help=f'Number of tests to run concurrently (must be between 1 and 20, default: {ExecutorConfig.DEFAULT_CONCURRENT_TESTS})'), max_failures: int = typer.Option(ExecutorConfig.MAX_FAILURES_DISPLAYED, '--max-failures', '-m', min=1, max=10, help=f'Maximum number of failures to display (must be between 1 and 10, default: {ExecutorConfig.MAX_FAILURES_DISPLAYED}).'), progress_interval: int = typer.Option(ExecutorConfig.PROGRESS_UPDATE_INTERVAL, '--progress-interval', '-p', min=1, max=10, help=f'Progress update interval (must be between 1 and 10, default: {ExecutorConfig.PROGRESS_UPDATE_INTERVAL}).'), container_config_file: Optional[str] = typer.Option(None, '--container-config', help='Optional path to container configuration YAML. If not provided, built-in defaults are used.'))

Execute only the test suite, skip score card evaluation (requires Docker).

src.asqi.main.evaluate_score_cards(input_file: str = typer.Option(..., help='Path to JSON file with existing test results.'), score_card_config: str = typer.Option(..., '--score-card-config', '-r', help='Path to grading score card YAML file.'), output_file: str | None = typer.Option('output_scorecard.json', '--output-file', '-o', help='Path to save evaluation results JSON file.'))

Evaluate score cards against existing test results from JSON file.

src.asqi.main.typer_click_object