src.asqi.validation

Attributes

logger

Functions

validate_test_volumes(, require_at_least_one)	Validate per-test volumes and raise ValueError on the first problem.
validate_test_parameters(→ List[str])	Validate test parameters against manifest schema.
validate_system_compatibility(→ List[str])	Validate system compatibility with test container.
find_manifest_for_image(→ Optional[asqi.schemas.Manifest])	Find manifest for a given image name.
validate_manifests_against_tests(→ List[str])	Validate that all tests can be executed with available manifests.
create_test_execution_plan(→ List[Dict[str, Any]])	Create execution plan for all valid test combinations.
validate_test_plan(→ List[str])	Validates the entire test plan by cross-referencing the suite, systems, and manifests.
validate_execution_inputs(→ None)	Validate inputs for test execution workflows.
validate_score_card_inputs(→ None)	Validate inputs for score card evaluation workflows.
validate_test_execution_inputs(→ None)	Validate inputs for individual test execution.
validate_workflow_configurations(→ List[str])	Comprehensive validation of workflow configurations.

Module Contents

src.asqi.validation.logger

src.asqi.validation.validate_test_volumes(suite: asqi.schemas.SuiteConfig, allowed_keys: tuple[str, Ellipsis] = ('input', 'output'), require_at_least_one: bool = True) → None

Validate per-test volumes and raise ValueError on the first problem.

Rules: - volumes may be omitted entirely (skip) - if present, must be a dict - require_at_least_one=True => at least one of allowed_keys must be present - only validate keys that are present - each provided path must be a non-empty string, exist, and be a directory

src.asqi.validation.validate_test_parameters(test, manifest: asqi.schemas.Manifest) → List[str]

Validate test parameters against manifest schema.

Args:

test: Test definition from suite config manifest: Manifest for the test container

Returns:

List of validation error messages

src.asqi.validation.validate_system_compatibility(test, system_definitions: Dict, manifest: asqi.schemas.Manifest) → List[str]

Validate system compatibility with test container.

Args:

test: Test definition from suite config system_definitions: Dictionary of system definitions manifest: Manifest for the test container

Returns:

List of validation error messages

src.asqi.validation.find_manifest_for_image(image_name: str, manifests: Dict[str, asqi.schemas.Manifest]) → asqi.schemas.Manifest | None

Find manifest for a given image name.

For runtime (workflow), manifests are keyed by full image names. For local validation, manifests are keyed by container directory names.

Args:

image_name: Full image name (e.g., “my-registry/mock_tester:latest”) manifests: Dictionary of available manifests

Returns:

Manifest if found, None otherwise

src.asqi.validation.validate_manifests_against_tests(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest]) → List[str]

Validate that all tests can be executed with available manifests.

Args:

suite: Test suite configuration systems: Systems configuration manifests: Dictionary of available manifests

Returns:

List of validation error messages

src.asqi.validation.create_test_execution_plan(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, image_availability: Dict[str, bool]) → List[Dict[str, Any]]

Create execution plan for all valid test combinations.

Args:

suite: Test suite configuration systems: Systems configuration image_availability: Dictionary of image availability status

Returns:

List of test execution plans

src.asqi.validation.validate_test_plan(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest]) → List[str]

Validates the entire test plan by cross-referencing the suite, systems, and manifests.

Args:

suite: The parsed SuiteConfig object. systems: The parsed systems configuration object. manifests: A dictionary mapping image names to their parsed Manifest objects.

Returns:

A list of error strings. An empty list indicates successful validation.

src.asqi.validation.validate_execution_inputs(suite_path: str, systems_path: str, execution_mode: str, output_path: str | None = None) → None

Validate inputs for test execution workflows.

Args:

suite_path: Path to test suite YAML file systems_path: Path to systems YAML file execution_mode: Execution mode string output_path: Optional output file path

Raises:

ValueError: If any input is invalid

src.asqi.validation.validate_score_card_inputs(input_path: str, score_card_configs: List[Dict[str, Any]], output_path: str | None = None) → None

Validate inputs for score card evaluation workflows.

Args:

input_path: Path to input JSON file score_card_configs: List of score card configurations output_path: Optional output file path

Raises:

ValueError: If any input is invalid

src.asqi.validation.validate_test_execution_inputs(test_name: str, image: str, system_name: str, system_params: Dict[str, Any], test_params: Dict[str, Any]) → None

Validate inputs for individual test execution.

Args:

test_name: Name of the test image: Docker image name system_name: Name of the system system_params: System parameters dictionary (flattened configuration) test_params: Test parameters dictionary

Raises:

ValueError: If any input is invalid

src.asqi.validation.validate_workflow_configurations(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest] | None = None) → List[str]

Comprehensive validation of workflow configurations.

Combines all configuration validation checks in one place.

Args:

suite: Test suite configuration systems: Systems configuration manifests: Optional manifests dictionary

Returns:

List of validation error messages

Raises:

ValueError: If configuration objects are invalid