src.asqi.validation

Attributes

logger
console

Functions

validate_ids(→ None)	Validate that all IDs within a config file are unique.
extract_ids(→ None)	Extract IDs from a config file and register them in the ID state.
get_duplicate_ids(→ Dict[str, Any])	Find only duplicate ID entries from the collected IDs.
validate_test_volumes(, require_at_least_one)	Validate per-test volumes and raise ValueError on the first problem.
validate_test_parameters(→ List[str])	Validate test parameters against manifest schema.
validate_system_compatibility(→ List[str])	Validate system compatibility with test container.
find_manifest_for_image(→ Optional[asqi.schemas.Manifest])	Find manifest for a given image name.
validate_manifests_against_tests(→ List[str])	Validate that all tests can be executed with available manifests.
create_test_execution_plan(→ List[Dict[str, Any]])	Create execution plan for all valid test combinations.
validate_test_plan(→ List[str])	Validates the entire test plan by cross-referencing the suite, systems, and manifests.
validate_execution_inputs(→ None)	Validate inputs for test execution workflows.
validate_score_card_inputs(→ None)	Validate inputs for score card evaluation workflows.
validate_test_execution_inputs(→ None)	Validate inputs for individual test execution.
build_env_var_error_message(→ str)	Build helpful error message for missing environment variables.
validate_workflow_configurations(→ List[str])	Comprehensive validation of workflow configurations.

Module Contents

src.asqi.validation.logger

src.asqi.validation.console

src.asqi.validation.validate_ids(*config_paths: str) → None

Validate that all IDs within a config file are unique. Supports multiple config files and reports duplicates found across all of them.

Args:

*config_paths: Variable number of configuration file paths to validate

Notes:

Only supports:

• test suite configs

• score card configs

Raises:

DuplicateIDError: If duplicate IDs are found MissingIDFieldError: If required ID fields are missing

src.asqi.validation.extract_ids(all_ids: Dict[str, Any], config_dict: Dict[str, Any], config_path: str) → None

Extract IDs from a config file and register them in the ID state.

Args:

all_ids: Dictionary tracking all collected IDs config_dict: Parsed configuration dictionary config_path: Path to the config file

Notes:

Only processes IDs from:

• score cards

• test suites

src.asqi.validation.get_duplicate_ids(all_ids: Dict[str, Any]) → Dict[str, Any]

Find only duplicate ID entries from the collected IDs.

Args:

all_ids: Current state of the project IDs

Returns:

Dictionary containing duplicate groups based on their key (ID and file type). Returns an empty dictionary if no duplicates found.

src.asqi.validation.validate_test_volumes(suite: asqi.schemas.SuiteConfig, allowed_keys: tuple[str, Ellipsis] = ('input', 'output'), require_at_least_one: bool = True) → None

Validate per-test volumes and raise ValueError on the first problem.

Rules: - volumes may be omitted entirely (skip) - if present, must be a dict - require_at_least_one=True => at least one of allowed_keys must be present - only validate keys that are present - each provided path must be a non-empty string, exist, and be a directory

src.asqi.validation.validate_test_parameters(test, manifest: asqi.schemas.Manifest) → List[str]

Validate test parameters against manifest schema.

Args:

test: Test definition from suite config manifest: Manifest for the test container

Returns:

List of validation error messages

src.asqi.validation.validate_system_compatibility(test, system_definitions: Dict, manifest: asqi.schemas.Manifest) → List[str]

Validate system compatibility with test container.

Args:

test: Test definition from suite config system_definitions: Dictionary of system definitions manifest: Manifest for the test container

Returns:

List of validation error messages

src.asqi.validation.find_manifest_for_image(image_name: str, manifests: Dict[str, asqi.schemas.Manifest]) → asqi.schemas.Manifest | None

Find manifest for a given image name.

For runtime (workflow), manifests are keyed by full image names. For local validation, manifests are keyed by container directory names.

Args:

image_name: Full image name (e.g., “my-registry/mock_tester:latest”) manifests: Dictionary of available manifests

Returns:

Manifest if found, None otherwise

src.asqi.validation.validate_manifests_against_tests(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest]) → List[str]

Validate that all tests can be executed with available manifests.

Args:

suite: Test suite configuration systems: Systems configuration manifests: Dictionary of available manifests

Returns:

List of validation error messages

src.asqi.validation.create_test_execution_plan(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, image_availability: Dict[str, bool]) → List[Dict[str, Any]]

Create execution plan for all valid test combinations.

Args:

suite: Test suite configuration systems: Systems configuration image_availability: Dictionary of image availability status

Returns:

List of test execution plans

src.asqi.validation.validate_test_plan(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest]) → List[str]

Validates the entire test plan by cross-referencing the suite, systems, and manifests.

Args:

suite: The parsed SuiteConfig object. systems: The parsed systems configuration object. manifests: A dictionary mapping image names to their parsed Manifest objects.

Returns:

A list of error strings. An empty list indicates successful validation.

src.asqi.validation.validate_execution_inputs(suite_path: str, systems_path: str, execution_mode: str, audit_responses_data: Dict[str, Any] | None = None, output_path: str | None = None) → None

Validate inputs for test execution workflows.

Args:

suite_path: Path to test suite YAML file systems_path: Path to systems YAML file execution_mode: Execution mode string audit_responses_data: Optional dictionary of audit responses data output_path: Optional output file path

Raises:

ValueError: If any input is invalid

src.asqi.validation.validate_score_card_inputs(input_path: str, score_card_configs: List[Dict[str, Any]], audit_responses_data: Dict[str, Any] | None = None, output_path: str | None = None) → None

Validate inputs for score card evaluation workflows.

Args:

input_path: Path to input JSON file score_card_configs: List of score card configurations audit_responses_data: Optional dictionary of audit responses data output_path: Optional output file path

Raises:

ValueError: If any input is invalid

src.asqi.validation.validate_test_execution_inputs(test_id: str, image: str, system_name: str, system_params: Dict[str, Any], test_params: Dict[str, Any]) → None

Validate inputs for individual test execution.

Args:

test_id: ID of the test image: Docker image name system_name: Name of the system system_params: System parameters dictionary (flattened configuration) test_params: Test parameters dictionary

Raises:

ValueError: If any input is invalid

src.asqi.validation.build_env_var_error_message(missing_vars: List[asqi.schemas.EnvironmentVariable], test_name: str, image: str) → str

Build helpful error message for missing environment variables.

Args:

missing_vars: List of EnvironmentVariable objects that are missing test_name: Name of the test being executed image: Docker image being used

Returns:

Formatted error message with suggestions for fixing the issue

src.asqi.validation.validate_workflow_configurations(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest] | None = None) → List[str]

Comprehensive validation of workflow configurations.

Combines all configuration validation checks in one place.

Args:

suite: Test suite configuration systems: Systems configuration manifests: Optional manifests dictionary

Returns:

List of validation error messages

Raises:

ValueError: If configuration objects are invalid