src.asqi.validation

Attributes

logger
console

Classes

ParameterType

Constants for parameter types.

Functions

normalize_types(→ List[str])	Normalize a type field to always return a list of types.
validate_ids(→ None)	Validate that all IDs within a config file are unique.
extract_ids(→ None)	Extract IDs from a config file and register them in the ID state.
get_duplicate_ids(→ Dict[str, Any])	Find only duplicate ID entries from the collected IDs.
validate_volumes(→ None)
validate_test_volumes(, require_at_least_one)	Validate per-test volumes and raise ValueError on the first problem.
validate_parameter_value(→ List[str])	Recursively validate a parameter value against its schema definition.
validate_parameters(→ List[str])	Validate job config parameters against manifest schema.
validate_dataset_configs(→ List[str])	Validate job config input_datasets against the manifest's input_datasets schema.
validate_system_compatibility(→ List[str])	Validate system compatibility with container.
find_manifest_for_image(→ Optional[asqi.schemas.Manifest])	Find manifest for a given image name.
validate_manifests_against_tests(→ List[str])	Validate that all tests can be executed with available manifests.
create_test_execution_plan(→ List[Dict[str, Any]])	Create execution plan for all valid test combinations.
validate_test_plan(→ List[str])	Validates the entire test plan by cross-referencing the suite, systems, and manifests.
validate_execution_inputs(→ None)	Validate inputs for test execution workflows.
validate_score_card_inputs(→ None)	Validate inputs for score card evaluation workflows.
validate_test_execution_inputs(→ None)	Validate inputs for individual test execution.
build_env_var_error_message(→ str)	Build helpful error message for missing environment variables.
validate_workflow_configurations(→ List[str])	Comprehensive validation of workflow configurations.
validate_indicator_display_reports(→ List[str])	Validate that all requested display_reports exist in the corresponding test container manifest and that there are no duplicate report names.
validate_generated_datasets(→ List[str])	Validate that generated datasets match the manifest declarations.
validate_data_generation_input(→ None)	Validate inputs for test execution workflows.
validate_data_gen_execution_inputs(→ None)	Validate inputs for individual test execution.
create_data_generation_plan(→ List[Dict[str, Any]])	Create execution plan for all valid test combinations.
validate_data_generation_plan(→ List[str])	Validate that all generation jobs can be executed with available manifests.
validate_data_generation_volumes(, require_at_least_one)	Validate per-test volumes and raise ValueError on the first problem.
resolve_dataset_references(...)	Resolve dataset name references to actual dataset configurations.

Module Contents

src.asqi.validation.logger

src.asqi.validation.console

class src.asqi.validation.ParameterType

Constants for parameter types.

STRING = 'string'

INTEGER = 'integer'

FLOAT = 'float'

BOOLEAN = 'boolean'

ENUM = 'enum'

LIST = 'list'

OBJECT = 'object'

src.asqi.validation.normalize_types(type_input: str | List[str]) → List[str]

Normalize a type field to always return a list of types.

Args:

type_input: Either a single type string or a list of type strings.

Can be a plain string, StrEnum, or list of either.

Returns:

List of type strings (normalized to str for consistent comparison)

Examples:

>>> normalize_types("llm_api")
["llm_api"]
>>> normalize_types(["llm_api", "vlm_api"])
["llm_api", "vlm_api"]

src.asqi.validation.validate_ids(*config_paths: str) → None

Validate that all IDs within a config file are unique. Supports multiple config files and reports duplicates found across all of them.

Args:

*config_paths: Variable number of configuration file paths to validate

Notes:

Only supports:

• test suite configs

• score card configs

Raises:

DuplicateIDError: If duplicate IDs are found MissingIDFieldError: If required ID fields are missing

src.asqi.validation.extract_ids(all_ids: Dict[str, Any], config_dict: Dict[str, Any], config_path: str) → None

Extract IDs from a config file and register them in the ID state.

Args:

all_ids: Dictionary tracking all collected IDs config_dict: Parsed configuration dictionary config_path: Path to the config file

Notes:

Only processes IDs from:

• score cards

• test suites

src.asqi.validation.get_duplicate_ids(all_ids: Dict[str, Any]) → Dict[str, Any]

Find only duplicate ID entries from the collected IDs.

Args:

all_ids: Current state of the project IDs

Returns:

Dictionary containing duplicate groups based on their key (ID and file type). Returns an empty dictionary if no duplicates found.

src.asqi.validation.validate_volumes(name: str, vols: dict | None, allowed: set[str], require_at_least_one: bool = True) → None

src.asqi.validation.validate_test_volumes(suite: asqi.schemas.SuiteConfig, allowed_keys: tuple[str, Ellipsis] = ('input', 'output'), require_at_least_one: bool = True) → None

Validate per-test volumes and raise ValueError on the first problem.

Rules: - volumes may be omitted entirely (skip) - if present, must be a dict - require_at_least_one=True => at least one of allowed_keys must be present - only validate keys that are present - each provided path must be a non-empty string, exist, and be a directory

src.asqi.validation.validate_parameter_value(param_name: str, param_value: Any, schema_param: asqi.schemas.InputParameter, context_path: str = '') → List[str]

Recursively validate a parameter value against its schema definition.

Args:

param_name: Name of the parameter being validated param_value: Actual value provided by the user schema_param: InputParameter schema definition context_path: Parent path for nested validation (e.g., “api_config”)

Returns:

List of validation error messages

src.asqi.validation.validate_parameters(item: asqi.schemas.TestDefinition | asqi.schemas.GenerationJobConfig, manifest: asqi.schemas.Manifest) → List[str]

Validate job config parameters against manifest schema.

Performs comprehensive validation including: - Presence validation (required parameters exist) - Unknown parameter detection - Type validation (string, integer, float, boolean, enum, list, object) - Enum choice validation - Recursive validation for nested structures (lists, objects)

Args:

item: Test definition or generation job config with params manifest: Manifest for the container

Returns:

List of validation error messages

src.asqi.validation.validate_dataset_configs(item: asqi.schemas.TestDefinition | asqi.schemas.GenerationJobConfig, manifest: asqi.schemas.Manifest) → List[str]

Validate job config input_datasets against the manifest’s input_datasets schema.

• Ensures required datasets in input_datasets are provided.

• Ensures no unknown dataset names are provided in item.input_datasets.

• Ensures dataset types match manifest expectations (single type or one of multiple accepted types).

Args:

item: Test definition or generation job config with input_datasets.

Note: input_datasets should be resolved dataset definitions (DatasetDefinition objects), not string references. Call resolve_dataset_references() first if needed.

manifest: Manifest defining the expected dataset schema

Returns:

List of validation error messages

src.asqi.validation.validate_system_compatibility(item: asqi.schemas.TestDefinition | asqi.schemas.GenerationJobConfig, system_definitions: Dict, manifest: asqi.schemas.Manifest) → List[str]

Validate system compatibility with container.

Args:

item: Test definition or generation job config with systems system_definitions: Dictionary of system definitions manifest: Manifest for the container

Returns:

List of validation error messages

src.asqi.validation.find_manifest_for_image(image_name: str, manifests: Dict[str, asqi.schemas.Manifest]) → asqi.schemas.Manifest | None

Find manifest for a given image name.

For runtime (workflow), manifests are keyed by full image names. For local validation, manifests are keyed by container directory names.

Args:

image_name: Full image name (e.g., “my-registry/mock_tester:latest”) manifests: Dictionary of available manifests

Returns:

Manifest if found, None otherwise

src.asqi.validation.validate_manifests_against_tests(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest]) → List[str]

Validate that all tests can be executed with available manifests.

Args:

suite: Test suite configuration systems: Systems configuration manifests: Dictionary of available manifests

Returns:

List of validation error messages

src.asqi.validation.create_test_execution_plan(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, image_availability: Dict[str, bool]) → List[Dict[str, Any]]

Create execution plan for all valid test combinations.

Args:

suite: Test suite configuration systems: Systems configuration image_availability: Dictionary of image availability status

Returns:

List of test execution plans

src.asqi.validation.validate_test_plan(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest]) → List[str]

Validates the entire test plan by cross-referencing the suite, systems, and manifests.

Args:

suite: The parsed SuiteConfig object. systems: The parsed systems configuration object. manifests: A dictionary mapping image names to their parsed Manifest objects.

Returns:

A list of error strings. An empty list indicates successful validation.

src.asqi.validation.validate_execution_inputs(suite_path: str, systems_path: str, execution_mode: asqi.config.ExecutionMode, audit_responses_data: Dict[str, Any] | None = None, output_path: str | None = None) → None

Validate inputs for test execution workflows.

Args:

suite_path: Path to test suite YAML file systems_path: Path to systems YAML file execution_mode: Execution mode audit_responses_data: Optional dictionary of audit responses data output_path: Optional output file path

Raises:

ValueError: If any input is invalid

src.asqi.validation.validate_score_card_inputs(input_path: str, score_card_configs: List[Dict[str, Any]], audit_responses_data: Dict[str, Any] | None = None, output_path: str | None = None) → None

Validate inputs for score card evaluation workflows.

Args:

input_path: Path to input JSON file score_card_configs: List of score card configurations audit_responses_data: Optional dictionary of audit responses data output_path: Optional output file path

Raises:

ValueError: If any input is invalid

src.asqi.validation.validate_test_execution_inputs(test_id: str, image: str, system_name: str, system_params: Dict[str, Any], test_params: Dict[str, Any]) → None

Validate inputs for individual test execution.

Args:

test_id: ID of the test image: Docker image name system_name: Name of the system system_params: System parameters dictionary (flattened configuration) test_params: Test parameters dictionary

Raises:

ValueError: If any input is invalid

src.asqi.validation.build_env_var_error_message(missing_vars: List[asqi.schemas.EnvironmentVariable], test_name: str, image: str) → str

Build helpful error message for missing environment variables.

Args:

missing_vars: List of EnvironmentVariable objects that are missing test_name: Name of the test being executed image: Docker image being used

Returns:

Formatted error message with suggestions for fixing the issue

src.asqi.validation.validate_workflow_configurations(suite: asqi.schemas.SuiteConfig, systems: asqi.schemas.SystemsConfig, manifests: Dict[str, asqi.schemas.Manifest] | None = None) → List[str]

Comprehensive validation of workflow configurations.

Combines all configuration validation checks in one place.

Args:

suite: Test suite configuration systems: Systems configuration manifests: Optional manifests dictionary

Returns:

List of validation error messages

Raises:

ValueError: If configuration objects are invalid

src.asqi.validation.validate_indicator_display_reports(manifests: Dict[str, asqi.schemas.Manifest], score_cards: List[asqi.schemas.ScoreCard], test_id_to_image: Dict[str, str]) → List[str]

Validate that all requested display_reports exist in the corresponding test container manifest and that there are no duplicate report names.

Args:

manifests: Dictionary linking each image to its manifest score_cards: List of score card configurations to validate test_id_to_image: Dictionary linking each test id to the image used

Returns:

List of validation error messages or empty list if none found

src.asqi.validation.validate_generated_datasets(manifest: asqi.schemas.Manifest, generated_datasets: List[asqi.response_schemas.GeneratedDataset], test_id: str, image: str) → List[str]

Validate that generated datasets match the manifest declarations.

Checks that all dataset names in generated_datasets exist in the manifest’s output_datasets field. Does not validate features/schema.

Args:

manifest: The container manifest generated_datasets: List of GeneratedDataset objects test_id: ID of the test/job for error messages image: Container image name for error messages

Returns:

List of warning messages (empty list if all datasets are valid)

src.asqi.validation.validate_data_generation_input(generation_config_path: str, systems_path: str | None, output_path: str | None = None) → None

Validate inputs for test execution workflows.

Args:

suite_path: Path to test suite YAML file systems_path: Path to systems YAML file execution_mode: Execution mode audit_responses_data: Optional dictionary of audit responses data output_path: Optional output file path

Raises:

ValueError: If any input is invalid

src.asqi.validation.validate_data_gen_execution_inputs(job_id: str, image: str, system_params: Dict[str, Any], generation_params: Dict[str, Any]) → None

Validate inputs for individual test execution.

Args:

test_id: ID of the test image: Docker image name system_name: Name of the system system_params: System parameters dictionary (flattened configuration) test_params: Test parameters dictionary

Raises:

ValueError: If any input is invalid

src.asqi.validation.create_data_generation_plan(data_generation_config: asqi.schemas.DataGenerationConfig, systems: asqi.schemas.SystemsConfig | None, image_availability: Dict[str, bool]) → List[Dict[str, Any]]

Create execution plan for all valid test combinations.

Args:

suite: Test suite configuration systems: Systems configuration (optional) image_availability: Dictionary of image availability status

Returns:

List of test execution plans

src.asqi.validation.validate_data_generation_plan(data_generation_config: asqi.schemas.DataGenerationConfig, systems: asqi.schemas.SystemsConfig | None, manifests: Dict[str, asqi.schemas.Manifest]) → List[str]

Validate that all generation jobs can be executed with available manifests.

Args:

data_generation_config: Data generation configuration systems: Systems configuration (optional, generation jobs may not need systems) manifests: Dictionary mapping image names to their parsed Manifest objects

Returns:

List of validation error messages (empty list indicates successful validation)

src.asqi.validation.validate_data_generation_volumes(generation_config: asqi.schemas.DataGenerationConfig, allowed_keys: tuple[str, Ellipsis] = ('input', 'output'), require_at_least_one: bool = True) → None

Validate per-test volumes and raise ValueError on the first problem.

Rules: - volumes may be omitted entirely (skip) - if present, must be a dict - require_at_least_one=True => at least one of allowed_keys must be present - only validate keys that are present - each provided path must be a non-empty string, exist, and be a directory

src.asqi.validation.resolve_dataset_references(config: asqi.schemas.SuiteConfig | asqi.schemas.DataGenerationConfig, datasets_config: asqi.schemas.DatasetsConfig) → asqi.schemas.SuiteConfig | asqi.schemas.DataGenerationConfig

Resolve dataset name references to actual dataset configurations.

Args:

config: Suite or generation config (Pydantic model) datasets_config: Datasets configuration to resolve from

Returns:

Config with resolved dataset references

Raises:

ValueError: If referenced dataset not found in datasets_config