src.asqi.schemas¶

Attributes¶

`IDsStringPattern`
`HFDtype`
`HFFeature`
`SystemConfig`
`LabelMap`
`DatasetDefinition`
`TestCase`

Classes¶

`ValueFeature`	Corresponds to HuggingFace's Value feature type.
`ListFeature`	Corresponds to HuggingFace's List/Sequence feature type.
`DictFeature`	Corresponds to HuggingFace's "Python dict" feature type.
`ClassLabelFeature`	Corresponds to HuggingFace's ClassLabel feature type. Represents categorical data with named categories.
`ImageFeature`	Corresponds to HuggingFace's Image feature type.
`AudioFeature`	Corresponds to HuggingFace's Audio feature type.
`VideoFeature`	Corresponds to HuggingFace's Video feature type.
`DatasetFeature`	Defines a feature/column within a dataset.
`SystemInput`	Defines a system input that the container requires.
`InputParameter`	Defines a parameter that can be passed to the test container.
`OutputMetric`	Defines a metric that will be present in the test container's output.
`OutputArtifact`	Defines a file artifact generated by the test container.
`EnvironmentVariable`	Defines an environment variable required by the test container.
`DatasetType`	Supported dataset types for InputDataset.
`InputDataset`	Defines a dataset input that the container requires.
`OutputReports`	Defines a report that will be generated by the test container.
`OutputDataset`	Defines a dataset output that will be generated by the test container.
`Manifest`	Schema for the manifest.yaml file inside a test container.
`SystemDefinition`	Base system definition.
`ThinkingParams`	Extended reasoning configuration for models that support it (e.g., Claude Opus 4.6 etc.).
`LLMAPIParams`	Parameters for the LLM API systems.
`VLMAPIParams`	Parameters for Vision Language Model API systems.
`AgentCLIParams`	Parameters for Agent CLI systems.
`LLMAPIConfig`	Configuration for LLM API systems.
`EmbeddingAPIConfig`	Configuration for Embedding API systems.
`CustomParams`	Parameters for custom systems (e.g., SAP analytics, proprietary APIs).
`CustomConfig`	Configuration for custom systems (e.g., SAP analytics, proprietary APIs).
`RAGAPIConfig`	Configuration for RAG API systems.
`ImageGenerationAPIConfig`	Configuration for Image Generation API systems.
`ImageEditingAPIConfig`	Configuration for Image Editing API systems.
`VLMAPIConfig`	Configuration for Vision Language Model API systems.
`AgentCLIConfig`	Configuration for Agent CLI systems.
`GenericSystemConfig`	Generic system configuration for system types without specific validation.
`SystemsConfig`	Schema for the top-level systems configuration file.
`DatasetLoaderParams`	Parameters for loading a HuggingFace dataset.
`LabelFieldDefinition`	Defines a rich field definition for classification tasks within a label map.
`HFDatasetDefinition`	Defines a reusable HuggingFace dataset that can be referenced by name in test suites and generation jobs.
`FileDatasetBase`	Base class for file-based dataset definitions.
`PDFDatasetDefinition`	Defines a PDF document dataset that can be referenced by name in test suites and generation jobs.
`TXTDatasetDefinition`	Defines a plain text file dataset that can be referenced by name in test suites and generation jobs.
`DatasetsConfig`	Schema for datasets configuration file.
`TestDefinitionBase`	Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.
`TestDefinition`	A single test to be executed.
`TestSuiteDefault`	Default values that apply to all tests in the suite unless overridden.
`SuiteConfig`	Schema for the top-level Test Suite configuration file.
`ScoreCardFilter`	Defines which test results an indicator applies to.
`AssessmentRule`	Individual assessment outcome with condition.
`MetricExpression`	Expression-based metric evaluation with explicit value declarations.
`ScoreCardIndicator`	Individual score card indicator with filtering and assessment.
`AuditAssessmentRule`	Assessment outcome for audit indicators.
`AuditScoreCardIndicator`	Manual audit indicator.
`ScoreCard`	Complete grading score card configuration.
`AuditResponse`	!!! abstract "Usage Documentation"
`AuditResponses`	!!! abstract "Usage Documentation"
`GenerationJobConfig`	!!! abstract "Usage Documentation"
`DataGenerationConfig`	Schema for the data generation configuration manifest.
`ExecutionTags`	Tags for workflow execution tracking.
`ExecutionMetadata`	Metadata structure passed from workflow to test containers.
`LLMTestCase`	Query-only base for text LLM API (llm_api) test cases (Q).
`UnansweredLLMTestCase`	LLM test case without a ground-truth reference answer.
`AnsweredLLMTestCase`	LLM test case with a required ground-truth reference answer.
`RAGTestCase`	Query-only base for RAG API (rag_api) test cases (Qc).
`UnansweredRAGTestCase`	RAG test case without ground-truth references.
`AnsweredRAGTestCase`	RAG test case with a ground-truth reference answer.
`ContextualizedRAGTestCase`	RAG test case with ground-truth retrieved document chunks.
`VLMTestCase`	Query + image(s) base for VLM API (vlm_api) test cases (QI).
`UnansweredVLMTestCase`	VLM test case without a ground-truth reference answer.
`AnsweredVLMTestCase`	VLM test case with a required ground-truth reference answer.
`ImageGenerationTestCase`	Request-only base for Image Generation API (image_generation_api) test cases.
`UnansweredImageGenerationTestCase`	Image generation test case without a reference output image.
`AnsweredImageGenerationTestCase`	Image generation test case with a reference / expected output image.
`ImageEditingTestCase`	Request-only base for Image Editing API (image_editing_api) test cases.
`UnansweredImageEditingTestCase`	Image editing test case without a reference edited output image.
`AnsweredImageEditingTestCase`	Image editing test case with a reference / expected edited output image.
`EmbeddingTestCase`	Test case for Embedding API (embedding_api) systems.
`BoundingBox`	Ground truth bounding box in xyxy format.
`ODImageEditingTestCase`	Image editing test case with reference object locations on the seed image (SDG / detection-aware metrics).
`ObjectDetectionTestCase`	Image-only base for Object Detection API (object_detection_api) test cases.
`UnansweredObjectDetectionTestCase`	Object detection test case without ground-truth bounding boxes.
`AnsweredObjectDetectionTestCase`	Object detection test case with ground-truth bounding boxes.

Module Contents¶

src.asqi.schemas.IDsStringPattern¶

src.asqi.schemas.HFDtype¶

class src.asqi.schemas.ValueFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s Value feature type. Represents a scalar value feature (string, int64, float32, bool, etc.)

feature_type: Literal['Value'] = None¶

name: str = None¶

dtype: HFDtype = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.ListFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s List/Sequence feature type.

feature_type: Literal['List'] = None¶

name: str = None¶

feature: HFDtype | Literal['Image', 'Audio', 'Video', 'ClassLabel', 'Dict', 'List'] = None¶

length: int = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.DictFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s “Python dict” feature type.

feature_type: Literal['Dict'] = None¶

name: str = None¶

fields: list[HFFeature] = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.ClassLabelFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s ClassLabel feature type. Represents categorical data with named categories.

feature_type: Literal['ClassLabel'] = None¶

name: str = None¶

names: list[str] = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.ImageFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s Image feature type.

feature_type: Literal['Image'] = None¶

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.AudioFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s Audio feature type.

feature_type: Literal['Audio'] = None¶

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.VideoFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s Video feature type.

feature_type: Literal['Video'] = None¶

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.DatasetFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a feature/column within a dataset.

The dtype field uses HuggingFace datasets dtype values. Common types: ‘string’, ‘int64’, ‘float32’, ‘bool’.

name: str = None¶

dtype: HFDtype = None¶

required: bool = None¶

description: str | None = None¶

src.asqi.schemas.HFFeature¶

class src.asqi.schemas.SystemInput(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a system input that the container requires.

name: str = None¶

type: str | list[str] = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.InputParameter(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a parameter that can be passed to the test container. Supports simple types (string, integer, float, boolean) as well as list, object, enum.

name: str = None¶

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object', 'enum'] = None¶

required: bool = None¶

description: str | None = None¶

items: Literal['string', 'integer', 'float', 'boolean', 'object', 'enum'] | InputParameter | None = None¶

properties: list[InputParameter] | None = None¶

choices: list[str | int | float] | None = None¶

default: str | int | float | bool | list | dict | None = None¶

ui_config: dict[str, Any] | None = None¶

validate_rich_fields() → InputParameter¶: Validate that rich-type fields are only used with appropriate types.

class src.asqi.schemas.OutputMetric(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a metric that will be present in the test container’s output.

name: str¶

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object']¶

description: str | None = None¶

class src.asqi.schemas.OutputArtifact(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a file artifact generated by the test container.

name: str¶

path: str¶

description: str | None = None¶

class src.asqi.schemas.EnvironmentVariable(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines an environment variable required by the test container.

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.DatasetType¶

Bases: enum.StrEnum

Supported dataset types for InputDataset.

HUGGINGFACE = 'huggingface'¶

PDF = 'pdf'¶

TXT = 'txt'¶

class src.asqi.schemas.InputDataset(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a dataset input that the container requires.

Supported dataset types: - ‘huggingface’: HuggingFace datasets (requires features to be defined) - ‘pdf’: PDF documents - ‘txt’: Plain text files

Supports “either/or” relationships by accepting multiple types: - type: “pdf” - Single type - type: [“pdf”, “txt”] - Either PDF or TXT - type: [“huggingface”, “pdf”, “txt”] - Any of these formats

name: str = None¶

required: bool = None¶

type: DatasetType | list[DatasetType] = None¶

description: str | None = None¶

features: list[DatasetFeature | HFFeature] | None = None¶

class src.asqi.schemas.OutputReports(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a report that will be generated by the test container.

name: str = None¶

type: Literal['pdf', 'html'] = None¶

description: str | None = None¶

class src.asqi.schemas.OutputDataset(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a dataset output that will be generated by the test container.

name: str = None¶

type: DatasetType = None¶

description: str | None = None¶

features: list[DatasetFeature | HFFeature] | None = None¶

class src.asqi.schemas.Manifest(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the manifest.yaml file inside a test container.

name: str = None¶

version: str¶

description: str | None = None¶

host_access: bool = None¶

input_systems: list[SystemInput] = None¶

input_schema: list[InputParameter] = None¶

input_datasets: list[InputDataset] = None¶

output_metrics: list[str] | list[OutputMetric] = None¶

output_artifacts: list[OutputArtifact] | None = None¶

environment_variables: list[EnvironmentVariable] = None¶

output_reports: list[OutputReports] = None¶

output_datasets: list[OutputDataset] = None¶

class src.asqi.schemas.SystemDefinition(/, **data: Any)¶

Bases: pydantic.BaseModel

Base system definition.

description: str | None = None¶

provider: str | None = None¶

class src.asqi.schemas.ThinkingParams(/, **data: Any)¶

Bases: pydantic.BaseModel

Extended reasoning configuration for models that support it (e.g., Claude Opus 4.6 etc.).

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['enabled', 'adaptive'] = None¶

budget_tokens: int | None = None¶

class src.asqi.schemas.LLMAPIParams(/, **data: Any)¶

Bases: pydantic.BaseModel

Parameters for the LLM API systems.

base_url: str = None¶

model: str = None¶

env_file: str | None = None¶

api_key: str | None = None¶

thinking: ThinkingParams | None = None¶

reasoning_effort: str | None = None¶

class src.asqi.schemas.VLMAPIParams(/, **data: Any)¶

Bases: LLMAPIParams

Parameters for Vision Language Model API systems.

supports_vision: Literal[True] = None¶

class src.asqi.schemas.AgentCLIParams(/, **data: Any)¶

Bases: LLMAPIParams

Parameters for Agent CLI systems.

provider: str = None¶

class src.asqi.schemas.LLMAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for LLM API systems.

type: Literal['llm_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.EmbeddingAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for Embedding API systems.

type: Literal['embedding_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.CustomParams(/, **data: Any)¶

Bases: pydantic.BaseModel

Parameters for custom systems (e.g., SAP analytics, proprietary APIs).

base_url: str = None¶

api_key: str | None = None¶

extra: dict[str, Any] = None¶

class src.asqi.schemas.CustomConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for custom systems (e.g., SAP analytics, proprietary APIs).

type: Literal['custom'] = None¶

params: CustomParams = None¶

class src.asqi.schemas.RAGAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for RAG API systems.

type: Literal['rag_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.ImageGenerationAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for Image Generation API systems.

type: Literal['image_generation_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.ImageEditingAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for Image Editing API systems.

type: Literal['image_editing_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.VLMAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for Vision Language Model API systems.

type: Literal['vlm_api'] = None¶

params: VLMAPIParams = None¶

class src.asqi.schemas.AgentCLIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for Agent CLI systems.

Agent CLI systems are autonomous agents and coding frameworks that can be invoked via CLI or API (e.g., Codex, Qwen Coder, OpenCode, Goose, Aider, Cline).

type: Literal['agent_cli'] = None¶

params: AgentCLIParams = None¶

class src.asqi.schemas.GenericSystemConfig(/, **data: Any)¶

Bases: SystemDefinition

Generic system configuration for system types without specific validation.

This allows backward compatibility and support for system types that don’t have dedicated config classes yet.

type: str = None¶

params: dict[str, Any] = None¶

src.asqi.schemas.SystemConfig¶

class src.asqi.schemas.SystemsConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the top-level systems configuration file.

Extension Guide:

Create a new XXXConfig class inheriting from SystemDefinition. e.g, RESTAPIConfig

2. Create a new XXXParam class for the parameters of the system. e.g. RESTAPIParams 2. Add the new system definition (XXXConfig) to the SystemConfig union type

e.g. SystemConfig = LLMAPIConfig | XXXConfig | … | GenericSystemConfig

systems: dict[str, SystemConfig] = None¶

class src.asqi.schemas.DatasetLoaderParams(/, **data: Any)¶

Bases: pydantic.BaseModel

Parameters for loading a HuggingFace dataset.

Supports two mutually exclusive modes: - Hub mode: Load from HuggingFace Hub using hub_path - Local mode: Load from local files using builder_name with data_dir or data_files

hub_path: str | None = None¶

name: str | None = None¶

split: str | None = None¶

trust_remote_code: bool = None¶

builder_name: Literal['json', 'csv', 'parquet', 'arrow', 'text', 'xml', 'webdataset', 'imagefolder', 'audiofolder', 'videofolder'] | None = None¶

data_dir: str | None = None¶

data_files: str | list[str] | None = None¶

revision: str | None = None¶

streaming: bool = None¶

token: str | None = None¶

class src.asqi.schemas.LabelFieldDefinition(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a rich field definition for classification tasks within a label map.

field_description: str¶

field_data_type: str¶

field_enum_values: list[str] | None = None¶

field_multi_label: bool = False¶

src.asqi.schemas.LabelMap¶

class src.asqi.schemas.HFDatasetDefinition(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a reusable HuggingFace dataset that can be referenced by name in test suites and generation jobs.

type: Literal['huggingface'] = None¶

description: str | None = None¶

loader_params: DatasetLoaderParams = None¶

mapping: dict[str, str] = None¶

label_map: LabelMap | None = None¶

tags: list[str] = None¶

class src.asqi.schemas.FileDatasetBase(/, **data: Any)¶

Bases: pydantic.BaseModel

Base class for file-based dataset definitions.

description: str | None = None¶

file_path: str = None¶

tags: list[str] = None¶

class src.asqi.schemas.PDFDatasetDefinition(/, **data: Any)¶

Bases: FileDatasetBase

Defines a PDF document dataset that can be referenced by name in test suites and generation jobs.

type: Literal['pdf'] = None¶

class src.asqi.schemas.TXTDatasetDefinition(/, **data: Any)¶

Bases: FileDatasetBase

Defines a plain text file dataset that can be referenced by name in test suites and generation jobs.

type: Literal['txt'] = None¶

src.asqi.schemas.DatasetDefinition¶

class src.asqi.schemas.DatasetsConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for datasets configuration file.

Datasets are defined in a dictionary where keys are dataset names. Each dataset must specify a ‘type’ field that determines its schema: - ‘huggingface’: HuggingFace datasets with loader_params - ‘pdf’: PDF document datasets with file_path - ‘txt’: Plain text file datasets with file_path

datasets: dict[str, DatasetDefinition] = None¶

class src.asqi.schemas.TestDefinitionBase(/, **data: Any)¶

Bases: pydantic.BaseModel

Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.

systems_under_test: list[str] | None = None¶

systems: dict[str, str] | None = None¶

tags: list[str] | None = None¶

params: dict[str, Any] | None = None¶

input_datasets: dict[str, str] | None = None¶

volumes: dict[str, Any] | None = None¶

env_file: str | None = None¶

environment: dict[str, str] | None = None¶

class src.asqi.schemas.TestDefinition(/, **data: Any)¶

Bases: TestDefinitionBase

A single test to be executed.

id: IDsStringPattern = None¶

name: str | None = None¶

description: str | None = None¶

image: str = None¶

class src.asqi.schemas.TestSuiteDefault(/, **data: Any)¶

Bases: TestDefinitionBase

Default values that apply to all tests in the suite unless overridden.

class src.asqi.schemas.SuiteConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the top-level Test Suite configuration file.

suite_name: str = None¶

test_suite_default: TestSuiteDefault | None = None¶

description: str | None = None¶

test_suite: list[TestDefinition] = None¶

class src.asqi.schemas.ScoreCardFilter(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines which test results an indicator applies to.

test_id: str = None¶

target_system_type: str | list[str] | None = None¶

class src.asqi.schemas.AssessmentRule(/, **data: Any)¶

Bases: pydantic.BaseModel

Individual assessment outcome with condition.

outcome: str = None¶

condition: Literal['equal_to', 'greater_than', 'less_than', 'greater_equal', 'less_equal', 'all_true', 'any_false', 'count_equals'] = None¶

threshold: int | float | bool | None = None¶

description: str | None = None¶

class src.asqi.schemas.MetricExpression(/, **data: Any)¶

Bases: pydantic.BaseModel

Expression-based metric evaluation with explicit value declarations.

expression: str = None¶

values: dict[str, str] = None¶

class src.asqi.schemas.ScoreCardIndicator(/, **data: Any)¶

Bases: pydantic.BaseModel

Individual score card indicator with filtering and assessment.

id: IDsStringPattern = None¶

name: str | None = None¶

apply_to: ScoreCardFilter = None¶

metric: str | MetricExpression = None¶

assessment: list[AssessmentRule] = None¶

display_reports: list[str] = None¶

class src.asqi.schemas.AuditAssessmentRule(/, **data: Any)¶

Bases: pydantic.BaseModel

Assessment outcome for audit indicators.

outcome: str = None¶

description: str | None = None¶

class src.asqi.schemas.AuditScoreCardIndicator(/, **data: Any)¶

Bases: pydantic.BaseModel

Manual audit indicator.

Outcome is provided via an external audit responses file.

id: IDsStringPattern = None¶

type: Literal['audit'] = None¶

name: str | None = None¶

assessment: list[AuditAssessmentRule] = None¶

class src.asqi.schemas.ScoreCard(/, **data: Any)¶

Bases: pydantic.BaseModel

Complete grading score card configuration.

score_card_name: str = None¶

indicators: list[ScoreCardIndicator | AuditScoreCardIndicator] = None¶

class src.asqi.schemas.AuditResponse(/, **data: Any)¶

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Attributes:

__class_vars__: The names of the class variables defined on the model. __private_attributes__: Metadata about the private attributes of the model. __signature__: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields. __pydantic_core_schema__: The core schema of the model. __pydantic_custom_init__: Whether the model has a custom __init__ function. __pydantic_decorators__: Metadata containing the decorators defined on the model.

This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to: __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models. __pydantic_post_init__: The name of the post-init method for the model, if defined. __pydantic_root_model__: Whether the model is a [RootModel][pydantic.root_model.RootModel]. __pydantic_serializer__: The pydantic-core SchemaSerializer used to dump instances of the model. __pydantic_validator__: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects. __pydantic_computed_fields__: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra]: is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

indicator_id: str = None¶

sut_name: str | None = None¶

selected_outcome: str = None¶

notes: str | None = None¶

class src.asqi.schemas.AuditResponses(/, **data: Any)¶

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Attributes:

This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to: __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra]: is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

responses: list[AuditResponse]¶

class src.asqi.schemas.GenerationJobConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Attributes:

This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to: __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra]: is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

id: str = None¶

systems: dict[str, str] | None = None¶

name: str = None¶

image: str = None¶

tags: list[str] | None = None¶

input_datasets: dict[str, str] | None = None¶

params: dict[str, Any] | None = None¶

volumes: dict[str, Any] | None = None¶

env_file: str | None = None¶

environment: dict[str, str] | None = None¶

class src.asqi.schemas.DataGenerationConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the data generation configuration manifest.

job_name: str = None¶

generation_jobs: list[GenerationJobConfig] = None¶

class src.asqi.schemas.ExecutionTags(/, **data: Any)¶

Bases: pydantic.BaseModel

Tags for workflow execution tracking.

parent_id: str = None¶

job_type: str = None¶

job_id: str = None¶

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class src.asqi.schemas.ExecutionMetadata(/, **data: Any)¶

Bases: pydantic.BaseModel

Metadata structure passed from workflow to test containers.

tags: ExecutionTags = None¶

user_id: str | None = None¶

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class src.asqi.schemas.LLMTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Query-only base for text LLM API (llm_api) test cases (Q).

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶: Computes the OpenAI-compatible request for llm_api / rag_api.

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.UnansweredLLMTestCase(**data: Any)¶

Bases: pydantic.BaseModel

LLM test case without a ground-truth reference answer.

Use with metrics that do not require a reference output, e.g. judge-based metrics for bias, toxicity, hallucination, policy compliance.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.AnsweredLLMTestCase(**data: Any)¶

Bases: pydantic.BaseModel

LLM test case with a required ground-truth reference answer.

Use with reference-based metrics such as BLEU, ROUGE, BERTScore, exact_match, and correctness.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

extra_params: dict[str, Any] = None¶

answer: str = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

property expected_response: dict[str, Any]¶: Computes the expected OpenAI response structure.

classmethod from_api_data(request: dict[str, Any], response: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → AnsweredLLMTestCase¶

class src.asqi.schemas.RAGTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Query-only base for RAG API (rag_api) test cases (Qc).

Request shape matches text LLM messages; use this type (or semantic variants) for RAG-specific metrics and routing.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.UnansweredRAGTestCase(**data: Any)¶

Bases: pydantic.BaseModel

RAG test case without ground-truth references.

Use with metrics that score without a reference answer or retrieved context, e.g. faithfulness, groundedness, helpfulness, retrieval_relevance.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.AnsweredRAGTestCase(**data: Any)¶

Bases: pydantic.BaseModel

RAG test case with a ground-truth reference answer.

Use with reference-based metrics that compare the model answer to answer (e.g. answer correctness, Ragas context_recall when keyed on the reference answer). For ground-truth retrieved document chunks, use ContextualizedRAGTestCase.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

extra_params: dict[str, Any] = None¶

answer: str = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

property expected_response: dict[str, Any] | None¶: Computes the expected RAG API response structure.

classmethod from_api_data(request: dict[str, Any], response: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → AnsweredRAGTestCase¶

class src.asqi.schemas.ContextualizedRAGTestCase(**data: Any)¶

Bases: pydantic.BaseModel

RAG test case with ground-truth retrieved document chunks.

Use with metrics that need reference contexts (e.g. context_precision, retrieval correctness, hit@k) without requiring a reference answer on the same object.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

extra_params: dict[str, Any] = None¶

context: list[str] = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

property expected_response: dict[str, Any] | None¶: Computes the expected RAG API response structure.

classmethod from_api_data(request: dict[str, Any], response: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → ContextualizedRAGTestCase¶

class src.asqi.schemas.VLMTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Query + image(s) base for VLM API (vlm_api) test cases (QI).

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

images: list[str] = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶: Computes the OpenAI multi-modal request for vlm_api.

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.UnansweredVLMTestCase(**data: Any)¶

Bases: pydantic.BaseModel

VLM test case without a ground-truth reference answer.

Use with metrics that do not require a reference output, e.g. judge-based metrics for hallucination and policy compliance.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

images: list[str] = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.AnsweredVLMTestCase(**data: Any)¶

Bases: pydantic.BaseModel

VLM test case with a required ground-truth reference answer.

Use with reference-based metrics such as correctness.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

query: str = None¶

system_prompt: str | None = None¶

images: list[str] = None¶

extra_params: dict[str, Any] = None¶

answer: str = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

property expected_response: dict[str, Any]¶: Computes the expected OpenAI response structure.

classmethod from_api_data(request: dict[str, Any], response: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → AnsweredVLMTestCase¶

class src.asqi.schemas.ImageGenerationTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Request-only base for Image Generation API (image_generation_api) test cases.

Use this type (or UnansweredImageGenerationTestCase) when no reference output image is provided. For a reference image for metrics (e.g. FID-style comparisons), use AnsweredImageGenerationTestCase.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

prompt: str = None¶

size: str | None = None¶

n: int = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶: Computes the OpenAI Image Generation API request.

classmethod from_api_data(request: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → ImageGenerationTestCase¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.UnansweredImageGenerationTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Image generation test case without a reference output image.

Use with metrics that do not require a reference image, e.g. judge-based or policy checks on generated content.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

prompt: str = None¶

size: str | None = None¶

n: int = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶

classmethod from_api_data(request: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → UnansweredImageGenerationTestCase¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.AnsweredImageGenerationTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Image generation test case with a reference / expected output image.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

prompt: str = None¶

size: str | None = None¶

n: int = None¶

extra_params: dict[str, Any] = None¶

generation: str = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

property expected_response: dict[str, Any]¶: Expected image output shape (reference image as data URI).

classmethod from_api_data(request: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, generation: str = '', metadata: dict[str, Any] | None = None) → AnsweredImageGenerationTestCase¶

class src.asqi.schemas.ImageEditingTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Request-only base for Image Editing API (image_editing_api) test cases.

Use this type (or UnansweredImageEditingTestCase) when no reference edited image is provided. For ground-truth edited output, use AnsweredImageEditingTestCase.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

image: str = None¶

edit_prompt: str = None¶

mask: str | None = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶: Computes the OpenAI Image Edit API request fields. The image and mask are base64 data URIs here; the test container is responsible for converting them to binary file objects when constructing the multipart/form-data call.

classmethod from_api_data(request: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → ImageEditingTestCase¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.UnansweredImageEditingTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Image editing test case without a reference edited output image.

Use with metrics that do not require a reference image, e.g. judge-based checks.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

image: str = None¶

edit_prompt: str = None¶

mask: str | None = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶

classmethod from_api_data(request: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → UnansweredImageEditingTestCase¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.AnsweredImageEditingTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Image editing test case with a reference / expected edited output image.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

image: str = None¶

edit_prompt: str = None¶

mask: str | None = None¶

extra_params: dict[str, Any] = None¶

generation: str = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

property expected_response: dict[str, Any]¶: Expected edited image as data URI.

classmethod from_api_data(request: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, generation: str = '', metadata: dict[str, Any] | None = None) → AnsweredImageEditingTestCase¶

class src.asqi.schemas.EmbeddingTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Test case for Embedding API (embedding_api) systems.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

text: str = None¶

expected_similar_texts: list[str] = None¶

expected_dissimilar_texts: list[str] = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶: Computes the OpenAI Embeddings API request.

property expected_response: None¶: No structured expected response — expected_similar_texts / expected_dissimilar_texts are evaluated by the metric after computing cosine similarity against returned vectors.

classmethod from_api_data(request: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, expected_similar_texts: list[str] | None = None, expected_dissimilar_texts: list[str] | None = None, metadata: dict[str, Any] | None = None) → EmbeddingTestCase¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.BoundingBox(/, **data: Any)¶

Bases: pydantic.BaseModel

Ground truth bounding box in xyxy format.

xyxy: tuple[float, float, float, float] = None¶

class_name: str = None¶

confidence: float | None = None¶

class src.asqi.schemas.ODImageEditingTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Image editing test case with reference object locations on the seed image (SDG / detection-aware metrics).

The synthetic or edited result still comes from the system response; expected_detections supplies the same structured box list as legacy detected_objects['bbox'] rows in SDG datasets. An empty list means no boxes (many per-object metrics degenerate or error — see metric docs).

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

image: str = None¶

edit_prompt: str = None¶

mask: str | None = None¶

extra_params: dict[str, Any] = None¶

expected_detections: list[BoundingBox] = None¶

property request: dict[str, Any]¶

classmethod from_api_data(request: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → ODImageEditingTestCase¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.ObjectDetectionTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Image-only base for Object Detection API (object_detection_api) test cases.

Use this type (or UnansweredObjectDetectionTestCase) when no reference bounding boxes are provided — e.g. judge-only or exploratory metrics. For ground-truth boxes (including an empty list meaning zero objects expected), use AnsweredObjectDetectionTestCase.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

image: str = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶: Computes the Object Detection API request fields. image is a base64 data URI here; the test container converts it to a binary file when constructing the multipart/form-data POST to /detect.

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.UnansweredObjectDetectionTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Object detection test case without ground-truth bounding boxes.

Use with metrics that do not require reference boxes, e.g. judge-based checks.

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

image: str = None¶

extra_params: dict[str, Any] = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

class src.asqi.schemas.AnsweredObjectDetectionTestCase(**data: Any)¶

Bases: pydantic.BaseModel

Object detection test case with ground-truth bounding boxes.

expected_detections uses a nested list[BoundingBox] structure — an explicit exception to the flat-field rule, agreed under PROG-32, because bounding-box ground truth is inherently structured. An empty list means no objects are expected in the image (distinct from omitting ground truth on the base / unanswered type).

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

lineage_id: str = None¶

scenario: str | None = None¶

image: str = None¶

extra_params: dict[str, Any] = None¶

expected_detections: list[BoundingBox] = None¶

confidence_threshold: float | None = None¶

property request: dict[str, Any]¶

get_debug_metadata() → dict[str, Any]¶

property expected_response: dict[str, Any]¶: Computes the expected /detect response structure.

classmethod from_api_data(request: dict[str, Any], response: dict[str, Any], lineage_id: str | None = None, scenario: str | None = None, metadata: dict[str, Any] | None = None) → AnsweredObjectDetectionTestCase¶

src.asqi.schemas.TestCase¶