src.asqi.schemas¶

Attributes¶

`IDsStringPattern`
`HFDtype`
`HFFeature`
`SystemConfig`
`DatasetDefinition`

Classes¶

`ValueFeature`	Corresponds to HuggingFace's Value feature type.
`ListFeature`	Corresponds to HuggingFace's List/Sequence feature type.
`DictFeature`	Corresponds to HuggingFace's "Python dict" feature type.
`ClassLabelFeature`	Corresponds to HuggingFace's ClassLabel feature type. Represents categorical data with named categories.
`ImageFeature`	Corresponds to HuggingFace's Image feature type.
`AudioFeature`	Corresponds to HuggingFace's Audio feature type.
`VideoFeature`	Corresponds to HuggingFace's Video feature type.
`DatasetFeature`	Defines a feature/column within a dataset.
`SystemInput`	Defines a system input that the container requires.
`InputParameter`	Defines a parameter that can be passed to the test container.
`OutputMetric`	Defines a metric that will be present in the test container's output.
`OutputArtifact`	Defines a file artifact generated by the test container.
`EnvironmentVariable`	Defines an environment variable required by the test container.
`DatasetType`	Supported dataset types for InputDataset.
`InputDataset`	Defines a dataset input that the container requires.
`OutputReports`	Defines a report that will be generated by the test container.
`OutputDataset`	Defines a dataset output that will be generated by the test container.
`Manifest`	Schema for the manifest.yaml file inside a test container.
`SystemDefinition`	Base system definition.
`LLMAPIParams`	Parameters for the LLM API systems.
`VLMAPIParams`	Parameters for Vision Language Model API systems.
`LLMAPIConfig`	Configuration for LLM API systems.
`RAGAPIConfig`	Configuration for RAG API systems.
`ImageGenerationAPIConfig`	Configuration for Image Generation API systems.
`ImageEditingAPIConfig`	Configuration for Image Editing API systems.
`VLMAPIConfig`	Configuration for Vision Language Model API systems.
`GenericSystemConfig`	Generic system configuration for system types without specific validation.
`SystemsConfig`	Schema for the top-level systems configuration file.
`DatasetLoaderParams`	Parameters for loading a HuggingFace dataset from the mounted folder in the container.
`HFDatasetDefinition`	Defines a reusable HuggingFace dataset that can be referenced by name in test suites and generation jobs.
`FileDatasetBase`	Base class for file-based dataset definitions.
`PDFDatasetDefinition`	Defines a PDF document dataset that can be referenced by name in test suites and generation jobs.
`TXTDatasetDefinition`	Defines a plain text file dataset that can be referenced by name in test suites and generation jobs.
`DatasetsConfig`	Schema for datasets configuration file.
`TestDefinitionBase`	Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.
`TestDefinition`	A single test to be executed.
`TestSuiteDefault`	Default values that apply to all tests in the suite unless overridden.
`SuiteConfig`	Schema for the top-level Test Suite configuration file.
`ScoreCardFilter`	Defines which test results an indicator applies to.
`AssessmentRule`	Individual assessment outcome with condition.
`MetricExpression`	Expression-based metric evaluation with explicit value declarations.
`ScoreCardIndicator`	Individual score card indicator with filtering and assessment.
`AuditAssessmentRule`	Assessment outcome for audit indicators.
`AuditScoreCardIndicator`	Manual audit indicator.
`ScoreCard`	Complete grading score card configuration.
`AuditResponse`	!!! abstract "Usage Documentation"
`AuditResponses`	!!! abstract "Usage Documentation"
`GenerationJobConfig`	!!! abstract "Usage Documentation"
`DataGenerationConfig`	Schema for the data generation configuration manifest.
`ExecutionTags`	Tags for workflow execution tracking.
`ExecutionMetadata`	Metadata structure passed from workflow to test containers.

Module Contents¶

src.asqi.schemas.IDsStringPattern¶

src.asqi.schemas.HFDtype¶

class src.asqi.schemas.ValueFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s Value feature type. Represents a scalar value feature (string, int64, float32, bool, etc.)

feature_type: Literal['Value'] = None¶

name: str = None¶

dtype: HFDtype = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.ListFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s List/Sequence feature type.

feature_type: Literal['List'] = None¶

name: str = None¶

feature: HFDtype | Literal['Image', 'Audio', 'Video', 'ClassLabel', 'Dict', 'List'] = None¶

length: int = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.DictFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s “Python dict” feature type.

feature_type: Literal['Dict'] = None¶

name: str = None¶

fields: List[HFFeature] = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.ClassLabelFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s ClassLabel feature type. Represents categorical data with named categories.

feature_type: Literal['ClassLabel'] = None¶

name: str = None¶

names: List[str] = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.ImageFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s Image feature type.

feature_type: Literal['Image'] = None¶

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.AudioFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s Audio feature type.

feature_type: Literal['Audio'] = None¶

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.VideoFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Corresponds to HuggingFace’s Video feature type.

feature_type: Literal['Video'] = None¶

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.DatasetFeature(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a feature/column within a dataset.

The dtype field uses HuggingFace datasets dtype values. Common types: ‘string’, ‘int64’, ‘float32’, ‘bool’.

name: str = None¶

dtype: HFDtype = None¶

required: bool = None¶

description: str | None = None¶

src.asqi.schemas.HFFeature¶

class src.asqi.schemas.SystemInput(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a system input that the container requires.

name: str = None¶

type: str | List[str] = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.InputParameter(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a parameter that can be passed to the test container. Supports simple types (string, integer, float, boolean) as well as list, object, enum.

name: str = None¶

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object', 'enum'] = None¶

required: bool = None¶

description: str | None = None¶

items: Literal['string', 'integer', 'float', 'boolean', 'object', 'enum'] | InputParameter | None = None¶

properties: List[InputParameter] | None = None¶

choices: List[str | int | float] | None = None¶

default: str | int | float | bool | List | Dict | None = None¶

validate_rich_fields() → InputParameter¶: Validate that rich-type fields are only used with appropriate types.

class src.asqi.schemas.OutputMetric(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a metric that will be present in the test container’s output.

name: str¶

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object']¶

description: str | None = None¶

class src.asqi.schemas.OutputArtifact(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a file artifact generated by the test container.

name: str¶

path: str¶

description: str | None = None¶

class src.asqi.schemas.EnvironmentVariable(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines an environment variable required by the test container.

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.DatasetType¶

Bases: enum.StrEnum

Supported dataset types for InputDataset.

HUGGINGFACE = 'huggingface'¶

PDF = 'pdf'¶

TXT = 'txt'¶

class src.asqi.schemas.InputDataset(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a dataset input that the container requires.

Supported dataset types: - ‘huggingface’: HuggingFace datasets (requires features to be defined) - ‘pdf’: PDF documents - ‘txt’: Plain text files

Supports “either/or” relationships by accepting multiple types: - type: “pdf” - Single type - type: [“pdf”, “txt”] - Either PDF or TXT - type: [“huggingface”, “pdf”, “txt”] - Any of these formats

name: str = None¶

required: bool = None¶

type: DatasetType | List[DatasetType] = None¶

description: str | None = None¶

features: List[DatasetFeature | HFFeature] | None = None¶

class src.asqi.schemas.OutputReports(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a report that will be generated by the test container.

name: str = None¶

type: Literal['pdf', 'html'] = None¶

description: str | None = None¶

class src.asqi.schemas.OutputDataset(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a dataset output that will be generated by the test container.

name: str = None¶

type: DatasetType = None¶

description: str | None = None¶

features: List[DatasetFeature | HFFeature] | None = None¶

class src.asqi.schemas.Manifest(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the manifest.yaml file inside a test container.

name: str = None¶

version: str¶

description: str | None = None¶

host_access: bool = None¶

input_systems: List[SystemInput] = None¶

input_schema: List[InputParameter] = None¶

input_datasets: list[InputDataset] = None¶

output_metrics: List[str] | List[OutputMetric] = None¶

output_artifacts: List[OutputArtifact] | None = None¶

environment_variables: List[EnvironmentVariable] = None¶

output_reports: List[OutputReports] = None¶

output_datasets: List[OutputDataset] = None¶

class src.asqi.schemas.SystemDefinition(/, **data: Any)¶

Bases: pydantic.BaseModel

Base system definition.

description: str | None = None¶

provider: str | None = None¶

class src.asqi.schemas.LLMAPIParams(/, **data: Any)¶

Bases: pydantic.BaseModel

Parameters for the LLM API systems.

base_url: str = None¶

model: str = None¶

env_file: str | None = None¶

api_key: str | None = None¶

class src.asqi.schemas.VLMAPIParams(/, **data: Any)¶

Bases: LLMAPIParams

Parameters for Vision Language Model API systems.

supports_vision: Literal[True] = None¶

class src.asqi.schemas.LLMAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for LLM API systems.

type: Literal['llm_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.RAGAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for RAG API systems.

type: Literal['rag_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.ImageGenerationAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for Image Generation API systems.

type: Literal['image_generation_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.ImageEditingAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for Image Editing API systems.

type: Literal['image_editing_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.VLMAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for Vision Language Model API systems.

type: Literal['vlm_api'] = None¶

params: VLMAPIParams = None¶

class src.asqi.schemas.GenericSystemConfig(/, **data: Any)¶

Bases: SystemDefinition

Generic system configuration for system types without specific validation.

This allows backward compatibility and support for system types that don’t have dedicated config classes yet.

type: str = None¶

params: Dict[str, Any] = None¶

src.asqi.schemas.SystemConfig¶

class src.asqi.schemas.SystemsConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the top-level systems configuration file.

Extension Guide:

Create a new XXXConfig class inheriting from SystemDefinition. e.g, RESTAPIConfig

2. Create a new XXXParam class for the parameters of the system. e.g. RESTAPIParams 2. Add the new system definition (XXXConfig) to the SystemConfig union type

e.g. SystemConfig = Union[LLMAPIConfig, XXXConfig, …, GenericSystemConfig]

systems: Dict[str, SystemConfig] = None¶

class src.asqi.schemas.DatasetLoaderParams(/, **data: Any)¶

Bases: pydantic.BaseModel

Parameters for loading a HuggingFace dataset from the mounted folder in the container.

builder_name: Literal['json', 'csv', 'parquet', 'arrow', 'text', 'xml', 'webdataset', 'imagefolder', 'audiofolder', 'videofolder'] = None¶

data_dir: str | None = None¶

data_files: str | list[str] | None = None¶

revision: str | None = None¶

class src.asqi.schemas.HFDatasetDefinition(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a reusable HuggingFace dataset that can be referenced by name in test suites and generation jobs.

type: Literal['huggingface'] = None¶

description: str | None = None¶

loader_params: DatasetLoaderParams = None¶

mapping: dict[str, str] = None¶

tags: list[str] = None¶

class src.asqi.schemas.FileDatasetBase(/, **data: Any)¶

Bases: pydantic.BaseModel

Base class for file-based dataset definitions.

description: str | None = None¶

file_path: str = None¶

tags: list[str] = None¶

class src.asqi.schemas.PDFDatasetDefinition(/, **data: Any)¶

Bases: FileDatasetBase

Defines a PDF document dataset that can be referenced by name in test suites and generation jobs.

type: Literal['pdf'] = None¶

class src.asqi.schemas.TXTDatasetDefinition(/, **data: Any)¶

Bases: FileDatasetBase

Defines a plain text file dataset that can be referenced by name in test suites and generation jobs.

type: Literal['txt'] = None¶

src.asqi.schemas.DatasetDefinition¶

class src.asqi.schemas.DatasetsConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for datasets configuration file.

Datasets are defined in a dictionary where keys are dataset names. Each dataset must specify a ‘type’ field that determines its schema: - ‘huggingface’: HuggingFace datasets with loader_params - ‘pdf’: PDF document datasets with file_path - ‘txt’: Plain text file datasets with file_path

datasets: Dict[str, DatasetDefinition] = None¶

class src.asqi.schemas.TestDefinitionBase(/, **data: Any)¶

Bases: pydantic.BaseModel

Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.

systems_under_test: List[str] | None = None¶

systems: Dict[str, str] | None = None¶

tags: List[str] | None = None¶

params: Dict[str, Any] | None = None¶

input_datasets: Dict[str, str] | None = None¶

volumes: Dict[str, Any] | None = None¶

env_file: str | None = None¶

environment: Dict[str, str] | None = None¶

class src.asqi.schemas.TestDefinition(/, **data: Any)¶

Bases: TestDefinitionBase

A single test to be executed.

id: IDsStringPattern = None¶

name: str | None = None¶

description: str | None = None¶

image: str = None¶

class src.asqi.schemas.TestSuiteDefault(/, **data: Any)¶

Bases: TestDefinitionBase

Default values that apply to all tests in the suite unless overridden.

class src.asqi.schemas.SuiteConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the top-level Test Suite configuration file.

suite_name: str = None¶

test_suite_default: TestSuiteDefault | None = None¶

description: str | None = None¶

test_suite: List[TestDefinition] = None¶

class src.asqi.schemas.ScoreCardFilter(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines which test results an indicator applies to.

test_id: str = None¶

target_system_type: str | List[str] | None = None¶

class src.asqi.schemas.AssessmentRule(/, **data: Any)¶

Bases: pydantic.BaseModel

Individual assessment outcome with condition.

outcome: str = None¶

condition: Literal['equal_to', 'greater_than', 'less_than', 'greater_equal', 'less_equal', 'all_true', 'any_false', 'count_equals'] = None¶

threshold: int | float | bool | None = None¶

description: str | None = None¶

class src.asqi.schemas.MetricExpression(/, **data: Any)¶

Bases: pydantic.BaseModel

Expression-based metric evaluation with explicit value declarations.

expression: str = None¶

values: Dict[str, str] = None¶

class src.asqi.schemas.ScoreCardIndicator(/, **data: Any)¶

Bases: pydantic.BaseModel

Individual score card indicator with filtering and assessment.

id: IDsStringPattern = None¶

name: str | None = None¶

apply_to: ScoreCardFilter = None¶

metric: str | MetricExpression = None¶

assessment: List[AssessmentRule] = None¶

display_reports: List[str] = None¶

class src.asqi.schemas.AuditAssessmentRule(/, **data: Any)¶

Bases: pydantic.BaseModel

Assessment outcome for audit indicators.

outcome: str = None¶

description: str | None = None¶

class src.asqi.schemas.AuditScoreCardIndicator(/, **data: Any)¶

Bases: pydantic.BaseModel

Manual audit indicator.

Outcome is provided via an external audit responses file.

id: IDsStringPattern = None¶

type: Literal['audit'] = None¶

name: str | None = None¶

assessment: List[AuditAssessmentRule] = None¶

class src.asqi.schemas.ScoreCard(/, **data: Any)¶

Bases: pydantic.BaseModel

Complete grading score card configuration.

score_card_name: str = None¶

indicators: List[ScoreCardIndicator | AuditScoreCardIndicator] = None¶

class src.asqi.schemas.AuditResponse(/, **data: Any)¶

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Attributes:

__class_vars__: The names of the class variables defined on the model. __private_attributes__: Metadata about the private attributes of the model. __signature__: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields. __pydantic_core_schema__: The core schema of the model. __pydantic_custom_init__: Whether the model has a custom __init__ function. __pydantic_decorators__: Metadata containing the decorators defined on the model.

This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to: __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models. __pydantic_post_init__: The name of the post-init method for the model, if defined. __pydantic_root_model__: Whether the model is a [RootModel][pydantic.root_model.RootModel]. __pydantic_serializer__: The pydantic-core SchemaSerializer used to dump instances of the model. __pydantic_validator__: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects. __pydantic_computed_fields__: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra]: is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

indicator_id: str = None¶

sut_name: str | None = None¶

selected_outcome: str = None¶

notes: str | None = None¶

class src.asqi.schemas.AuditResponses(/, **data: Any)¶

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Attributes:

This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to: __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra]: is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

responses: List[AuditResponse]¶

class src.asqi.schemas.GenerationJobConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Attributes:

This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to: __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra]: is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

id: str = None¶

systems: Dict[str, str] | None = None¶

name: str = None¶

image: str = None¶

tags: List[str] | None = None¶

input_datasets: Dict[str, str] | None = None¶

params: Dict[str, Any] | None = None¶

volumes: Dict[str, Any] | None = None¶

env_file: str | None = None¶

environment: Dict[str, str] | None = None¶

class src.asqi.schemas.DataGenerationConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the data generation configuration manifest.

job_name: str = None¶

generation_jobs: List[GenerationJobConfig] = None¶

class src.asqi.schemas.ExecutionTags(/, **data: Any)¶

Bases: pydantic.BaseModel

Tags for workflow execution tracking.

parent_id: str = None¶

job_type: str = None¶

job_id: str = None¶

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class src.asqi.schemas.ExecutionMetadata(/, **data: Any)¶

Bases: pydantic.BaseModel

Metadata structure passed from workflow to test containers.

tags: ExecutionTags = None¶

user_id: str | None = None¶

model_config¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].