src.asqi.schemas¶

Attributes¶

`IDsStringPattern`
`SystemConfig`

Classes¶

`SystemInput`	Defines a system input that the container requires.
`InputParameter`	Defines a parameter that can be passed to the test container.
`OutputMetric`	Defines a metric that will be present in the test container's output.
`OutputArtifact`	Defines a file artifact generated by the test container.
`EnvironmentVariable`	Defines an environment variable required by the test container.
`Manifest`	Schema for the manifest.yaml file inside a test container.
`SystemDefinition`	Base system definition.
`LLMAPIParams`	Parameters for the LLM API systems.
`LLMAPIConfig`	Configuration for LLM API systems.
`RAGAPIConfig`	Configuration for RAG API systems.
`GenericSystemConfig`	Generic system configuration for system types without specific validation.
`SystemsConfig`	Schema for the top-level systems configuration file.
`TestDefinitionBase`	Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.
`TestDefinition`	A single test to be executed.
`TestSuiteDefault`	Default values that apply to all tests in the suite unless overridden.
`SuiteConfig`	Schema for the top-level Test Suite configuration file.
`ScoreCardFilter`	Defines which test results an indicator applies to.
`AssessmentRule`	Individual assessment outcome with condition.
`MetricExpression`	Expression-based metric evaluation with explicit value declarations.
`ScoreCardIndicator`	Individual score card indicator with filtering and assessment.
`AuditAssessmentRule`	Assessment outcome for audit indicators.
`AuditScoreCardIndicator`	Manual audit indicator.
`ScoreCard`	Complete grading score card configuration.
`AuditResponse`	!!! abstract "Usage Documentation"
`AuditResponses`	!!! abstract "Usage Documentation"

Module Contents¶

src.asqi.schemas.IDsStringPattern¶

class src.asqi.schemas.SystemInput(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a system input that the container requires.

name: str = None¶

type: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.InputParameter(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a parameter that can be passed to the test container.

name: str¶

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object']¶

required: bool = False¶

description: str | None = None¶

class src.asqi.schemas.OutputMetric(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a metric that will be present in the test container’s output.

name: str¶

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object']¶

description: str | None = None¶

class src.asqi.schemas.OutputArtifact(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a file artifact generated by the test container.

name: str¶

path: str¶

description: str | None = None¶

class src.asqi.schemas.EnvironmentVariable(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines an environment variable required by the test container.

name: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.Manifest(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the manifest.yaml file inside a test container.

name: str = None¶

version: str¶

description: str | None = None¶

host_access: bool = None¶

input_systems: List[SystemInput] = None¶

input_schema: List[InputParameter] = None¶

output_metrics: List[str] | List[OutputMetric] = None¶

output_artifacts: List[OutputArtifact] | None = None¶

environment_variables: List[EnvironmentVariable] = None¶

class src.asqi.schemas.SystemDefinition(/, **data: Any)¶

Bases: pydantic.BaseModel

Base system definition.

description: str | None = None¶

provider: str | None = None¶

class src.asqi.schemas.LLMAPIParams(/, **data: Any)¶

Bases: pydantic.BaseModel

Parameters for the LLM API systems.

base_url: str = None¶

model: str = None¶

env_file: str | None = None¶

api_key: str | None = None¶

class src.asqi.schemas.LLMAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for LLM API systems.

type: Literal['llm_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.RAGAPIConfig(/, **data: Any)¶

Bases: SystemDefinition

Configuration for RAG API systems.

type: Literal['rag_api'] = None¶

params: LLMAPIParams = None¶

class src.asqi.schemas.GenericSystemConfig(/, **data: Any)¶

Bases: SystemDefinition

Generic system configuration for system types without specific validation.

This allows backward compatibility and support for system types that don’t have dedicated config classes yet.

type: str = None¶

params: Dict[str, Any] = None¶

src.asqi.schemas.SystemConfig¶

class src.asqi.schemas.SystemsConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the top-level systems configuration file.

Extension Guide:

Create a new XXXConfig class inheriting from SystemDefinition. e.g, RESTAPIConfig

2. Create a new XXXParam class for the parameters of the system. e.g. RESTAPIParams 2. Add the new system definition (XXXConfig) to the SystemConfig union type

e.g. SystemConfig = Union[LLMAPIConfig, XXXConfig, …, GenericSystemConfig]

systems: Dict[str, SystemConfig] = None¶

class src.asqi.schemas.TestDefinitionBase(/, **data: Any)¶

Bases: pydantic.BaseModel

Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.

systems_under_test: List[str] | None = None¶

systems: Dict[str, str] | None = None¶

tags: List[str] | None = None¶

params: Dict[str, Any] | None = None¶

volumes: Dict[str, Any] | None = None¶

env_file: str | None = None¶

environment: Dict[str, str] | None = None¶

class src.asqi.schemas.TestDefinition(/, **data: Any)¶

Bases: TestDefinitionBase

A single test to be executed.

id: IDsStringPattern = None¶

name: str | None = None¶

description: str | None = None¶

image: str = None¶

class src.asqi.schemas.TestSuiteDefault(/, **data: Any)¶

Bases: TestDefinitionBase

Default values that apply to all tests in the suite unless overridden.

class src.asqi.schemas.SuiteConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the top-level Test Suite configuration file.

suite_name: str = None¶

test_suite_default: TestSuiteDefault | None = None¶

description: str | None = None¶

test_suite: List[TestDefinition] = None¶

class src.asqi.schemas.ScoreCardFilter(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines which test results an indicator applies to.

test_id: str = None¶

class src.asqi.schemas.AssessmentRule(/, **data: Any)¶

Bases: pydantic.BaseModel

Individual assessment outcome with condition.

outcome: str = None¶

condition: Literal['equal_to', 'greater_than', 'less_than', 'greater_equal', 'less_equal', 'all_true', 'any_false', 'count_equals'] = None¶

threshold: int | float | bool | None = None¶

description: str | None = None¶

class src.asqi.schemas.MetricExpression(/, **data: Any)¶

Bases: pydantic.BaseModel

Expression-based metric evaluation with explicit value declarations.

expression: str = None¶

values: Dict[str, str] = None¶

class src.asqi.schemas.ScoreCardIndicator(/, **data: Any)¶

Bases: pydantic.BaseModel

Individual score card indicator with filtering and assessment.

id: IDsStringPattern = None¶

name: str | None = None¶

apply_to: ScoreCardFilter = None¶

metric: str | MetricExpression = None¶

assessment: List[AssessmentRule] = None¶

class src.asqi.schemas.AuditAssessmentRule(/, **data: Any)¶

Bases: pydantic.BaseModel

Assessment outcome for audit indicators.

outcome: str = None¶

description: str | None = None¶

class src.asqi.schemas.AuditScoreCardIndicator(/, **data: Any)¶

Bases: pydantic.BaseModel

Manual audit indicator.

Outcome is provided via an external audit responses file.

id: IDsStringPattern = None¶

type: Literal['audit'] = None¶

name: str | None = None¶

assessment: List[AuditAssessmentRule] = None¶

class src.asqi.schemas.ScoreCard(/, **data: Any)¶

Bases: pydantic.BaseModel

Complete grading score card configuration.

score_card_name: str = None¶

indicators: List[ScoreCardIndicator | AuditScoreCardIndicator] = None¶

class src.asqi.schemas.AuditResponse(/, **data: Any)¶

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Attributes:

__class_vars__: The names of the class variables defined on the model. __private_attributes__: Metadata about the private attributes of the model. __signature__: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields. __pydantic_core_schema__: The core schema of the model. __pydantic_custom_init__: Whether the model has a custom __init__ function. __pydantic_decorators__: Metadata containing the decorators defined on the model.

This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to: __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models. __pydantic_post_init__: The name of the post-init method for the model, if defined. __pydantic_root_model__: Whether the model is a [RootModel][pydantic.root_model.RootModel]. __pydantic_serializer__: The pydantic-core SchemaSerializer used to dump instances of the model. __pydantic_validator__: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects. __pydantic_computed_fields__: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra]: is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

indicator_id: str = None¶

sut_name: str | None = None¶

selected_outcome: str = None¶

notes: str | None = None¶

class src.asqi.schemas.AuditResponses(/, **data: Any)¶

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Attributes:

This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to: __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra]: is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

responses: List[AuditResponse]¶