src.asqi.schemas¶

Classes¶

`SystemInput`	Defines a system input that the container requires.
`InputParameter`	Defines a parameter that can be passed to the test container.
`OutputMetric`	Defines a metric that will be present in the test container's output.
`OutputArtifact`	Defines a file artifact generated by the test container.
`Manifest`	Schema for the manifest.yaml file inside a test container.
`LLMAPIConfig`	Configuration for LLM API systems.
`SystemDefinition`	A single system definition.
`SystemsConfig`	Schema for the top-level systems configuration file.
`TestDefinitionBase`	Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.
`TestDefinition`	A single test to be executed.
`TestSuiteDefault`	Default values that apply to all tests in the suite unless overridden.
`SuiteConfig`	Schema for the top-level Test Suite configuration file.
`ScoreCardFilter`	Defines which test results an indicator applies to.
`AssessmentRule`	Individual assessment outcome with condition.
`ScoreCardIndicator`	Individual score card indicator with filtering and assessment.
`ScoreCard`	Complete grading score card configuration.

Module Contents¶

class src.asqi.schemas.SystemInput(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a system input that the container requires.

name: str = None¶

type: str = None¶

required: bool = None¶

description: str | None = None¶

class src.asqi.schemas.InputParameter(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a parameter that can be passed to the test container.

name: str¶

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object']¶

required: bool = False¶

description: str | None = None¶

class src.asqi.schemas.OutputMetric(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a metric that will be present in the test container’s output.

name: str¶

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object']¶

description: str | None = None¶

class src.asqi.schemas.OutputArtifact(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines a file artifact generated by the test container.

name: str¶

path: str¶

description: str | None = None¶

class src.asqi.schemas.Manifest(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the manifest.yaml file inside a test container.

name: str = None¶

version: str¶

description: str | None = None¶

input_systems: List[SystemInput] = None¶

input_schema: List[InputParameter] = None¶

output_metrics: List[str] | List[OutputMetric] = None¶

output_artifacts: List[OutputArtifact] | None = None¶

class src.asqi.schemas.LLMAPIConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Configuration for LLM API systems.

base_url: str = None¶

model: str = None¶

env_file: str | None = None¶

api_key: str | None = None¶

class src.asqi.schemas.SystemDefinition(/, **data: Any)¶

Bases: pydantic.BaseModel

A single system definition.

type: str = None¶

params: Dict[str, Any] = None¶

class src.asqi.schemas.SystemsConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the top-level systems configuration file.

systems: Dict[str, SystemDefinition] = None¶

class src.asqi.schemas.TestDefinitionBase(/, **data: Any)¶

Bases: pydantic.BaseModel

Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.

systems_under_test: List[str] | None = None¶

systems: Dict[str, str] | None = None¶

tags: List[str] | None = None¶

params: Dict[str, Any] | None = None¶

volumes: Dict[str, Any] | None = None¶

class src.asqi.schemas.TestDefinition(/, **data: Any)¶

Bases: TestDefinitionBase

A single test to be executed.

name: str = None¶

image: str = None¶

class src.asqi.schemas.TestSuiteDefault(/, **data: Any)¶

Bases: TestDefinitionBase

Default values that apply to all tests in the suite unless overridden.

class src.asqi.schemas.SuiteConfig(/, **data: Any)¶

Bases: pydantic.BaseModel

Schema for the top-level Test Suite configuration file.

suite_name: str¶

test_suite_default: TestSuiteDefault | None = None¶

test_suite: List[TestDefinition]¶

class src.asqi.schemas.ScoreCardFilter(/, **data: Any)¶

Bases: pydantic.BaseModel

Defines which test results an indicator applies to.

test_name: str = None¶

class src.asqi.schemas.AssessmentRule(/, **data: Any)¶

Bases: pydantic.BaseModel

Individual assessment outcome with condition.

outcome: str = None¶

condition: Literal['equal_to', 'greater_than', 'less_than', 'greater_equal', 'less_equal', 'all_true', 'any_false', 'count_equals'] = None¶

threshold: int | float | bool | None = None¶

description: str | None = None¶

class src.asqi.schemas.ScoreCardIndicator(/, **data: Any)¶

Bases: pydantic.BaseModel

Individual score card indicator with filtering and assessment.

name: str = None¶

apply_to: ScoreCardFilter = None¶

metric: str = None¶

assessment: List[AssessmentRule] = None¶

class src.asqi.schemas.ScoreCard(/, **data: Any)¶

Bases: pydantic.BaseModel

Complete grading score card configuration.

score_card_name: str = None¶

indicators: List[ScoreCardIndicator] = None¶