src.asqi.schemas

Attributes

SystemConfig

Classes

SystemInput	Defines a system input that the container requires.
InputParameter	Defines a parameter that can be passed to the test container.
OutputMetric	Defines a metric that will be present in the test container's output.
OutputArtifact	Defines a file artifact generated by the test container.
Manifest	Schema for the manifest.yaml file inside a test container.
SystemDefinition	Base system definition.
LLMAPIParams	Parameters for the LLM API systems.
LLMAPIConfig	Configuration for LLM API systems.
GenericSystemConfig	Generic system configuration for system types without specific validation.
SystemsConfig	Schema for the top-level systems configuration file.
TestDefinitionBase	Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.
TestDefinition	A single test to be executed.
TestSuiteDefault	Default values that apply to all tests in the suite unless overridden.
SuiteConfig	Schema for the top-level Test Suite configuration file.
ScoreCardFilter	Defines which test results an indicator applies to.
AssessmentRule	Individual assessment outcome with condition.
ScoreCardIndicator	Individual score card indicator with filtering and assessment.
ScoreCard	Complete grading score card configuration.

Module Contents

class src.asqi.schemas.SystemInput(/, **data: Any)

Bases: pydantic.BaseModel

Defines a system input that the container requires.

name: str = None

type: str = None

required: bool = None

description: str | None = None

class src.asqi.schemas.InputParameter(/, **data: Any)

Bases: pydantic.BaseModel

Defines a parameter that can be passed to the test container.

name: str

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object']

required: bool = False

description: str | None = None

class src.asqi.schemas.OutputMetric(/, **data: Any)

Bases: pydantic.BaseModel

Defines a metric that will be present in the test container’s output.

name: str

type: Literal['string', 'integer', 'float', 'boolean', 'list', 'object']

description: str | None = None

class src.asqi.schemas.OutputArtifact(/, **data: Any)

Bases: pydantic.BaseModel

Defines a file artifact generated by the test container.

name: str

path: str

description: str | None = None

class src.asqi.schemas.Manifest(/, **data: Any)

Bases: pydantic.BaseModel

Schema for the manifest.yaml file inside a test container.

name: str = None

version: str

description: str | None = None

host_access: bool = None

input_systems: List[SystemInput] = None

input_schema: List[InputParameter] = None

output_metrics: List[str] | List[OutputMetric] = None

output_artifacts: List[OutputArtifact] | None = None

class src.asqi.schemas.SystemDefinition(/, **data: Any)

Bases: pydantic.BaseModel

Base system definition.

description: str | None = None

provider: str | None = None

class src.asqi.schemas.LLMAPIParams(/, **data: Any)

Bases: pydantic.BaseModel

Parameters for the LLM API systems.

base_url: str = None

model: str = None

env_file: str | None = None

api_key: str | None = None

class src.asqi.schemas.LLMAPIConfig(/, **data: Any)

Bases: SystemDefinition

Configuration for LLM API systems.

type: Literal['llm_api'] = None

params: LLMAPIParams = None

class src.asqi.schemas.GenericSystemConfig(/, **data: Any)

Bases: SystemDefinition

Generic system configuration for system types without specific validation.

This allows backward compatibility and support for system types that don’t have dedicated config classes yet.

type: str = None

params: Dict[str, Any] = None

src.asqi.schemas.SystemConfig

class src.asqi.schemas.SystemsConfig(/, **data: Any)

Bases: pydantic.BaseModel

Schema for the top-level systems configuration file.

Extension Guide:

1. Create a new XXXConfig class inheriting from SystemDefinition. e.g, RESTAPIConfig

2. Create a new XXXParam class for the parameters of the system. e.g. RESTAPIParams 2. Add the new system definition (XXXConfig) to the SystemConfig union type

e.g. SystemConfig = Union[LLMAPIConfig, XXXConfig, …, GenericSystemConfig]

systems: Dict[str, SystemConfig] = None

class src.asqi.schemas.TestDefinitionBase(/, **data: Any)

Bases: pydantic.BaseModel

Base class for test configuration fields shared between TestDefinition and TestSuiteDefault.

systems_under_test: List[str] | None = None

systems: Dict[str, str] | None = None

tags: List[str] | None = None

params: Dict[str, Any] | None = None

volumes: Dict[str, Any] | None = None

class src.asqi.schemas.TestDefinition(/, **data: Any)

Bases: TestDefinitionBase

A single test to be executed.

name: str = None

description: str | None = None

image: str = None

class src.asqi.schemas.TestSuiteDefault(/, **data: Any)

Bases: TestDefinitionBase

Default values that apply to all tests in the suite unless overridden.

class src.asqi.schemas.SuiteConfig(/, **data: Any)

Bases: pydantic.BaseModel

Schema for the top-level Test Suite configuration file.

suite_name: str = None

test_suite_default: TestSuiteDefault | None = None

description: str | None = None

test_suite: List[TestDefinition] = None

class src.asqi.schemas.ScoreCardFilter(/, **data: Any)

Bases: pydantic.BaseModel

Defines which test results an indicator applies to.

test_name: str = None

class src.asqi.schemas.AssessmentRule(/, **data: Any)

Bases: pydantic.BaseModel

Individual assessment outcome with condition.

outcome: str = None

condition: Literal['equal_to', 'greater_than', 'less_than', 'greater_equal', 'less_equal', 'all_true', 'any_false', 'count_equals'] = None

threshold: int | float | bool | None = None

description: str | None = None

class src.asqi.schemas.ScoreCardIndicator(/, **data: Any)

Bases: pydantic.BaseModel

Individual score card indicator with filtering and assessment.

name: str = None

apply_to: ScoreCardFilter = None

metric: str = None

assessment: List[AssessmentRule] = None

class src.asqi.schemas.ScoreCard(/, **data: Any)

Bases: pydantic.BaseModel

Complete grading score card configuration.

score_card_name: str = None

indicators: List[ScoreCardIndicator] = None