src.asqi.pytest_plugin¶

pytest plugin for ASQI test container authors.

Registered via the pytest11 entry point when asqi-engineer[test] is installed. Provides the container_root and asqi_execute fixtures and the requires_hf_token / requires_llm_api marks.

Fixtures¶

container_root: Absolute path to the container’s root directory (parent of tests/).
asqi_execute: High-level runner that mirrors asqi execute CLI. Accepts the same config file arguments, resolves YAML → param dicts, sets up mount dirs, imports entrypoint.main in-process, and returns (ContainerOutput, exit_code).

Marks¶

requires_hf_token: Skip the test when HF_TOKEN is not set in the environment.
requires_llm_api: Skip the test when no recognised LLM API key is set in the environment.

Attributes¶

`logger`
`requires_hf_token`
`requires_llm_api`

Functions¶

`extract_container_output`(...)	DBOS-free equivalent of `asqi.output.extract_container_json_output_fields`.
`container_root`(→ pathlib.Path)	Absolute path to the container's root directory (parent of `tests/`).
`asqi_execute`(→ collections.abc.Callable[Ellipsis, ...)	High-level container runner that mirrors the `asqi execute` CLI.

Module Contents¶

src.asqi.pytest_plugin.logger¶

src.asqi.pytest_plugin.requires_hf_token¶

src.asqi.pytest_plugin.requires_llm_api¶

src.asqi.pytest_plugin.extract_container_output(container_json_output: dict[str, Any]) → asqi.response_schemas.ContainerOutput¶

DBOS-free equivalent of asqi.output.extract_container_json_output_fields.

The version in asqi.output calls DBOS.logger.warning() on validation failures, which requires a running DBOS context. This version uses the standard logging module instead so it is safe in test environments.

src.asqi.pytest_plugin.container_root(request: pytest.FixtureRequest) → pathlib.Path¶

Absolute path to the container’s root directory (parent of tests/).

Derived automatically from the location of the test file so container authors never need to compute Path(__file__).parent.parent themselves.

src.asqi.pytest_plugin.asqi_execute(container_root: pathlib.Path, monkeypatch: pytest.MonkeyPatch, tmp_path: pathlib.Path) → collections.abc.Callable[Ellipsis, list[tuple[asqi.response_schemas.ContainerOutput, int]]]¶

High-level container runner that mirrors the asqi execute CLI.

Returns a callable that accepts the same config file arguments as asqi execute and:

Resolves each path against container_root.
Loads and validates the YAML files using the same logic as the CLI.
Resolves dataset name references when datasets_config is provided.
Derives systems_params / test_params via the CLI’s config resolution pipeline (create_test_execution_plan).
Sets up OUTPUT_MOUNT_PATH / INPUT_MOUNT_PATH in tmp_path.
Imports entrypoint.main in-process (importable because the container root is on sys.path via pythonpath = ["."] in pyproject.toml).
Patches sys.argv, redirects stdout, calls main(), captures the exit code, parses the JSON output, and returns (ContainerOutput, exit_code) for a single plan entry, or a list of such tuples for multiple entries.

Usage:

def test_scores_within_range(mock_full_run, api_success, asqi_execute):
    mock_full_run(dataset_rows=[...], api_side_effect=api_success())
    results = asqi_execute(
        test_suite_config="config/suite.yaml",
        systems_config="config/systems.yaml",
        datasets_config="config/datasets.yaml",  # optional
    )
    output, exit_code = results[0]
    assert exit_code == 0