src.asqi.storage.s3

Pure S3 primitives for both asqi-engineer and asqi-runner.

This module intentionally has no dependency on core.config (runner settings), core.logging (runner structlog), or FastAPI. Callers inject S3 endpoint / credentials / region via S3ClientConfig and pass the resulting client around.

Runner-side wrappers that read settings and add UploadFile support live in services/asqi-runner/storage/s3.py and delegate the actual S3 calls back to this module.

Attributes

logger
AddressingStyle

Classes

S3ClientConfig

Caller-injected S3 client configuration.

Functions

make_s3_client(→ Any)	Build (and cache) a boto3 S3 client for config.
ensure_bucket_exists(→ None)	Create bucket if it does not already exist.
upload_file(→ None)	Upload a local file to s3://<bucket>/<key>.
download_file_to_path(→ None)	Download s3://<bucket>/<key> to local_path.
upload_folder(→ list[str])	Recursively upload every file under local_dir to key_prefix.
download_prefix_to_folder(→ list[str])	Mirror every object under s3://<bucket>/<key_prefix>/ into local_dir.

Module Contents

src.asqi.storage.s3.logger

src.asqi.storage.s3.AddressingStyle

class src.asqi.storage.s3.S3ClientConfig

Caller-injected S3 client configuration.

Frozen so it can be used as a cache key for make_s3_client().

Attributes:

endpoint_url: S3 endpoint (e.g. AWS regional endpoint or a MinIO URL). region: AWS region name (e.g. "ap-southeast-1"). Some MinIO

deployments accept any non-empty string here.

addressing_style: "auto" (default), "path" (required for

most MinIO deployments), or "virtual".

access_key: Optional static access key. When both access_key

and secret_key are None, boto3’s default credential chain is used (IRSA, instance profile, env vars, etc.).

secret_key: Optional static secret key.

endpoint_url: str

region: str

addressing_style: AddressingStyle = 'auto'

access_key: str | None = None

secret_key: str | None = None

src.asqi.storage.s3.make_s3_client(config: S3ClientConfig) → Any

Build (and cache) a boto3 S3 client for config.

The cache key is the full S3ClientConfig instance, so callers that rotate credentials must construct a new config. The cache is bounded at 4 entries to avoid unbounded growth in test fixtures.

src.asqi.storage.s3.ensure_bucket_exists(s3_client: Any, bucket: str, region: str) → None

Create bucket if it does not already exist.

A successful head_bucket short-circuits without creating anything. A 404 / missing bucket response triggers creation. Other boto3 ClientError responses, such as 403, are re-raised.

src.asqi.storage.s3.upload_file(s3_client: Any, local_path: str | pathlib.Path, bucket: str, key: str, content_type: str | None = None) → None

Upload a local file to s3://<bucket>/<key>.

Raises:

FileNotFoundError: If local_path does not exist. Exception: Any boto3 ClientError raised by put_object is

propagated; callers decide whether to retry or fail-closed.

src.asqi.storage.s3.download_file_to_path(s3_client: Any, bucket: str, key: str, local_path: str | pathlib.Path) → None

Download s3://<bucket>/<key> to local_path.

Parent directories are created if missing. Raises on any boto3 error.

src.asqi.storage.s3.upload_folder(s3_client: Any, local_dir: str | pathlib.Path, bucket: str, key_prefix: str) → list[str]

Recursively upload every file under local_dir to key_prefix.

Returns the list of S3 keys written, in the same order they were uploaded. Relative paths under local_dir map directly onto S3 keys under key_prefix (POSIX separators).

Raises:

ValueError: If local_dir does not exist or is not a directory.

src.asqi.storage.s3.download_prefix_to_folder(s3_client: Any, bucket: str, key_prefix: str, local_dir: str | pathlib.Path) → list[str]

Mirror every object under s3://<bucket>/<key_prefix>/ into local_dir.

Uses the list_objects_v2 paginator so it handles prefixes with thousands of objects. Returns the list of relative paths written (relative to local_dir), in the order they were downloaded.