src.asqi.score_card_engine

Attributes

logger

Classes

ScoreCardEvaluationResult	Result of evaluating a single score_card indicator.
ScoreCardEngine	Core score_card evaluation engine.

Functions

parse_metric_path(→ List[str])	Parse a metric path supporting both dot notation and bracket notation.
get_nested_value(→ Tuple[Any, Optional[str]])	Extract a nested value from a dictionary using dot/bracket notation.

Module Contents

src.asqi.score_card_engine.logger

src.asqi.score_card_engine.parse_metric_path(path: str) → List[str]

Parse a metric path supporting both dot notation and bracket notation.

Examples:

‘success’ -> [‘success’] ‘vulnerability_stats.Toxicity.overall_pass_rate’ -> [‘vulnerability_stats’, ‘Toxicity’, ‘overall_pass_rate’] ‘probe_results[“encoding.InjectHex”][“encoding.DecodeMatch”].passed’ -> [‘probe_results’, ‘encoding.InjectHex’, ‘encoding.DecodeMatch’, ‘passed’] ‘probe_results[“encoding.InjectHex”].total_attempts’ -> [‘probe_results’, ‘encoding.InjectHex’, ‘total_attempts’]

Args:

path: Metric path string to parse

Returns:

List of keys to traverse

Raises:

ValueError: If path contains invalid syntax

src.asqi.score_card_engine.get_nested_value(data: Dict[str, Any], path: str) → Tuple[Any, str | None]

Extract a nested value from a dictionary using dot/bracket notation.

Args:

data: Dictionary to extract value from path: Path to the nested value (e.g., ‘a.b.c’ or ‘a[“key.with.dots”].c’)

Returns:

Tuple of (value, error_message). If successful, error_message is None. If failed, value is None and error_message describes the issue.

class src.asqi.score_card_engine.ScoreCardEvaluationResult(indicator_name: str, test_name: str)

Result of evaluating a single score_card indicator.

indicator_name

test_name

outcome: str | None = None

metric_value: Any | None = None

test_result_id: str | None = None

sut_name: str | None = None

computed_value: int | float | bool | None = None

details: str = ''

description: str | None = None

error: str | None = None

to_dict() → Dict[str, Any]

Convert to dictionary for JSON serialization.

class src.asqi.score_card_engine.ScoreCardEngine

Core score_card evaluation engine.

filter_results_by_test_name(test_results: List[asqi.workflow.TestExecutionResult], target_test_name: str) → List[asqi.workflow.TestExecutionResult]

Filter test results to only include those with the specified test name.

Args:

test_results: List of test execution results to filter target_test_name: Name of test to filter for

Returns:

Filtered list of test results matching the target test name

extract_metric_values(test_results: List[asqi.workflow.TestExecutionResult], metric_path: str) → List[Any]

Extract metric values from test results using the specified path.

Supports both flat and nested metric access: - Flat: ‘success’, ‘score’ - Nested: ‘vulnerability_stats.Toxicity.overall_pass_rate’ - Bracket notation: ‘probe_results[“encoding.InjectHex”][“encoding.DecodeMatch”].passed’

Args:

test_results: List of test execution results metric_path: Path to metric within test results (supports dot and bracket notation)

Returns:

List of extracted metric values

apply_condition_to_value(value: Any, condition: str, threshold: int | float | None = None) → Tuple[bool, str]

Apply the specified condition to a single value.

Args:

value: Value to evaluate condition: Condition to apply (e.g., ‘equal_to’, ‘greater_than’) threshold: Threshold value for comparison (required for most conditions)

Returns:

Tuple of (condition_met, description)

evaluate_indicator(test_results: List[asqi.workflow.TestExecutionResult], indicator: asqi.schemas.ScoreCardIndicator) → List[ScoreCardEvaluationResult]

Evaluate a single score_card indicator against individual test results.

Args:

test_results: List of test execution results to evaluate indicator: Score card indicator configuration

Returns:

List of evaluation results for each matching test

evaluate_scorecard(test_results: List[asqi.workflow.TestExecutionResult], score_card: asqi.schemas.ScoreCard) → List[Dict[str, Any]]

Evaluate a complete grading score_card against test results.

Args:

test_results: List of test execution results to evaluate score_card: Complete score card configuration

Returns:

List of evaluation result dictionaries