scorable.calibration_runs

Classes

CalibrationRuns

Calibration runs API.

Module Contents

class scorable.calibration_runs.CalibrationRuns(client_context: scorable.utils.ClientContextCallable)

Calibration runs API.

A calibration run measures the agreement between an evaluator and the human annotations on a dataset, producing per-example results and aggregate metrics. Runs execute asynchronously: a freshly created run has status="pending"; poll get() until it is completed or failed to read its metrics.

Note

The construction of the API instance should be handled by accessing an attribute of a scorable.client.Scorable instance.

Parameters:

client_context (scorable.utils.ClientContextCallable)

async acreate(*, evaluator_id: str, dataset_id: str, score_config_id: str | None = None, evaluator_version_id: str | None = None, _request_timeout: int | None = None, _client: scorable.generated.openapi_aclient.ApiClient) → scorable.generated.openapi_aclient.models.calibration_run.CalibrationRun

Asynchronously start a calibration run.

Parameters:

  • evaluator_id (str)

  • dataset_id (str)

  • score_config_id (Optional[str])

  • evaluator_version_id (Optional[str])

  • _request_timeout (Optional[int])

  • _client (scorable.generated.openapi_aclient.ApiClient)

Return type:

scorable.generated.openapi_aclient.models.calibration_run.CalibrationRun

async aget(run_id: str, *, _request_timeout: int | None = None, _client: scorable.generated.openapi_aclient.ApiClient) → scorable.generated.openapi_aclient.models.calibration_run.CalibrationRun

Asynchronously get a calibration run by ID.

Parameters:

  • run_id (str)

  • _request_timeout (Optional[int])

  • _client (scorable.generated.openapi_aclient.ApiClient)

Return type:

scorable.generated.openapi_aclient.models.calibration_run.CalibrationRun

async alist(*, evaluator_id: str | None = None, limit: int = 100) → AsyncIterator[scorable.generated.openapi_aclient.models.calibration_run.CalibrationRun]

Asynchronously iterate through calibration runs.

Parameters:

  • evaluator_id (Optional[str])

  • limit (int)

Return type:

AsyncIterator[scorable.generated.openapi_aclient.models.calibration_run.CalibrationRun]

async alist_items(run_id: str, *, limit: int = 100) → AsyncIterator[scorable.generated.openapi_aclient.models.calibration_run_item.CalibrationRunItem]

Asynchronously iterate through the per-example results of a calibration run.

Parameters:

  • run_id (str)

  • limit (int)

Return type:

AsyncIterator[scorable.generated.openapi_aclient.models.calibration_run_item.CalibrationRunItem]

create(*, evaluator_id: str, dataset_id: str, score_config_id: str | None = None, evaluator_version_id: str | None = None, _request_timeout: int | None = None, _client: scorable.generated.openapi_client.ApiClient) → scorable.generated.openapi_client.models.calibration_run.CalibrationRun

Start a calibration run against a labelled dataset and return it (status="pending").

Parameters:

  • evaluator_id (str) – The saved evaluator to calibrate.

  • dataset_id (str) – The dataset whose published annotations to calibrate against.

  • score_config_id (Optional[str]) – Optional score config; omitting it uses the dataset’s continuous scores.

  • evaluator_version_id (Optional[str]) – Optional specific evaluator version.

  • _request_timeout (Optional[int])

  • _client (scorable.generated.openapi_client.ApiClient)

Return type:

scorable.generated.openapi_client.models.calibration_run.CalibrationRun

get(run_id: str, *, _request_timeout: int | None = None, _client: scorable.generated.openapi_client.ApiClient) → scorable.generated.openapi_client.models.calibration_run.CalibrationRun

Get a calibration run by ID (poll this for status and metrics).

Parameters:

  • run_id (str)

  • _request_timeout (Optional[int])

  • _client (scorable.generated.openapi_client.ApiClient)

Return type:

scorable.generated.openapi_client.models.calibration_run.CalibrationRun

list(*, evaluator_id: str | None = None, limit: int = 100, _client: scorable.generated.openapi_client.ApiClient) → Iterator[scorable.generated.openapi_client.models.calibration_run.CalibrationRun]

Iterate through calibration runs, optionally filtered by evaluator.

Parameters:

  • evaluator_id (Optional[str]) – Filter by evaluator external id.

  • limit (int) – Number of entries to iterate through at most.

  • _client (scorable.generated.openapi_client.ApiClient)

Return type:

Iterator[scorable.generated.openapi_client.models.calibration_run.CalibrationRun]

list_items(run_id: str, *, limit: int = 100, _client: scorable.generated.openapi_client.ApiClient) → Iterator[scorable.generated.openapi_client.models.calibration_run_item.CalibrationRunItem]

Iterate through the per-example results of a calibration run.

Parameters:

  • run_id (str)

  • limit (int)

  • _client (scorable.generated.openapi_client.ApiClient)

Return type:

Iterator[scorable.generated.openapi_client.models.calibration_run_item.CalibrationRunItem]

client_context