Install & Compatibility
Where this runs
tested against v0.2.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 2.622s · 119.2MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 12.4s · import 2.438s · 127MB
129MB installed
● package 129MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
create_llm_as_judge
✓ from openevals.llm import create_llm_as_judge
CORRECTNESS_PROMPT
✓ from openevals.prompts import CORRECTNESS_PROMPT
CONCISENESS_PROMPT
✓ from openevals.prompts import CONCISENESS_PROMPT
HALLUCINATION_PROMPT
✓ from openevals.prompts import HALLUCINATION_PROMPT
This quickstart demonstrates how to set up and run a basic LLM-as-judge correctness evaluation. It uses a prebuilt prompt and an OpenAI model. Ensure your `OPENAI_API_KEY` environment variable is set for the example to run successfully. The evaluator returns a dictionary containing a score and a comment based on the LLM's judgment.
import os
from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT
# Ensure your OpenAI API key is set as an environment variable
# For example: os.environ["OPENAI_API_KEY"] = "sk-..."
# For quickstart, we use .get to avoid immediate error if not set, but it's required for actual use.
if not os.environ.get("OPENAI_API_KEY"): print("WARNING: OPENAI_API_KEY not set. Quickstart will fail without it.")
# Create a correctness evaluator using an LLM-as-judge
correctness_evaluator = create_llm_as_judge(
prompt=CORRECTNESS_PROMPT,
model="openai:o3-mini", # 'o3-mini' refers to gpt-3.5-turbo-0125
)
# Define inputs, outputs, and reference outputs for evaluation
inputs = "How much has the price of doodads changed in the past year?"
outputs = "Doodads have increased in price by 10% in the past year."
reference_outputs = "The price of doodads has decreased by 50% in the past year."
# Run the evaluator
eval_result = correctness_evaluator(
inputs=inputs,
outputs=outputs,
reference_outputs=reference_outputs
)
print(eval_result)
# Expected output (score might vary slightly based on LLM, but structure is consistent):
# { 'key': 'score', 'score': False, 'comment': 'The provided answer stated that doodads increased in price by 10%, which conflicts with the reference output...' }
Errors
Common errors & fixes
TypeError: list indices must be integers or slices, not str
This error typically occurs when an LLM-as-judge evaluator, especially with certain models like `google_genai:gemini-2.0-flash`, returns a list instead of the expected dictionary, and the `openevals` library attempts to access a key (e.g., 'score', 'reasoning') using string indexing on that list.
fixThis issue was reported and resolved in later versions of `openevals`. Ensure you are using the latest compatible versions of `openevals` and any related LangChain or LLM client libraries. If the issue persists, explicitly check the LLM's raw response format and adapt your parsing logic or report it as a bug to the `openevals` maintainers if using a custom `output_schema`.
WARNING: OPENAI_API_KEY not set. Quickstart will fail without it.
Many of the core evaluators in `openevals`, particularly LLM-as-judge evaluators, require an API key for an external LLM provider (e.g., OpenAI, Anthropic). This warning indicates that the necessary environment variable, such as `OPENAI_API_KEY`, has not been configured.
fixSet the required API key as an environment variable before running evaluators that rely on external LLMs. For example, in your shell, use `export OPENAI_API_KEY="your_key_here"` or set it programmatically via `os.environ["OPENAI_API_KEY"] = "your_key_here"`.
The `model` parameter in `create_llm_as_judge` expects specific string formats (e.g., "openai:o3-mini"). This implies integration with LangChain's model abstraction and might differ from direct LLM client instantiation methods.
`openevals`'s `create_llm_as_judge` function expects the `model` parameter to be a string following a specific format (e.g., `"provider:model_name"`) to correctly integrate with LangChain's model abstraction. Passing an incorrect string format or a direct LLM client object can lead to configuration errors or unexpected behavior.
fixRefer to the `openevals` documentation or LangChain's model integration guides for the correct model string formats for your chosen LLM provider. Ensure the string matches the expected pattern (e.g., `"openai:gpt-4"`, `"google_genai:gemini-pro"`).
KeyError: 'score' (or similar for 'comment', 'reasoning') after using custom output_schema
When providing a custom `output_schema` to `create_llm_as_judge`, the default return structure of the evaluator (which includes keys like 'score' and 'comment') is overridden. Downstream code that expects the default keys will then fail with a `KeyError`.
fixIf using a custom `output_schema`, be explicit about the expected return structure and update any downstream code that processes the evaluation results to match your custom schema. Avoid relying on the default `score` and `comment` keys if they are not part of your custom schema.
Upgrade
Version history
0.2.0latest on PyPI · released Apr 7, 2026
Audit
Dependencies
openairequiredRequired for LLM-as-judge evaluators that use OpenAI models.