Install & Compatibility
Where this runs
tested against v? · npm install
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslnode 18–226 runs
build_error
glibcnode 18–226 runs
build_error
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
evalite
✓ import { evalite } from 'evalite';
✗ const evalite = require('evalite');
Evalite is primarily designed for ESM usage with TypeScript. CommonJS require() may lead to `TypeError: evalite is not a function` in certain setups.
defineConfig
✓ import { defineConfig } from 'evalite/config';
✗ import { defineConfig } from 'evalite';
The `defineConfig` utility for `evalite.config.ts` is imported from a specific subpath to avoid bundling issues and clearly separate configuration from runtime eval definition.
CLI
✓ evalite run
✗ vitest run --evalite
Evalite provides its own CLI for running evaluations (`evalite run`, `evalite watch`), which orchestrates Vitest internally. Direct `vitest` commands typically won't trigger Evalite's specific evaluation logic or UI.
This quickstart demonstrates how to set up `evalite.config.ts`, define an evaluation suite with input data, integrate a mock LLM call, assert expected outputs using Vitest's `expect`, and utilize custom scorers and columns for detailed results. It covers the core `evalite` function and its integration into a project to test LLM-powered applications.
/* package.json excerpt */
/*
{
"name": "my-llm-app",
"version": "0.1.0",
"description": "My LLM powered application",
"type": "module",
"devDependencies": {
"evalite": "^0.19.0",
"vitest": "^4.0.0" /* Ensure compatible Vitest version */
},
"scripts": {
"eval": "evalite run",
"eval:watch": "evalite watch"
}
}
*/
// evalite.config.ts (create this file at your project root)
import { defineConfig } from 'evalite/config';
export default defineConfig({
testTimeout: 60000, // LLM calls can be slow, increase timeout if needed
maxConcurrency: 5,
// Optional: Specify files to run before tests, e.g., for environment variables
// setupFiles: ['./evalite.setup.ts'],
columns: {
'Input Length': ({ input }) => input.query.length,
'LLM Output Starts With': ({ output }) => output?.startsWith('Processed:'),
},
});
// my-llm-app.eval.ts (your evaluation file)
import { evalite } from 'evalite';
import { describe, expect } from 'vitest'; // Vitest utilities are still available
// A dummy LLM function to simulate an actual LLM call
const mockLLM = async (input: string): Promise<string> => {
// Simulate network delay and LLM processing
await new Promise(resolve => setTimeout(resolve, Math.random() * 200 + 100));
return `Processed: ${input.toUpperCase()}`;
};
interface MyEvalInput {
query: string;
expectedOutput: string;
}
evalite<MyEvalInput>('My LLM Application Evaluation Suite', ({ test, evalSuite }) => {
// Define the dataset for your evaluations
evalSuite.data([
{ query: 'hello world', expectedOutput: 'PROCESSED: HELLO WORLD' },
{ query: 'how are you', expectedOutput: 'PROCESSED: HOW ARE YOU' },
{ query: 'evalite rocks', expectedOutput: 'PROCESSED: EVALITE ROCKS' },
{ query: 'quick brown fox', expectedOutput: 'PROCESSED: QUICK BROWN FOX' },
]);
// Define a test for each data point
test('LLM output should match expected format and content', async ({ input, expect: evalExpect }) => {
const llmResponse = await mockLLM(input.query);
evalExpect(llmResponse).toBe(input.expectedOutput);
return { output: llmResponse }; // The output will be available to scorers and columns
});
// Optional: Define custom scorers to evaluate LLM outputs quantitatively
evalSuite.scorer('ExactMatchScorer', ({ output, expected }) => {
const score = output === expected ? 1 : 0;
return { score, verdict: score === 1 ? 'Perfect Match' : 'Mismatch' };
});
// Optional: Add custom columns to display scorer results or other data in the UI
evalSuite.column('Match Score', ({ scores }) => scores.ExactMatchScorer?.score);
});
// To run:
// 1. Install dependencies: npm install evalite vitest
// 2. Run the evals: npm run eval
// (or npm run eval:watch for watch mode and UI at http://localhost:3006)
evalite --version
Errors
Common errors & fixes
Error: evalite requires Vitest version X.Y.Z or higher.
Your installed `vitest` package is older than the minimum version required by `evalite`.
fixUpgrade your `vitest` dependency: `npm install vitest@latest`.
Error: evalite.config.ts not found. Please create one...
After `evalite` v0.16.0, a configuration file (`evalite.config.ts`) is expected at the project root for global settings.
fixCreate `evalite.config.ts` in your project root with content like `import { defineConfig } from 'evalite/config'; export default defineConfig({});`. TypeError: (0 , evalite_1.evalite) is not a function
This usually indicates an incorrect import, often trying to use CommonJS `require()` syntax or an incorrect named import in an ESM context.
fixEnsure you are using `import { evalite } from 'evalite';` and that your project is configured for ESM (e.g., `"type": "module"` in `package.json`). Command failed, Error: Could not locate the bindings file.
This error is related to the `better-sqlite3` dependency that Evalite uses for local data storage, often seen with certain package managers or build environments.
fixTry rebuilding `better-sqlite3`: `npm rebuild better-sqlite3` (or `pnpm rebuild better-sqlite3`). You might also need to approve the build: `pnpm approve-builds`.
Audit
Dependencies
vitestrequiredEvalite is built on top of Vitest and requires it as a peer dependency for running evaluations.
autoevalsoptionalOften used for scoring LLM outputs, as shown in quickstart examples. Not strictly required, but common.