datacompy

library1.0.4pypypi✓ verified 29d ago

DataComPy is a powerful Python library designed to simplify the comparison of two DataFrames, supporting various backends including Pandas, Spark, Polars, and Snowflake. It generates detailed, human-readable reports highlighting discrepancies at both row and column levels, and allows for the specification of absolute or relative tolerance levels for numeric comparisons. The library is currently at version 0.19.5 and is actively progressing towards a v1 release, with new features targeting development branches while the 0.19.x branch is for maintenance and critical fixes.

data testing

pip install datacompy

github ↗package ↗docs ↗

INSTALL

IMPORT

SIG · DATACOMPY

datacompy

datapythonv1.0.4

Install

20.4s avg

Import

2563ms

Disk

687MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.0.4 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

4/5 runs

✓ 20.36s

py 3.11

4/5 runs

✓ 19.16s

py 3.12

4/5 runs

✓ 19.12s

py 3.13

4/5 runs

✓ 18.76s

py 3.9

✕ build_error

✓ 24.66s

687MB installed

● package 687MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

PandasCompare

✓ from datacompy import PandasCompare

✗ import datacompy

PolarsCompare

✓ from datacompy import PolarsCompare

✗ import datacompy

BaseCompare

✓ from datacompy import BaseCompare

✗ import datacompy

This quickstart demonstrates how to compare two Pandas DataFrames using `datacompy.Compare`. It initializes two sample DataFrames, then creates a `Compare` object specifying the join column and tolerance levels for numeric comparisons. Finally, it prints a comprehensive report of differences and checks for an overall match.

import pandas as pd import datacompy from io import StringIO data1 = """acct_id,dollar_amt,name 1,123.45,Alice 2,67.89,Bob 3,99.99,Charlie 4,10.00,David """ df1 = pd.read_csv(StringIO(data1)) data2 = """acct_id,dollar_amt,name 1,123.45,Alice 2,67.90,Bobbert 3,99.99,Charlie 5,11.00,Eve """ df2 = pd.read_csv(StringIO(data2)) # Perform comparison, joining on 'acct_id' # abs_tol and rel_tol are crucial for float comparisons compare = datacompy.Compare( df1, df2, join_columns='acct_id', abs_tol=0.001, # Absolute tolerance for numeric fields rel_tol=0.0 # Relative tolerance for numeric fields ) # Print a detailed report print(compare.report()) # Check if dataframes match entirely print(f"DataFrames match: {compare.matches()}")

datacompy --version

Debug

Known issues

breaking`LegacySparkCompare` and `SparkPandasCompare` classes were removed in version 0.17.0. For Spark DataFrame comparison, use `SparkSQLCompare` (from `datacompy.spark.sql`) or the general `datacompy.is_match` function with the appropriate backend setup (often leveraging Fugue).

fix

Migrate to `datacompy.spark.sql.SparkSQLCompare` or `datacompy.is_match` with Fugue backend. Consult the documentation for specific backend usage.

affects: >=0.17.0

gotchaThe default `datacompy.Compare` (Pandas native) requires DataFrames to fit into memory. Comparing very large datasets may lead to out-of-memory errors. For larger-than-memory datasets, consider using the Spark SQL or Polars backend implementations, which are more performant for big data.

fix

For large datasets, use `datacompy` with Spark, Polars, or Fugue backends after installing the relevant extras (e.g., `pip install datacompy[spark]`).

affects: All

gotchaWhen comparing floating-point numbers, minor precision differences can cause mismatches even if values are conceptually the same. Always use `abs_tol` (absolute tolerance) and/or `rel_tol` (relative tolerance) parameters in `datacompy.Compare` to account for these expected deviations.

fix

Set appropriate `abs_tol` and `rel_tol` values when initializing the `Compare` object (e.g., `abs_tol=0.0001`, `rel_tol=0.01`).

affects: All

gotchaDataComPy's duplicate row matching logic can be 'naïve' if `join_columns` do not uniquely identify rows. If many duplicates exist, `datacompy` sorts by other fields to create a temporary ID, which might not align with desired matching.

fix

Ensure `join_columns` provide sufficient granularity to uniquely identify rows, or pre-process duplicates if specific matching behavior is required.

affects: All

gotchaIf 'DATACOMPY_NULL' exists as a legitimate string value in your `join_columns`, it can conflict with how `datacompy` internally handles null values during duplicate matching, potentially causing merge failures.

fix

Either rename or replace 'DATACOMPY_NULL' values in your join columns before comparison, or fill nulls with a different sentinel value of your choice.

affects: All

deprecatedDataComPy is actively moving towards a v1 release. The `0.19.x` branch will receive only dependency updates and critical bug fixes, with no new features. Future `v1` releases may introduce breaking changes as development targets `v1` branches (`develop` and `main`).

fix

Monitor GitHub releases and documentation for `v1` migration guides when upgrading from `0.19.x`.

affects: 0.19.x

gotchaPython 3.12 and above currently have limited support with Spark and Ray within the Fugue backend. Pandas and Polars comparisons should work fine.

fix

If using Spark or Ray with Fugue, consider using Python versions <3.12 until full compatibility is announced. Otherwise, Pandas and Polars backends are supported.

affects: >=3.12.0

Upgrade

Version history

1.0.4latest on PyPI · released Jul 30, 2026

Audit

Dependencies

pandasrequiredCore dependency for Pandas DataFrame comparisons.

pysparkoptionalOptional, required for Spark DataFrame comparisons.

polarsoptionalOptional, required for Polars DataFrame comparisons.

fugueoptionalOptional, enables comparison across various backends like Dask, DuckDB, Ray. Integrates with DataComPy.

snowflake-snowpark-pythonoptionalOptional, required for Snowflake DataFrame comparisons.

Agent activity