Registry / data / databricks-labs-dqx

databricks-labs-dqx

library0.16.0pypypi✓ verified 29d ago

Data Quality eXtended (DQX) is a Python library for defining, executing, and monitoring data quality checks. It leverages Apache Spark and is designed to integrate seamlessly within the Databricks ecosystem, supporting features like Delta Lake, DLT, and Unity Catalog. The library is actively maintained with frequent releases, currently at version 0.13.0, introducing features like an enhanced data quality dashboard and AI-assisted rule generation.

data database

pip install databricks-labs-dqx pyspark

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · DATABRICKS-LABS-DQ

databricks-labs-dqx

datapythonv0.16.0

Install

37.2s avg

Import

—

Disk

587MB

Pass rate

8/ 10

Env Coverage8 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.16.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✓ —

✓ 38.8s

py 3.11

✓ —

✓ 37.4s

py 3.12

✓ —

✓ 35.6s

py 3.13

✓ —

✓ 36.9s

py 3.9

✕ build_error

587MB installed

● package 587MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

DQEngine

✓ from dqx.core.dq_engine import DQEngine

DQRule

✓ from dqx.core.rule import DQRule

DQGenerator

✓ from dqx.core.dq_generator import DQGenerator

Introduced in v0.12.0 for AI-assisted rule generation.

This quickstart demonstrates how to set up a local SparkSession, create a sample DataFrame, define data quality rules using `DQRule` objects, and execute them with `DQEngine.run_checks()`. The results are displayed and the SparkSession is stopped.

from pyspark.sql import SparkSession from dqx.core.dq_engine import DQEngine from dqx.core.rule import DQRule # Initialize SparkSession (for local execution) spark = SparkSession.builder.appName("DQXQuickstart") \ .master("local[*]") \ .getOrCreate() # Create a sample DataFrame data = [("A", 1, "2023-01-01"), ("B", 2, "2023-01-02"), ("C", None, "2023-01-03"), ("D", 4, "2023-01-01")] columns = ["id", "value", "event_date"] df = spark.createDataFrame(data, columns) # Define data quality rules rules = [ DQRule("value_not_null", "value IS NOT NULL", "value column should not be null"), DQRule("id_is_unique", "COUNT(DISTINCT id) = COUNT(id)", "id column should be unique", dq_check_type="Aggregated"), DQRule("event_date_freshness", "event_date >= '2023-01-01'", "event_date should be recent") ] # Initialize DQEngine dq_engine = DQEngine(spark_session=spark) # Apply checks results = dq_engine.run_checks(df, checks=rules) # Print results print("Data Quality Check Results:") results.display() # Stop SparkSession spark.stop()

dqx --version

Debug

Known issues

gotchaDQX requires an active SparkSession. When running outside a Databricks environment (e.g., locally), you must explicitly install `pyspark` and create a `SparkSession` instance before initializing `DQEngine`.

fix

Ensure `pyspark` is installed (`pip install pyspark`) and a `SparkSession` is created and passed to the `DQEngine` constructor (e.g., `DQEngine(spark_session=my_spark_session)`).

affects: All versions

breakingStarting from v0.7.1, the `apply_checks` method enforces strict type validation for rules. Rules must be passed as a list of `DQRule` objects. Passing dictionaries or other types directly will raise a `TypeError`.

fix

Migrate any dictionary-based rule definitions to explicit `DQRule` objects: `DQRule('rule_name', 'rule_condition', 'rule_description')`. Refer to the `DQRule` constructor for available parameters.

affects: >=0.7.1

breakingThe Data Quality Dashboard has been significantly enhanced and restructured in v0.13.0. Existing custom dashboard integrations or deployment scripts might require updates to align with the new three-tab structure and underlying APIs.

fix

Review the DQX documentation for the new dashboard structure and update any code or configurations related to dashboard generation or access. Re-evaluate if existing custom UI components are still compatible.

affects: >=0.13.0

gotchaThe `DQGenerator` class for AI-assisted rule generation (v0.12.0) and ODCS Data Contract rule generation (v0.11.0) introduces new APIs. Users expecting rule generation might need to adopt these new classes and methods instead of manual rule creation.

fix

For AI-assisted rules, use `from dqx.core.dq_generator import DQGenerator` and its methods like `generate_dq_rules_ai_assisted`. For ODCS integration, refer to the documentation on generating rules from data contracts.

affects: >=0.11.0

Upgrade

Version history

0.16.0latest on PyPI · released Aug 13, 2026

Audit

Dependencies

pysparkrequiredDQX is built on Apache Spark and requires an active SparkSession to run any data quality checks. PySpark is needed for local development outside of a Databricks environment.

Agent activity

30 hits · last 30 days

node

Bingbot

OpenAI (training)

Resources

githubhttps://github.com/databrickslabs/dqx ↗

packagehttps://pypi.org/project/databricks-labs-dqx/ ↗

homepagehttps://databricks.com ↗