Install & Compatibility
Where this runs
tested against v54.0.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
build_error
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 4.7s · import 0.206s · 288MB
279MB installed
● package 279MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
SessionContext
✓ from datafusion import SessionContext
col
✓ from datafusion import col
Used for DataFrame API operations, especially column selection and expressions.
udf
✓ from datafusion import udf
For defining User-Defined Scalar Functions (UDFs).
functions
✓ from datafusion import functions
Provides access to built-in DataFusion functions like `functions.sum()`.
Demonstrates how to create an in-memory PyArrow table, register it with DataFusion's `SessionContext`, and then query it using both SQL and the DataFrame API. Results are converted to Pandas DataFrames for easy display.
from datafusion import SessionContext, col
import pyarrow as pa
# Create a DataFusion session context
ctx = SessionContext()
# Create an in-memory PyArrow table
data = {
"id": [1, 2, 3, 4],
"value": [10, 20, 15, 25],
"category": ["A", "B", "A", "C"]
}
pyarrow_table = pa.table(data)
# Register the PyArrow table as a DataFusion table
ctx.register_record_batches("my_table", [pyarrow_table.to_batches()])
# Execute a SQL query
df_sql = ctx.sql("SELECT category, SUM(value) FROM my_table GROUP BY category ORDER BY category")
print("SQL Query Result:")
print(df_sql.to_pandas())
# Execute a DataFrame API query
df_dataframe = ctx.table("my_table")
df_dataframe = df_dataframe.group_by(col("category")) \
.aggregate([col("value").sum().alias("total_value")]) \
.sort(col("category"))
print("\nDataFrame API Query Result:")
print(df_dataframe.to_pandas())
datafusion --version
Debug
Known issues
breakingBreaking changes to Foreign Function Interface (FFI) for Python extensions (e.g., custom CatalogProvider, TableProvider). Users implementing custom FFI-based providers must now provide `LogicalExtensionCodec` and `TaskContextProvider`, and method signatures have changed.fixUpdate custom FFI implementations to include `LogicalExtensionCodec` and `TaskContextProvider` and adapt to new function signatures. Refer to the DataFusion Python Extensions documentation for migration details.
affects: >= 52.0.0
gotchaDataFusion's Python bindings are tightly coupled with the core Rust DataFusion library. Downstream libraries (e.g., `deltalake`, `pyiceberg`) that provide DataFusion table providers often require exact version matches. This can lead to dependency conflicts when using multiple such libraries.fixCarefully manage dependencies and their DataFusion version requirements. Consider using `pip freeze` and `pip check` to identify conflicts. Check release notes of downstream libraries for compatible DataFusion versions.
affects: All versions
breakingThe way schemas are passed to `FileSource` constructors and `FileScanConfigBuilder` has been refactored. File sources now require the schema (including partition columns) at construction, and `FileScanConfigBuilder` no longer accepts a separate schema parameter. Additionally, `FilePruner::try_new()` signature changed.fixAdjust custom `FileSource` and `FileScanConfigBuilder` implementations to provide schemas upfront during construction. Update `FilePruner` usage as per the migration guide.
affects: >= 44.0.0
deprecatedThe `SchemaAdapterFactory` has been fully removed from Parquet scanning. This includes the `SchemaAdapter`, `SchemaMapper`, `DefaultSchemaAdapterFactory` traits/structs.fixRemove reliance on `SchemaAdapterFactory` and related components for Parquet scanning. DataFusion now handles schema adaptation differently.
affects: >= 49.0.0 (deprecated in 49.0.0, removed later)
gotchaThe default value of the `datafusion.execution.collect_statistics` configuration setting changed from `false` to `true`. This means DataFusion will now collect and store statistics by default when a table is first created via `CREATE EXTERNAL TABLE` or DataFrame `register_*` APIs.fixBe aware of potential performance implications due to statistics collection on table registration. If undesired, explicitly set `ctx.session_config().with_collect_statistics(False)` or configure via `config.set('datafusion.execution.collect_statistics', 'false')`. affects: >= 48.0.0
breakingFor advanced User-Defined Functions (UDFs), `UDF` traits now use `FieldRef` rather than `DataType` and nullability directly. `FieldRef` provides access to metadata fields, supporting extension types.fixUpdate custom UDF implementations to utilize `FieldRef` where type and nullability information is accessed.
affects: >= 48.0.0
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'datafusion'
The datafusion Python package has not been installed in the current Python environment or the environment is not active.
fixRun `pip install datafusion` in your terminal to install the package.
DataFusion error: Internal("PhysicalOptimizer rule 'join_selection' failed. Schema mismatch.")
This error often arises when joining DataFrames or converting data, particularly from Pandas, due to subtle differences in schema metadata (e.g., nullability, specific types, or internal Arrow metadata) that prevent DataFusion from successfully optimizing or executing the join plan.
fixEnsure consistent schema definitions across joined tables, explicitly cast columns to matching types if necessary, and verify that Pandas DataFrame dtypes are compatible with PyArrow/DataFusion before conversion.
DataFusion error: Plan("Schema contains duplicate qualified field name '...' ")
When performing a join operation and implicitly selecting all columns (e.g., `SELECT *`), if both joined tables contain columns with identical names, DataFusion's planner encounters ambiguity and reports duplicate qualified field names.
fixExplicitly select and alias any duplicate column names in your SQL query or DataFrame API operation to ensure unique qualified field names in the resulting schema.
DataFusion error: Error during planning: Unsupported operator in the subquery plan.
DataFusion's SQL planner does not yet support all complex SQL operators or patterns within subqueries, leading to a planning error.
fixSimplify the SQL query by breaking down complex subqueries into multiple, simpler steps, or rewrite the query using supported DataFusion DataFrame API operations if a direct SQL translation is not working.
ArrowInvalid: Schema at index 0 was different
This error can occur when converting a DataFusion DataFrame, especially one containing complex types like struct columns, to a Pandas DataFrame using `to_pandas()`, if the underlying Arrow schema of the struct fields does not align with Pandas' expectations.
fixInspect the schema of the struct columns and, if necessary, cast the fields within the struct to compatible or simpler types (e.g., string) before attempting the `to_pandas()` conversion.
Upgrade
Version history
54.0.0latest on PyPI · released Jun 29, 2026
Audit
Dependencies
pyarrowrequiredCore data format and interoperability.
pandasoptionalCommonly used for converting DataFusion results to Pandas DataFrames.
deltalakeoptionalRequired for interacting with Delta Lake tables.
pyicebergoptionalRequired for interacting with Iceberg tables.