Install & Compatibility
Where this runs
tested against v1.7.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.940 runs
installs and imports cleanly · install 0.0s · import 0.543s · 505.9MB
glibcpy 3.10–3.940 runs
installs and imports cleanly · install 16.2s · import 0.476s · 506MB
501MB installed
● package 501MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
Column
✓ from typedspark import Column
✗ from typedspark import Column
This quickstart demonstrates how to define a schema using `typedspark.Schema`, create a `DataSet` from a PySpark DataFrame, and apply transformations with type annotations. It also shows how to create an empty `DataSet` for testing purposes.
import pyspark.sql.functions as F
from pyspark.sql import SparkSession
from pyspark.sql.types import LongType, StringType
from typedspark import Column, DataSet, Schema
# Initialize Spark Session (if not already present)
spark = SparkSession.builder.appName("TypedSparkQuickstart").getOrCreate()
class Person(Schema):
id: Column[LongType]
name: Column[StringType]
age: Column[LongType]
def process_person_data(df: DataSet[Person]) -> DataSet[Person]:
# Example transformation: add 1 to age
return df.withColumn(Person.age, F.col(Person.age) + 1)
# Create a dummy DataFrame conforming to the Person schema
data = [
(1, "Alice", 30),
(2, "Bob", 24),
(3, "Charlie", 35)
]
schema_spark = Person.get_structtype()
df_untyped = spark.createDataFrame(data, schema=schema_spark)
# Convert to a TypedSpark DataSet
df_typed = DataSet[Person](df_untyped)
# Process the data using the typed function
df_processed = process_person_data(df_typed)
# Show results
df_processed.show()
# You can also generate an empty DataSet for testing
empty_person_dataset = Person.create_empty_dataset(spark)
empty_person_dataset.show()
spark.stop()
Debug
Known issues
gotchaTypedspark's compatibility is tested with specific PySpark versions (e.g., 3.5.7 and 4.1.0 for v1.6.3). Using untested or significantly different PySpark versions may lead to unexpected behavior or incompatibilities.fixRefer to the `typedspark` documentation or GitHub README for the officially supported PySpark versions for your `typedspark` release. Pin your `pyspark` dependency accordingly.
affects: <1.6.3 (for newer PySpark versions)
breakingPrior to version 1.6.2, `Column` comparison in multi-threaded environments could lead to issues. This was fixed by explicitly using `SparkSession.active()` for thread-safe operations.fixUpgrade to `typedspark` version 1.6.2 or newer to ensure correct `Column` comparison behavior in threaded contexts. If upgrading is not possible, ensure Spark operations are not performed in parallel threads where `typedspark.Column` instances are compared.
affects: <1.6.2
gotchaWhile `typedspark` provides compile-time type-checking, runtime schema mismatches can still occur if the underlying PySpark DataFrame's schema changes unexpectedly after a `DataSet` is created (e.g., due to an external data source modification or an untyped transformation).fixImplement runtime validation mechanisms if strict schema enforcement is required at various pipeline stages. `typedspark` itself provides methods for validation, but these need to be explicitly invoked in your data pipeline after untyped operations or external data reads.
affects: All
Upgrade
Version history
1.7.0latest on PyPI · released Apr 28, 2026
Audit
Dependencies
pysparkoptionalCore dependency for PySpark DataFrame functionality; optional during installation for environments with pre-installed PySpark (e.g., Databricks, EMR).