Install & Compatibility
Where this runs
tested against v0.4.0.post1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.920 runs
installs and imports cleanly · install 0.0s · import 0.000s · 18.7MB
glibcpy 3.10–3.920 runs
installs and imports cleanly · install 1.6s · import 0.000s · 19MB
17MB installed
● package 17MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
dbldatagen
✓ import dbldatagen
✗ import dbldatagen as dg
This quickstart demonstrates how to generate a synthetic dataset using `dbldatagen`'s `Datasets` class, which provides pre-configured data generation recipes. It initializes a SparkSession (if not already present), creates a 'basic/user' dataset with 1 million rows and 4 partitions, then displays its schema and the first few rows. This approach is recommended for quickly generating common synthetic data patterns.
from pyspark.sql import SparkSession
import dbldatagen as dg
# Initialize SparkSession (if not in Databricks already)
try:
spark
except NameError:
spark = SparkSession.builder.appName("dbldatagen_quickstart").getOrCreate()
# Generate a basic user dataset using standard datasets feature
# This creates 1 million rows and 4 partitions
print("Generating a basic user dataset...")
df = dg.Datasets(spark, "basic/user").get(rows=1_000_000, partitions=4).build()
# Display schema and a few rows
print("Schema:")
df.printSchema()
print("Sample data:")
df.show(5, truncate=False)
# Stop SparkSession if it was created here (optional)
# spark.stop()
Debug
Known issues
breakingVersion 0.4.0 increased the minimum `pyspark` version to 3.2.1 and requires Databricks runtime 10.4 LTS or later. Older PySpark versions or Databricks runtimes will not be compatible.fixUpgrade your PySpark installation (`pip install 'pyspark>=3.2.1'`) and ensure your Databricks Runtime is 10.4 LTS or newer.
affects: 0.4.0+
gotchaSpark SQL column names are case-insensitive. Defining new columns with the same name but different casing than existing ones may lead to unexpected behavior or errors in downstream operations.fixEnsure consistent casing for column names throughout your data generation specifications to avoid conflicts.
affects: All
gotchaWhen using `dbldatagen.constraints.UniqueCombinations` with streaming dataframes, deduplication is performed only within a batch. For full stream-wide deduplication, you must implement explicit watermarking and deduplication logic on the resultant DataFrame, which can be resource-intensive for high-volume streams.fixFor stateful deduplication across an entire stream, apply watermarking and deduplication using Spark's native streaming APIs on the DataFrame produced by `build()`.
affects: All
gotchaThe column name 'id' is reserved internally by `dbldatagen` as the seed column for data generation. If your generated data requires a column named 'id' with different semantics, it will conflict with this internal mechanism.fixCustomize the internal seed column name by setting the `seedColumnName` attribute when creating the `DataGenerator` instance (e.g., `DataGenerator(..., seedColumnName="_internal_id")`).
affects: All
gotchaWhen running on Databricks Unity Catalog enabled environments with Runtimes prior to 13.2, `dbldatagen` requires 'Single User' or 'No Isolation Shared' access modes. 'Shared' access mode in these older runtimes lacks necessary features (e.g., 3rd party libraries, Python UDFs) for `dbldatagen` to function correctly. This limitation is resolved in Databricks Runtimes 13.2 and newer.fixUse Databricks Runtime 13.2 or later, or configure your cluster to use 'Single User' or 'No Isolation Shared' access modes if using older runtimes.
affects: <13.2 Databricks Runtimes on Unity Catalog
Upgrade
Version history
0.4.0.post1latest on PyPI · released Jul 26, 2024
Audit
Dependencies
pysparkrequiredCore dependency for Spark DataFrame operations, requires >=3.2.1 for dbldatagen v0.4.0.
jmespathoptionalMay be an implicit dependency in some environments (e.g., Google Colab) to avoid import errors related to JSON processing.