Registry / workflow / flytekitplugins-spark

flytekitplugins-spark

library1.16.23pypypi✓ verified 88d ago

Spark 3 plugin for flytekit, enabling PySpark jobs within Flyte workflows. Current version 1.16.19, part of the Flytekit ecosystem, released regularly alongside Flytekit.

workflow data devops

pip install flytekitplugins-spark

package ↗homepage ↗

INSTALL

IMPORT

SIG · FLYTEKITPLUGINS-SP

flytekitplugins-spark

workflowpythonv1.16.23

Install

27.1s avg

Import

4621ms

Disk

802MB

Pass rate

8/ 10

Env Coverage8 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.16.23 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✓ 0.1s

✓ 55.8s

py 3.11

✓ 0.1s

✓ 51.5s

py 3.12

✓ 0.1s

✓ 51s

py 3.13

✕ build_error

py 3.9

✓ 0.1s

✓ 58.2s

802MB installed

● package 802MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Spark

✓ from flytekitplugins.spark import Spark

Correct import for the Spark task type.

PySparkTask

✓ from flytekitplugins.spark import PySparkTask

✗ from flytekitplugins.spark.task import PySparkTask

PySparkTask is exposed at the top level; nested import was used in older docs.

Define a Spark task using task_config=Spark(...), then build a workflow.

from flytekit import task, workflow from flytekitplugins.spark import Spark, PySparkTask @task(task_config=Spark( spark_conf={ "spark.executor.cores": "1", "spark.executor.instances": "1" } )) def my_spark_task() -> int: import pyspark spark = pyspark.sql.SparkSession.builder.getOrCreate() df = spark.range(10) return df.count() @workflow def wf() -> int: return my_spark_task() if __name__ == "__main__": print(wf())

Debug

Known issues

breakingIn flytekit>=1.10, Spark task configuration requires explicit spark_conf; default configs may be removed.

fix

Always provide spark_conf dictionary in Spark() task config.

affects: >=1.10.0

gotchaSparkSession must be obtained inside the task at runtime, not at import time. Using getOrCreate() outside the task scope can cause serialization errors.

fix

Always call SparkSession.builder.getOrCreate() inside the task function body.

affects: all

deprecatedThe old import path from 'flytekitplugins.spark.task' is deprecated in favor of top-level imports from 'flytekitplugins.spark'.

fix

Use 'from flytekitplugins.spark import Spark, PySparkTask'.

affects: >=1.0.0

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'flytekitplugins.spark'

Plugin package not installed.

fix

Run 'pip install flytekitplugins-spark'.

AttributeError: module 'flytekitplugins.spark' has no attribute 'PySparkTask'

Deprecated import path used; PySparkTask not exposed in older versions or wrong import.

fix

Use 'from flytekitplugins.spark import PySparkTask' (or upgrade flytekitplugins-spark).

Py4JJavaError: An error occurred while calling o123.showString.

Spark configuration missing required settings or cluster resources insufficient.

fix

Ensure spark_conf includes 'spark.executor.cores' and 'spark.executor.instances'. Verify cluster is running.

Upgrade

Version history

1.16.23latest on PyPI · released Jun 2, 2026

Audit

Dependencies

flytekitrequiredCore Flyte SDK required.

pysparkrequiredSpark runtime dependency.

Agent activity

19 hits · last 30 days

node

OpenAI (training)

Resources

packagehttps://pypi.org/project/flytekitplugins-spark/ ↗

homepagehttps://flyte.org ↗