findspark

library2.0.1pypypi✓ verified 29d ago

Findspark is a Python library that simplifies the process of making Apache PySpark importable in standard Python environments, like Jupyter notebooks or IDEs. It automatically locates a Spark installation on the system (using `SPARK_HOME` or common paths) and adds the necessary PySpark and Py4J directories to `sys.path`. The current version, 2.0.1, was released on February 11, 2022, and focuses on stability and bug fixes. It's maintained as a utility for PySpark users.

data devops

pip install findspark

github ↗package ↗

INSTALL

IMPORT

SIG · FINDSPARK

findspark

datapythonv2.0.1

Install

1.6s avg

Import

—

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.0.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 17.8MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.6s · import 0.000s · 18MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

init

✓ from findspark import init

✗ import findspark

This quickstart demonstrates how to initialize findspark and then import PySpark to create a SparkSession. It's crucial that `findspark.init()` is called before `import pyspark`.

import os import findspark # Optional: Set SPARK_HOME if it's not already an environment variable # findspark will try to auto-detect if not set. # Example: os.environ['SPARK_HOME'] = '/opt/spark' findspark.init() try: import pyspark from pyspark.sql import SparkSession print("PySpark is now importable.") spark = SparkSession.builder.appName("FindsparkTest").master("local[*]").getOrCreate() print(f"SparkContext version: {spark.sparkContext.version}") print("SparkSession created successfully.") spark.stop() except ImportError as e: print(f"Error importing PySpark: {e}") print("Please ensure Spark is installed and SPARK_HOME is correctly configured.") except Exception as e: print(f"An unexpected error occurred: {e}")

Debug

Known issues

gotcha`findspark.init()` must be called *before* any `import pyspark` statement. If `pyspark` is imported first, `findspark.init()` will do nothing (since `findspark` 2.0.0) or fail to configure `sys.path` correctly, leading to `ModuleNotFoundError` or similar issues when trying to use Spark functionalities.

fix

Always place `import findspark; findspark.init()` at the very beginning of your script or notebook, prior to any `import pyspark` statements.

affects: >=2.0.0

gotchaIncorrect `SPARK_HOME` or incomplete Spark installation can cause errors. `findspark` relies on the `SPARK_HOME` environment variable or specific default paths. If Spark is not found or `SPARK_HOME` points to an installation missing crucial components (e.g., `py4j` JARs), `findspark.init()` may raise a `ValueError` or `IndexError`.

fix

Ensure `SPARK_HOME` is correctly set and points to a valid Spark installation. Verify the presence of `python/lib/py4j-*.zip` and `python/pyspark.zip` within your `SPARK_HOME` directory. You can manually specify the path using `findspark.init('/path/to/spark_home')`.

affects: All

gotchaChanging the `SPARK_HOME` environment variable or specifying a new `spark_home` path in `findspark.init()` within a long-running Python session (e.g., Jupyter notebook) typically requires restarting the Python kernel for the changes to take effect, especially if `pyspark` has already been loaded.

fix

Always restart your Python kernel or shell session after modifying `SPARK_HOME` or calling `findspark.init()` with a new `spark_home` if you intend to switch Spark versions or installations.

affects: All

deprecatedThe `edit_rc=True` and `edit_profile=True` arguments in `findspark.init()` modify shell configuration files (`~/.bashrc`) or IPython profiles. While intended for convenience, these methods can lead to unintended side effects, make environments less reproducible, or cause issues if the Spark installation moves. Since `findspark` 2.0.0, the related internal methods were made private.

fix

Prefer setting `SPARK_HOME` as an environment variable explicitly in your environment setup (e.g., `~/.bashrc`, `~/.profile`, virtual environment activation scripts) rather than relying on `findspark` to persist changes. Manually manage your shell or IPython profiles if persistence is required.

affects: >=2.0.0 (internal methods deprecated), All (usage gotcha)

Upgrade

Version history

2.0.1latest on PyPI · released Feb 11, 2022

Audit

Dependencies

No dependency data recorded yet.

Agent activity

22 hits · last 30 days

node

Resources

githubhttps://github.com/minrk/findspark ↗

packagehttps://pypi.org/project/findspark/ ↗