Install & Compatibility
Where this runs
tested against v1.6.5 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.920 runs
installs and imports cleanly · install 0.0s · import 1.182s · 167.7MB
glibcpy 3.10–3.920 runs
installs and imports cleanly · install 8.8s · import 1.087s · 160MB
167MB installed
● package 167MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
pandarallel
✓ from pandarallel import pandarallel
This quickstart demonstrates how to initialize pandarallel and then use `parallel_apply` on a Pandas Series. It includes a simple, CPU-bound function to showcase the parallelization effect. The `nb_workers` is explicitly set to the CPU count for clarity, and a progress bar is enabled.
import pandas as pd
from pandarallel import pandarallel
import os
# Initialize pandarallel. It defaults to using all available CPU cores.
# progress_bar=True is often useful to visualize progress.
pandarallel.initialize(nb_workers=os.cpu_count(), progress_bar=True)
# Create a sample DataFrame with some data
data = {'col1': range(1_000_000), 'col2': [f'item_{i}' for i in range(1_000_000)]}
df = pd.DataFrame(data)
# Define a CPU-bound function to apply
def example_computation(x):
# Simulate a computationally intensive task
res = 0
for i in range(50):
res += (x * i) ** 0.5
return res
# Apply the function in parallel using pandarallel's parallel_apply
# This replaces df['col1'].apply(example_computation)
print("Starting parallel computation...")
df['result'] = df['col1'].parallel_apply(example_computation)
print("Computation complete. First 5 rows of the DataFrame with results:")
print(df.head())
Debug
Known issues
gotchaPandarallel can require up to twice the memory of standard Pandas operations. Ensure your system has sufficient RAM, especially for large datasets.fixMonitor memory usage. For data larger than available memory, consider alternatives like Dask or PySpark.
affects: All versions
gotchaOn Windows, functions passed to `pandarallel` must be self-contained and should not depend on external resources (e.g., global variables, complex closures) due to Python's `multiprocessing` 'spawn' start method.fixDefine functions at the top level of the module. For complex scenarios, consider using Windows Subsystem for Linux (WSL) or refactor functions to be entirely self-contained.
affects: All versions on Windows
gotchaParallelization introduces overhead. For small datasets or very fast operations, `pandarallel` might not provide a speedup, or could even be slower than native Pandas.fixBenchmark performance with and without `pandarallel` for your specific use case to determine if parallelization is beneficial.
affects: All versions
gotchaPandarallel scales best with the number of *physical* CPU cores, not necessarily logical cores (hyperthreading). Setting `nb_workers` higher than physical cores may not yield further performance gains.fixFor optimal performance, set `nb_workers` to your system's number of physical CPU cores or allow `pandarallel` to determine it automatically.
affects: All versions
gotchaThe `shm_size_mb` parameter in `pandarallel.initialize()` is deprecated and should no longer be used.fixRemove `shm_size_mb` from `pandarallel.initialize()` calls. Memory file system usage is now controlled by `use_memory_fs`.
affects: >=1.x.x
gotchaPandarallel can sometimes get stuck without raising errors if all physical cores are heavily utilized by other background processes. The progress bar might stop updating.fixMonitor system CPU usage. Try reducing `nb_workers` to leave some cores free, or ensure your environment has sufficient idle CPU resources.
affects: All versions
gotchaFunctions defined locally (e.g., inside another function) or using closures may lead to `AttributeError: Can't pickle local object` errors, a common issue with Python's multiprocessing.fixEnsure functions passed to `pandarallel` methods are defined at the top level of a module, not nested within other functions.
affects: All versions
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'pandarallel'
The 'pandarallel' library has not been installed in the current Python environment.
fixpip install pandarallel
AttributeError: Can't pickle local object 'prepare_worker..closure..wrapper'
Functions passed to `pandarallel` methods (which use Python's multiprocessing) must be picklable. This error occurs when a function is defined locally (e.g., inside another function or a method of a class) and thus cannot be serialized for distribution to worker processes. This is especially common on Windows due to its default 'spawn' multiprocessing start method.
fixDefine the function at the top level of the module (globally) or ensure it is a static method if part of a class, so it can be properly pickled and accessed by worker processes.
AttributeError: 'DataFrameGroupBy' object has no attribute 'parallel_apply'
This usually means `pandarallel` has not been initialized with `pandarallel.initialize()` before attempting to use its parallelized methods, or the method is called on an unsupported Pandas object/operation.
fixEnsure `pandarallel.initialize()` is called once at the beginning of your script. For `groupby().apply()`, use `df.groupby(...).parallel_apply(func)`. Note that `pandarallel` only supports specific parallelized Pandas APIs.
pandarallel does not work at all. On Windows, because of the multiprocessing system (spawn), the function you send to pandarallel must be self contained.
On Windows, due to the `multiprocessing` 'spawn' start method, functions passed to `pandarallel` must be self-contained and cannot depend on external resources (like global variables or complex closures) defined in the main script.
fixRefactor the function to be entirely self-contained, ensuring it does not rely on global variables or objects defined outside its scope. Define helper functions at the top level of the module. For complex scenarios, consider using Windows Subsystem for Linux (WSL).
AttributeError: 'DataFrame' object has no attribute 'parallel_apply'
The `pandarallel` library was imported but its `initialize()` method was not called, preventing it from patching Pandas DataFrames and Series with parallel methods.
fixCall `pandarallel.initialize()` after importing `pandarallel` to enable the parallel methods:
```python
import pandas as pd
from pandarallel import pandarallel
pandarallel.initialize()
df = pd.DataFrame({'a': range(100)})
df['b'] = df.parallel_apply(lambda x: x['a'] * 2, axis=1)
``` Upgrade
Version history
1.6.5latest on PyPI · released May 2, 2023
Audit
Dependencies
pandasrequiredPandarallel is built on top of Pandas and parallelizes its operations.
numpyrequiredFrequently used in conjunction with Pandas for numerical operations.