Registry / workflow / luigi
library3.8.1pypypi✓ verified 27d ago

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, and much more. It's developed by Spotify and is currently in version 3.8.0, with minor releases typically occurring every few months.

pip install luigi
INSTALL
IMPORT
SIG · LUIGI
L
luigi
workflowpythonv3.8.1
Install
2.4s avg
Import
462ms
Disk
26MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.9–3.13
musl
3.9–3.13
Install & Compatibility
Where this runs
tested against v3.8.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.484s · 27.4MB
glibc
py 3.10–3.95 runs
installs and imports cleanly · install 2.4s · import 0.440s · 28MB
26MB installed
● package 26MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

Task
✓ import luigi class MyTask(luigi.Task): ...
Parameter
✓ import luigi class MyTask(luigi.Task): param = luigi.Parameter()
LocalTarget
✓ import luigi luigi.LocalTarget('path/to/file')
build
✓ import luigi luigi.build([my_task_instance], local_scheduler=True)

This quickstart defines a simple Luigi task `GenerateReport` that takes a `date` parameter, simulates generating a report, and writes it to a local file. It demonstrates defining a task, its `run` method, and its `output` target using `luigi.LocalTarget`. The example also shows how to trigger tasks programmatically using `luigi.build` with a local scheduler.

import luigi import datetime import os class GenerateReport(luigi.Task): date = luigi.DateParameter(default=datetime.date.today()) def run(self): # Simulate some data processing report_content = f"Daily Report for {self.date.isoformat()}\n" \ f"Generated by Luigi.\n" # Write the report to a target file with self.output().open('w') as f: f.write(report_content) def output(self): # Define where the output of this task will be stored # Using an environment variable for flexibility or defaulting to current dir output_dir = os.environ.get('LUIGI_OUTPUT_DIR', '.') return luigi.LocalTarget(os.path.join(output_dir, f'report_{self.date.isoformat()}.txt')) if __name__ == '__main__': # To run this task using the command line (most common): # 1. Start the Luigi scheduler daemon in a separate terminal: luigid --port 8082 # 2. Run your script: python your_script_name.py GenerateReport --date 2023-10-26 # (or omit --date for today's date if default is set) # 3. If you don't want to run luigid, use the --local-scheduler flag: # python your_script_name.py GenerateReport --local-scheduler # For programmatic execution (e.g., in a wrapper script or test): # The 'local_scheduler=True' ensures it runs without an external luigid daemon. luigi.build([GenerateReport(date=datetime.date(2023, 10, 26))], local_scheduler=True)
luigi --version
Debug
Known issues
breakingLuigi 3.x series requires Python >= 3.10 and < 3.14. Projects on older Python 3 versions (e.g., 3.6-3.9) or newer versions (3.14+) will not be compatible.
fix
Upgrade your Python environment to a compatible version (3.10, 3.11, 3.12, 3.13) before upgrading Luigi to 3.x.
affects: 3.x.x
gotchaLuigi tasks require a scheduler to run. By default, it looks for a running `luigid` daemon. For simple local execution or development, remember to use the `--local-scheduler` flag when running tasks from the command line, or `local_scheduler=True` when using `luigi.build()` programmatically.
fix
Start `luigid` in a separate terminal (`luigid --port 8082`) or always include `--local-scheduler` in your command line execution or `local_scheduler=True` in `luigi.build()` calls.
affects: All versions
deprecatedThe `pkg_resources` library, which was historically used for introspection and entry points, has been removed in Luigi 3.7.3. While this is primarily an internal change, it might affect custom plugins or integrations that directly or indirectly relied on Luigi's usage of `pkg_resources`.
fix
Review custom plugins or extensions for any direct calls or implicit dependencies on `pkg_resources` and migrate to `importlib.metadata` (for Python 3.8+) or alternative methods.
affects: >=3.7.3
gotchaLuigi determines task completion solely based on whether the `output()` targets `exist()`. If a task's `run()` method completes but fails to create its defined `output()`, Luigi will consider it incomplete and re-run it in subsequent invocations. Similarly, manually deleting an output file will cause Luigi to re-run the producing task.
fix
Ensure that every `run()` method reliably creates the target specified by its `output()` method. For tasks that don't produce physical outputs, consider using a dummy `LocalTarget('/tmp/luigi_flag_file')` or `luigi.MockTarget` for testing, or ensure their dependencies cover the actual work.
affects: All versions
Errors
Common errors & fixes
SyntaxError: ('invalid syntax', ('/.pyenv/versions/2.7.14/lib/python2.7/site-packages/luigi/task.py', 145, 21, 'class Task(metaclass=Register):\n'))
This error occurs when attempting to run Luigi, which requires Python 3.x, on Python 2.7.
fix
Upgrade your Python environment to version 3.6 or higher to ensure compatibility with Luigi.
ImportError: No module named us
This error indicates that the 'us' module is not found in the Python environment, possibly due to it being installed in a different environment or not installed at all.
fix
Ensure that the 'us' module is installed in the correct Python environment by running 'pip install us' in the same environment where Luigi is executed.
ImportError: No module named tasks
This error occurs when Luigi cannot locate the 'tasks' module, often due to the module not being in the Python path.
fix
Set the PYTHONPATH environment variable to include the directory containing 'tasks.py', or run Luigi from the directory where 'tasks.py' is located.
ModuleNotFoundError: No module named 'pwd'
The 'pwd' module is specific to Unix-like systems and is not available on Windows, leading to this error when running Luigi on Windows.
fix
Avoid using the '--background' option when running 'luigid' on Windows, as it relies on the 'pwd' module.
RuntimeError: Unfulfilled dependency at run time: Generate_TV_WebScraping_File_X__DataMining_Lu_8213e479cf
This error indicates that a required dependency task has not been completed successfully before the current task is executed.
fix
Ensure that all prerequisite tasks are correctly defined and executed before running dependent tasks in Luigi.
Upgrade
Version history
3.8.1latest on PyPI · released May 7, 2026
Audit
Dependencies

No dependency data recorded yet.

Agent activity
18 hits · last 30 days
node
14
OpenAI (training)
1
Resources
luigi — pip install luigi · libregistry