iopath

library0.1.10pypypi✓ verified 29d ago

iopath is a Python library from Facebook Research that provides an abstraction layer for I/O operations, allowing a unified API to interact with local filesystems and various remote storage services like S3 and HDFS. It simplifies reading from and writing to different storage backends. The current version is 0.1.10, and releases are infrequent, often tied to major projects from Facebook Research.

data http-networking aws

pip install iopath

github ↗package ↗

INSTALL

IMPORT

SIG · IOPATH

iopath

datapythonv0.1.10

Install

2.7s avg

Import

217ms

Disk

19MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.10 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 0.223s · 21.1MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 2.7s · import 0.212s · 22MB

19MB installed

● package 19MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

PathManager

✓ from iopath.common.file_io import PathManager

✗ from iopath import PathManager

PathManager is not directly exposed at the top-level `iopath` package, requiring import from its specific submodule.

S3PathHandler

✓ from iopath.common.s3 import S3PathHandler

Used for explicitly registering S3 support, though `PathManager` often registers default handlers upon first S3 access.

This quickstart demonstrates how to use `PathManager` for basic file operations (write, read, check existence, remove) on a local filesystem. `PathManager` can transparently handle different schemes (e.g., `s3://`, `hdfs://`) once appropriate handlers are registered or if the optional dependencies are installed for common schemes like S3.

import os from iopath.common.file_io import PathManager # Initialize PathManager pathmgr = PathManager() # Define a local test file path test_file_name = "iopath_test_file.txt" # Ensure clean slate if previous run failed if pathmgr.exists(test_file_name): pathmgr.rm(test_file_name) print(f"Checking if '{test_file_name}' exists: {pathmgr.exists(test_file_name)}") # Write content to the file using PathManager content_to_write = "Hello from iopath! This is an I/O abstraction library." with pathmgr.open(test_file_name, "w") as f: f.write(content_to_write) print(f"After writing, does '{test_file_name}' exist? {pathmgr.exists(test_file_name)}") # Read content from the file using PathManager read_content = "" with pathmgr.open(test_file_name, "r") as f: read_content = f.read() print(f"Content read from '{test_file_name}':\n'{read_content}'") # Clean up the test file pathmgr.rm(test_file_name) print(f"After removal, does '{test_file_name}' exist? {pathmgr.exists(test_file_name)}")

Debug

Known issues

gotchaWhen using `pathmgr.get_local_path(uri)`, if the URI points to a remote file, `iopath` will download it to a temporary local path. This operation can be slow for large files and consumes local disk space, potentially leading to unexpected performance bottlenecks or out-of-disk issues if not managed carefully.

fix

Be mindful of calling `get_local_path` only when necessary. For direct streaming, prefer `pathmgr.open(uri, 'rb')`. Consider using `tempfile.TemporaryDirectory` for managing local copies if explicit caching is desired.

affects: All versions

gotchaWhile `iopath` provides a unified API, support for remote file systems like S3 or HDFS requires installing optional dependencies (e.g., `iopath[s3]`) and ensuring the respective `PathHandler` is registered. Although `PathManager` often auto-registers common handlers, explicit registration (e.g., `pathmgr.register_handler(S3PathHandler())`) can prevent unexpected 'scheme not supported' errors.

fix

Install `iopath` with the necessary extras (e.g., `pip install iopath[s3]`). If issues persist, explicitly import and register the handler for your specific remote storage scheme: `from iopath.common.s3 import S3PathHandler; pathmgr.register_handler(S3PathHandler())`.

affects: All versions

gotchaThe core `PathManager` class is not directly exposed at the top-level `iopath` package. A common mistake is attempting `from iopath import PathManager` which will fail. It must be imported from `iopath.common.file_io`.

fix

Always use `from iopath.common.file_io import PathManager` for importing the main PathManager class.

affects: All versions

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'iopath'

The 'iopath' library is not installed in the current Python environment or is not accessible.

fix

Install the library using pip: `pip install iopath` or, if using Anaconda, `conda install -c iopath iopath`.

ModuleNotFoundError: No module named 'iopath.tabular'

The 'tabular' submodule is either missing in the installed version of 'iopath', has been moved, or the installation is incomplete. This can occur if a dependency requires a specific or older version of iopath.

fix

First, try updating 'iopath': `pip install -U iopath`. If the issue persists, consider installing directly from the GitHub repository for the latest development version: `pip install -U 'git+https://github.com/facebookresearch/iopath'`. If a dependent library explicitly requires a different version, install that specific version, e.g., `pip install iopath==0.1.7`.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. detectron2 ... requires iopath<0.1.10,>=0.1.7, but you have iopath 0.1.10 which is incompatible.

A direct or transitive dependency of your project requires a version of 'iopath' that conflicts with the currently installed version (0.1.10 in this example).

fix

Identify the package causing the conflict and either downgrade 'iopath' to a compatible version (e.g., `pip install iopath<0.1.10,>=0.1.7`) or upgrade the conflicting package if a newer version supports 'iopath 0.1.10'. It's often best to install the primary demanding package first, letting it resolve its dependencies.

FileNotFoundError: [Errno 2] No such file or directory: 's3://your_bucket/your_file.txt'

iopath is unable to locate or access the specified file on a remote storage backend (like S3 or HDFS). This can be due to an incorrect path, the file not existing, or missing/improperly configured credentials/permissions for the remote service.

fix

Verify the exact file path on the remote storage. Ensure that necessary backend-specific client libraries (e.g., `boto3` for S3) are installed and properly configured with valid access credentials (e.g., AWS environment variables, `~/.aws/credentials`, or IAM roles for S3).

Upgrade

Version history

0.1.10latest on PyPI · released Jul 9, 2022

Audit

Dependencies

boto3optionalRequired for S3PathHandler to interact with Amazon S3. Installed via 'pip install iopath[s3]'.

hdfsoptionalRequired for HdfsPathHandler to interact with HDFS. Installed via 'pip install iopath[hdfs]'.

Agent activity

22 hits · last 30 days

node

Resources

githubhttps://github.com/facebookresearch/iopath ↗

packagehttps://pypi.org/project/iopath/ ↗