s3fs

library2026.7.0pypypi✓ verified 29d ago

S3Fs is a Pythonic filesystem interface to Amazon S3, built on top of aiobotocore and fsspec. The top-level class S3FileSystem exposes familiar file-system operations (ls, cp, mv, du, glob, put, get) and a file open() API that emulates Python's standard file protocol, making it a drop-in for libraries like pandas, dask, and gzip that accept file-like objects. It also supports S3-compatible stores (MinIO, Ceph, R2) via the endpoint_url parameter. Versions follow calendar versioning (YYYY.MM.PATCH); the current release is 2026.2.0, released February 2026, with roughly monthly cadence.

aws data http-networking

pip install s3fs

github ↗package ↗

INSTALL

IMPORT

SIG · S3FS

s3fs

awspythonv2026.7.0

Install

7.4s avg

Import

1059ms

Disk

62MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2026.7.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 1.110s · 61.7MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 7.4s · import 1.008s · 64MB

62MB installed

● package 62MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

S3FileSystem

✓ import s3fs s3 = s3fs.S3FileSystem()

✗ from s3fs.core import S3FileSystem

S3FileSystem is the only public entrypoint; importing from s3fs.core is internal and may break across releases. Always use import s3fs then s3fs.S3FileSystem().

S3File

✓ with s3fs.S3FileSystem().open('bucket/key', 'rb') as f: ...

S3File is returned by open(); never instantiate it directly. Only binary modes are supported: r, w, a, rb, wb, ab.

open via fsspec URL

✓ import fsspec with fsspec.open('s3://bucket/key', 'rb') as f: ...

fsspec.open with an s3:// URL will automatically dispatch to S3FileSystem; pass storage_options dict for credentials.

Connect with explicit credentials from environment variables, list a bucket, read a file, and write a file.

import os import s3fs # Credentials via env vars (boto chain also checks ~/.aws/credentials, IAM roles, etc.) fs = s3fs.S3FileSystem( key=os.environ.get('AWS_ACCESS_KEY_ID', ''), secret=os.environ.get('AWS_SECRET_ACCESS_KEY', ''), # token=os.environ.get('AWS_SESSION_TOKEN', ''), # uncomment for STS/assumed-role # endpoint_url='https://s3.example.com', # uncomment for MinIO / S3-compatible ) # List bucket contents bucket = os.environ.get('S3_BUCKET', 'my-bucket') print(fs.ls(bucket)) # Read a file with fs.open(f'{bucket}/hello.txt', 'rb') as f: print(f.read()) # Write a file (must flush >5 MiB for multipart; context manager handles this) with fs.open(f'{bucket}/output.txt', 'wb') as f: f.write(b'hello s3fs') # Works transparently with pandas via storage_options import pandas as pd df = pd.read_csv( f's3://{bucket}/data.csv', storage_options={ 'key': os.environ.get('AWS_ACCESS_KEY_ID', ''), 'secret': os.environ.get('AWS_SECRET_ACCESS_KEY', ''), }, ) print(df.head())

Debug

Known issues

breakingaiobotocore pins an extremely narrow botocore version range (often a single patch). Installing s3fs alongside boto3 or awscli frequently produces irresolvable dependency conflicts because boto3 requires a different botocore range.

fix

Use 'pip install s3fs[boto3]' to get a pre-validated boto3+botocore combination, or prefer conda-forge which pre-solves the triangle. Never pin boto3 and s3fs independently with ^ in Poetry without checking aiobotocore's exact botocore requirement first.

affects: all

breakingUsing multiprocessing with the default 'fork' start method causes deadlocks and hard-to-reproduce bugs because s3fs keeps open async sockets and a background thread.

fix

Set multiprocessing.set_start_method('spawn') or 'forkserver' before creating S3FileSystem instances, or avoid sharing S3FileSystem objects across fork boundaries.

affects: all

breakings3fs version 2023.12.0 was yanked from PyPI due to an authentication regression. pip may still resolve to it on some platforms if not using --pre filtering.

fix

Pin to >=2023.12.1. Run 'pip install s3fs>=2023.12.1' to avoid the yanked release.

affects: 2023.12.0

gotchaThe directory listing cache (dircache) is not invalidated automatically. If an object is written or resized externally (e.g. by boto3 or another process) after fs.ls() or fs.info() has cached its metadata, subsequent reads via the same S3FileSystem instance will use stale size information and may return corrupted or truncated data.

fix

Call fs.invalidate_cache() after external writes, or pass refresh=True to fs.info(). For high-churn workloads consider listings_expiry_time= when constructing S3FileSystem.

affects: all

gotchaFile access is always binary. Text mode ('r', 'w') is technically accepted but returns bytes or requires an explicit encoding wrapper. readline() and line iteration work but the underlying stream is always bytes.

fix

Use 'rb'/'wb' modes explicitly. For text, wrap with io.TextIOWrapper: io.TextIOWrapper(fs.open('bucket/file', 'rb'), encoding='utf-8').

affects: all

gotchaS3FileSystem instances are cached as singletons by default (skip_instance_cache=False). Two calls with the same credentials return the same object, which can cause credential or config bleed between parts of an application that expect independent connections.

fix

Pass skip_instance_cache=True when constructing S3FileSystem if you need isolated instances, e.g. with different endpoint_urls or IAM roles.

affects: all

gotchaWrites to S3 are not flushed until the file is closed (or the multipart threshold of ~150 MiB is hit). Calling f.write() without closing inside a context manager means data is buffered locally and nothing is committed to S3 on partial writes.

fix

Always use 'with fs.open(..., "wb") as f:' context managers for writes. Do not rely on explicit flush() calls for durability; only close() / __exit__ commits the upload.

affects: all

breakingS3FileSystem requires valid AWS credentials to be configured (e.g., via environment variables, IAM roles, or ~/.aws/credentials). Failure to provide these results in an 'AuthorizationHeaderMalformed' error.

fix

Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set, or that an appropriate IAM role is available to the execution environment. Alternatively, pass explicit credentials to the S3FileSystem constructor.

affects: all

breakingAn `AuthorizationHeaderMalformed` error, often stating 'a non-empty Access Key (AKID) must be provided in the credential', means that s3fs could not find or use valid AWS credentials. This typically happens when environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) are unset or empty, or explicit `key` and `secret` parameters are missing/incorrect from the `S3FileSystem` constructor.

fix

Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are correctly set, or pass `key` and `secret` parameters directly to the S3FileSystem constructor. If using IAM roles, verify the instance profile or role assumption is properly configured and providing valid temporary credentials.

affects: all

Errors

Common errors & fixes

ModuleNotFoundError: No module named 's3fs'

The 's3fs' library has not been installed in the current Python environment.

fix

pip install s3fs

fsspec.exceptions.NoS3FSImplementation: No s3fs implementation found, install s3fs to use s3://

When using `fsspec`-dependent libraries like pandas or dask to access S3 paths, `s3fs` must be installed to provide the S3 filesystem implementation.

fix

pip install s3fs

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied.

The AWS credentials configured for `s3fs` lack the necessary IAM permissions to perform the requested S3 operation on the specified bucket or object.

fix

Ensure your AWS user or role has the required IAM policies (e.g., `s3:GetObject`, `s3:ListBucket`) for the target S3 resources, and that your credentials are correctly configured (environment variables, `~/.aws/credentials`, or IAM role).

TypeError: S3FileSystem.__init__() got an unexpected keyword argument 'endpoint_url'

The `endpoint_url` parameter for S3-compatible storage should be passed within the `client_kwargs` dictionary, not as a direct argument to `S3FileSystem`.

fix

s3 = s3fs.S3FileSystem(client_kwargs={'endpoint_url': 'http://localhost:9000'})

FileNotFoundError: No such file or directory: 's3://my-bucket/non-existent-path'

The specified S3 path (bucket or object) does not exist or is inaccessible with the provided credentials.

fix

Verify the S3 path for correctness, ensure the bucket exists, and confirm that your AWS credentials have appropriate list and read permissions for the specified location.

Upgrade

Version history

2026.7.0latest on PyPI · released Jul 28, 2026

Audit

Dependencies

aiobotocorerequiredCore async AWS client used internally; pins a narrow botocore range — this is the primary source of dependency conflicts when boto3 is also installed

fsspecrequiredAbstract filesystem interface that S3FileSystem inherits from; s3fs and fsspec versions must be kept in sync

boto3optionalAvailable as the [boto3] extra; needed only if you use get_delegated_s3pars() or STS credential delegation

aiohttpoptionalTransitive dependency of aiobotocore; guards ClientPayloadError handling for incomplete-read retries

Agent activity

56 hits · last 30 days

node

OpenAI (training)

Resources

githubhttp://github.com/fsspec/s3fs ↗

packagehttps://pypi.org/project/s3fs/ ↗