Install & Compatibility
Where this runs
tested against v1.8.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.910 runs
installs and imports cleanly · install 0.0s · import 0.230s · 63.9MB
glibcpy 3.10–3.910 runs
installs and imports cleanly · install 4.3s · import 0.216s · 66MB
64MB installed
● package 64MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
ArchiveIterator
✓ from warcio.archiveiterator import ArchiveIterator
capture_http
✓ from warcio.capture_http import capture_http
WARCWriter
✓ from warcio.warcwriter import WARCWriter
WARCRecord
✓ from warcio.warcwriter import WARCRecord
✗ from warcio.record import WARCRecord
WARCRecord for manual creation is part of warcwriter since v1.6.
This quickstart demonstrates both writing and reading WARC files. The writing section uses `warcio.capture_http` to automatically capture HTTP traffic from a `requests` call into a WARC file. The reading section then iterates through the created WARC file using `warcio.archiveiterator.ArchiveIterator`, printing details of each record. The example includes commented-out code for creating the WARC file and for reading from a remote S3 URL, highlighting the flexibility of the library.
import requests
import os
from warcio.capture_http import capture_http
from warcio.archiveiterator import ArchiveIterator
# --- Writing a WARC file by capturing HTTP traffic ---
output_warc_file = 'example.warc.gz'
# Ensure requests is imported AFTER capture_http if monkey-patching
# with capture_http(output_warc_file, warc_version='1.1') as writer:
# # You can optionally set WARC-IP-Address for records if available
# os.environ['WARC_IP_ADDRESS'] = '192.168.1.1' # Example
# resp = requests.get('http://httpbin.org/get?q=test')
# print(f"Captured GET request to {resp.url} with status {resp.status_code}")
# del os.environ['WARC_IP_ADDRESS'] # Clean up env var
#
# print(f"WARC file '{output_warc_file}' created successfully.")
# --- Reading records from the WARC file (or a remote one) ---
# For remote files (e.g., S3), ensure warcio[s3] is installed
# remote_warc_url = 's3://commoncrawl/crawl-data/CC-MAIN-2023-50/segments/1701389650426.47/warc/CC-MAIN-20231130201438-00000-ip-10-2-12-106.warc.gz'
# If using a local file, ensure it exists from the writing step or provide your own
input_warc_source = output_warc_file # or remote_warc_url
if os.path.exists(output_warc_file):
print(f"\n--- Reading records from '{input_warc_source}' ---")
try:
with open(input_warc_source, 'rb') as stream:
for record in ArchiveIterator(stream):
if record.rec_type == 'response':
uri = record.rec_headers.get_header('WARC-Target-URI')
status = record.http_headers.get_statuscode() if record.http_headers else 'N/A'
print(f" Response Record: URI={uri}, Status={status}")
elif record.rec_type == 'request':
uri = record.rec_headers.get_header('WARC-Target-URI')
print(f" Request Record: URI={uri}")
elif record.rec_type == 'warcinfo':
filename = record.rec_headers.get_header('WARC-Filename')
print(f" Warcinfo Record: Filename={filename}")
except FileNotFoundError:
print(f"Error: Local WARC file '{output_warc_file}' not found. Skipping read example.")
except Exception as e:
print(f"An error occurred while reading the WARC file: {e}")
else:
print(f"Local WARC file '{output_warc_file}' not found. Skipping read example. Uncomment the writing section to create it.")
warcio --version
Debug
Known issues
gotchaTo utilize remote file system capabilities (e.g., reading/writing to S3 or HTTP/HTTPS URLs), you must explicitly install optional dependencies like `fsspec` and `s3fs`. Use `pip install warcio[s3]` or `pip install warcio[all]`.fixInstall `warcio` with the appropriate extras: `pip install warcio[s3]` for S3, or `pip install warcio[all]` for all optional features.
affects: >=1.8.0
deprecatedOlder versions of `warcio` (prior to 1.7.5) might have used `pkg_resources` for version checks, which is deprecated. While `warcio` itself has updated to `importlib` for this, users might still encounter `DeprecationWarning` messages depending on their `setuptools` or `pip` versions, or if other dependencies still use `pkg_resources`.fixEnsure your `setuptools` and `pip` are updated to their latest versions to minimize `pkg_resources` warnings. `warcio` v1.7.5 migrated to `importlib` for version retrieval.
affects: <1.7.5 (and potentially later due to transitive dependencies or environment setup)
breakingThe `setup.py test` command was removed as `setuptools` version 72 deprecated this functionality. Projects that relied on `python setup.py test` for running `warcio`'s tests or their own tests against it will break.fixDirectly use `pytest` or other standard test runners instead of `python setup.py test`.
affects: >=1.7.5
gotchaThe function `open_or_default` was re-added as an alias for `fsspec_open` in `v1.8.1`. This implies that `open_or_default` might have been removed or renamed in `v1.8.0`, potentially causing `AttributeError` or `NameError` for users upgrading from older `1.x` versions to `1.8.0` before `1.8.1` was released, if they were using this specific function.fixUpgrade to `warcio` v1.8.1 or later to ensure `open_or_default` is available as an alias for `fsspec_open`. If on `1.8.0`, use `fsspec_open` directly.
affects: Potentially 1.8.0 (fixed in 1.8.1)
gotchaFor very large-scale web crawls (tera- or petabyte scale), `warcio` (being pure Python) might be less performant than C++/Cython alternatives like `FastWARC`. `FastWARC` offers speedups but is not a drop-in replacement and lacks ARC file support.fixEvaluate performance needs for large datasets. For maximum speed, consider `FastWARC` while being aware of its API differences and lack of ARC support. Otherwise, `warcio` remains a robust option for general use.
affects: All versions
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'warcio'
The 'warcio' library has not been installed in the current Python environment.
ImportError: cannot import name 'WARCWriter' from 'warcio'
The 'WARCWriter' class is located in the 'warcio.warcwriter' submodule, not directly under the top-level 'warcio' package.
fixfrom warcio.warcwriter import WARCWriter
AttributeError: 'bytes' object has no attribute 'write'
The 'WARCWriter' was initialized with a 'bytes' object instead of a file-like object opened in binary write mode.
fixInitialize 'WARCWriter' with a file-like object, such as one created using `open('filename.warc', 'wb')`. TypeError: expected path or file-like object, got <class 'bytes'>
The 'ArchiveIterator' was initialized with a raw 'bytes' object, but it expects a file path string or a file-like object (e.g., from `open()` or `io.BytesIO`) to read from.
fixInitialize 'ArchiveIterator' with a file-like object opened in binary read mode (`open('filename.warc', 'rb')`) or a valid path string to the WARC file, or wrap bytes in `io.BytesIO`. Upgrade
Version history
1.8.1latest on PyPI · released Mar 31, 2026
Audit
Dependencies
sixrequiredMinimal external dependency for Python 3.7+.
fsspecoptionalRequired for remote file system access (e.g., HTTP, S3, GCS). Installed automatically with 'warcio[all]' or 'warcio[s3]'.
s3fsoptionalSpecifically for Amazon S3 remote file system support. Installed with 'warcio[s3]'.