Registry / http-networking / ultimate-sitemap-parser

ultimate-sitemap-parser

library1.8.0pypypi✓ verified 89d ago

Ultimate Sitemap Parser (USP) is a performant and robust Python library for parsing and crawling sitemaps. It supports all major sitemap formats (XML, Google News, plain text, RSS/Atom), handles nested sitemaps, is error-tolerant, and efficiently processes large hierarchies using a lazy-loading generator for pages. The library is actively maintained, with frequent releases; the current version is 1.8.0.

http-networking data

pip install ultimate-sitemap-parser

github ↗package ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · ULTIMATE-SITEMAP-P

ultimate-sitemap-parser

http-networkingpythonv1.8.0

Install

2.3s avg

Import

671ms

Disk

21MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.8.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 0.708s · 22.4MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 2.3s · import 0.634s · 23MB

21MB installed

● package 21MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

sitemap_tree_for_homepage

✓ from usp.tree import sitemap_tree_for_homepage

✗ from ultimate_sitemap_parser.tree import sitemap_tree_for_homepage

The top-level package for import is `usp`, not `ultimate_sitemap_parser`.

sitemap_from_str

✓ from usp.tree import sitemap_from_str

For parsing sitemaps from local string content.

This quickstart fetches and parses the sitemap for a given homepage URL, automatically discovering and traversing any linked sitemap index files. It then iterates through all unique pages found across the entire sitemap hierarchy, printing each page's URL. The `all_pages()` method uses a generator for memory-efficient processing of large sitemaps.

from usp.tree import sitemap_tree_for_homepage # Replace with the target website URL for sitemap discovery target_url = "https://www.example.org/" try: # Fetches sitemaps, discovers nested sitemaps, and builds a tree structure tree = sitemap_tree_for_homepage(target_url) print(f"Successfully parsed sitemap for: {target_url}") print("Listing all discovered pages:") # Iterate through all pages found across the sitemap hierarchy # Uses a generator for memory efficiency with large sitemaps page_count = 0 for page in tree.all_pages(): print(page.url) page_count += 1 print(f"Found {page_count} pages.") except Exception as e: print(f"An error occurred: {e}")

Debug

Known issues

breakingPython 3.8 is no longer supported as of version 1.3.0. The minimum required Python version for recent releases (including 1.8.0) is >=3.10.

fix

Ensure your environment uses Python 3.10 or newer before installing or upgrading `ultimate-sitemap-parser`.

affects: >=1.3.0

breakingIf you use custom web clients by subclassing `AbstractWebClient`, you must implement the new `url()` method as of version 1.3.0. This method should return the actual URL fetched after any redirects.

fix

Update your custom `AbstractWebClient` implementation to include the `url()` method: `def url(self) -> str: ...`.

affects: >=1.3.0

gotchaMalformed sitemap XML (e.g., incorrect namespace, missing tags, invalid URLs, improper encoding) can lead to `InvalidSitemap` objects or parsing failures, even though the library is error-tolerant.

fix

Always validate sitemap files for correct XML syntax, proper `http://www.sitemaps.org/schemas/sitemap/0.9` namespace, and valid UTF-8 encoding. The library will return an `InvalidSitemap` object if it cannot be parsed.

affects: All versions

gotchaWhile designed for efficiency, processing extremely large sitemaps (e.g., >50MB uncompressed or >50,000 URLs) can still be resource-intensive or hit memory limits.

fix

The `all_pages()` method uses a generator to lazily load pages, which helps with memory. For optimal performance with massive sites, ensure ample system resources or consider processing sitemap subsets if possible.

affects: All versions

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'ultimate_sitemap_parser'

The top-level package for importing is `usp`, not `ultimate_sitemap_parser`.

fix

Change your import statements from `import ultimate_sitemap_parser` or `from ultimate_sitemap_parser.tree import ...` to `import usp` or `from usp.tree import ...` respectively.

AttributeError: 'InvalidSitemap' object has no attribute 'all_pages'

The `sitemap_tree_for_homepage` function or other parsing methods returned an `InvalidSitemap` object, indicating that the sitemap could not be fetched or parsed successfully.

fix

Check the URL for correctness and accessibility. The `InvalidSitemap` object itself might contain information about the failure (e.g., HTTP status code or parsing error). Inspect the logs for details during the parsing process.

SitemapException: Maximum recursion depth exceeded (URL: ...)

The library detected a circular reference within the sitemap hierarchy (a sitemap linking back to itself or an ancestor) or an excessively deep, potentially infinite, recursion.

fix

This usually indicates a malformed sitemap structure on the target website. Review the sitemap files for unintended circular dependencies. The library prevents infinite loops, but this exception signals a problematic sitemap design.

Upgrade

Version history

1.8.0latest on PyPI · released Jan 25, 2026

Audit

Dependencies

python-dateutilrequiredUsed for date parsing and manipulation within sitemap entries.

requestsrequiredUsed as the default HTTP client for fetching sitemap content from URLs.

Agent activity

34 hits · last 30 days

node

Amazon

OpenAI (training)

Resources

githubhttps://github.com/GateNLP/ultimate-sitemap-parser ↗

packagehttps://pypi.org/project/ultimate-sitemap-parser/ ↗

homepagehttps://ultimate-sitemap-parser.readthedocs.io/ ↗

docshttps://ultimate-sitemap-parser.readthedocs.io/ ↗