newspaper4k

library0.9.5pypypiunverified

Newspaper4k is an open-source Python library for simplified article discovery and extraction from news websites. It is an actively maintained fork of the 'newspaper3k' project, offering new features, bug fixes, and improved parsing performance. The current version is 0.9.5, with frequent updates to enhance language support, address compatibility issues, and improve article content extraction.

data serialization http-networking

pip install newspaper4k

package ↗

INSTALL

IMPORT

SIG · NEWSPAPER4K

newspaper4k

datapythonv0.9.5

Install

13.8s avg

Import

3500ms

Disk

134MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.9.5 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.960 runs

installs and imports cleanly · install 0.0s · import 2.846s · 65.2MB

glibc

py 3.10–3.960 runs

installs and imports cleanly · install 13.8s · import 2.754s · 68MB

134MB installed

● package 134MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

article

✓ import newspaper article = newspaper.article(url)

✗ from newspaper import Article; Article(url).download().parse()

The 'newspaper.article()' helper function streamlines downloading and parsing, combining the Article object instantiation, download(), and parse() calls into one step. The direct Article class import and method calls still work but are more verbose for a single article.

build

✓ import newspaper source = newspaper.build(url)

Used for building a 'Source' object to crawl an entire news website.

This quickstart demonstrates how to extract key information from a single news article using the `newspaper.article()` helper. It retrieves the title, authors, publish date, top image, and then performs NLP to get a summary and keywords. A commented-out example shows how to initialize and crawl an entire news source using `newspaper.build()` and iterate through its articles.

import newspaper # Example for a single article url = "https://edition.cnn.com/2023/11/08/china/china-blizzard-disruption-intl-hnk/index.html" article = newspaper.article(url) print(f"Title: {article.title}") print(f"Authors: {article.authors}") print(f"Publish Date: {article.publish_date}") print(f"Top Image: {article.top_image}") # Perform NLP for keywords and summary (requires NLTK and other NLP dependencies if installed) article.nlp() print(f"Summary: {article.summary}") print(f"Keywords: {article.keywords}") # Example for processing a news source (website) # cnn_paper = newspaper.build('http://cnn.com') # for article_obj in cnn_paper.articles: # print(article_obj.url) # article_obj.download() # article_obj.parse() # print(article_obj.title)

Debug

Known issues

breakingNewspaper4k requires Python 3.10 or higher. Older Python versions (3.8 and 3.9) are no longer officially supported as of version 0.9.4, though they might still function. Ensure your Python environment meets this requirement.

fix

Upgrade your Python environment to 3.10 or newer.

affects: >=0.9.4

gotchaThe Google News integration (`GoogleNewsSource`) can be unstable. Google frequently changes its HTML structure and URL encoding, which may cause this functionality to break without notice. This requires the `gnews` optional dependency.

fix

Be prepared for potential breakages and monitor for updates to the library. Consider alternative methods if reliability is critical.

affects: All

gotchaThe `article.nlp()` method, which extracts keywords and summaries, currently works most reliably on Western languages. Its performance and accuracy might be limited for non-Western languages, even with language-specific optional dependencies installed.

fix

Verify NLP output for non-Western languages and consider external NLP libraries for more robust analysis if needed.

affects: All

gotchaAggressively downloading many articles from a single source using multi-threading or rapid requests can lead to rate limiting, IP blocks, or CAPTCHA challenges from websites. Always respect `robots.txt` if enabled.

fix

Implement delays between requests, use proxy rotation, and rotate user-agent strings. Install `protego` optional dependency for robots.txt enforcement. For heavily protected sites (e.g., Cloudflare), `cloudscraper` (optional dependency) or external tools like Playwright might be necessary.

affects: All

gotchaWhen using the `Article` class directly (not `newspaper.article()`), you must explicitly call `article.download()` and `article.parse()` before attempting to access most article attributes (like `title`, `text`, `authors`, `publish_date`) or calling `article.nlp()`. Failure to do so will result in errors or empty data.

fix

Ensure `article.download()` and `article.parse()` are called sequentially before accessing article properties or running NLP.

affects: All

deprecatedThe `text_cleaned` and `clean_doc` attributes/methods have been deprecated and removed. Direct access to `article.clean_top_node` is also removed.

fix

Remove any usage of `text_cleaned` or `clean_doc`. The primary article text is available via `article.text`.

affects: >=0.9.2

Upgrade

Version history

0.9.5latest on PyPI · released Feb 28, 2026

Audit

Dependencies

beautifulsoup4requiredHTML parsing and navigation

PillowrequiredImage processing for top image extraction

PyYAMLrequiredConfiguration handling

lxml[html_clean]requiredEfficient XML/HTML parsing and cleaning

nltkoptionalNatural Language Toolkit for text processing (optional, for nlp() functionality)

requestsrequiredHTTP requests for downloading article content

feedparserrequiredRSS/Atom feed parsing

tldextractrequiredExtracting top-level domain from URLs

python-dateutilrequiredRobust date/time parsing

typing-extensionsrequiredTyping support

brotlirequiredBrotli compression support

cloudscraperoptionalBypassing Cloudflare protection (optional)

gnewsoptionalGoogle News integration (optional)

jiebaoptionalChinese language support (optional)

pythainlpoptionalThai language support (optional)

tinysegmenteroptionalJapanese language support (optional)

indic-nlp-libraryoptionalBengali, Hindi, Nepali, Tamil language support (optional)

protegooptionalrobots.txt enforcement (optional)

Agent activity

14 hits · last 30 days

node

Resources

packagehttps://pypi.org/project/newspaper4k/ ↗