Registry / web-framework / robots.txt

robots.txt

JSON →
library1.1.0jsnpmunverified

The `robots.txt` package provides simple Connect/Express middleware to serve a static `robots.txt` file from the filesystem. Currently at version 1.1.0, this package was last published over a decade ago, indicating it is no longer actively maintained. Its core functionality involves a single synchronous read of the `robots.txt` file upon application startup, with subsequent requests served from memory. It automatically sets appropriate cache headers. Unlike many other `robots.txt` related packages on npm, this library is strictly for serving the file content at the `/robots.txt` endpoint, not for parsing or evaluating robots exclusion rules for crawlers. Its simplicity and 'fire-and-forget' setup were its primary differentiators in its active period, but it lacks modern features like asynchronous file handling, ESM support, or TypeScript typings.

web-framework
npm install robots.txt
package ↗
INSTALL
IMPORT
SIG · ROBOTS.TXT
R
robots.txt
web-frameworkjavascriptv1.1.0
Install
—
Import
—
Disk
—
Pass rate
0/ 6
Env Coverage0 / 6
glibc
18–22
musl
18–22
Install & Compatibility
Where this runs
tested against v? · npm install
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
node 18–226 runs
build_error
glibc
node 18–226 runs
build_error
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

robots
✓ const robots = require('robots.txt')
✗ import robots from 'robots.txt'
This package exclusively uses CommonJS `require()` syntax as it was developed before widespread ESM adoption in Node.js.
robotsMiddleware
✓ app.use(robots(__dirname + '/robots.txt'))
✗ app.use('/robots.txt', robots(__dirname + '/robots.txt'))
The middleware should be registered without a path argument to `app.use()` as it internally handles the `/robots.txt` route.

This quickstart demonstrates setting up an Express application to serve a `robots.txt` file using the middleware. It creates a dummy `robots.txt` on the fly and shows how the middleware integrates into an Express app, serving the file at the `/robots.txt` endpoint.

const express = require('express'); const path = require('path'); const robots = require('robots.txt'); const app = express(); // Create a dummy robots.txt file for demonstration const robotsTxtContent = `User-agent: * Disallow: /admin/ Allow: / Sitemap: http://localhost:3000/sitemap.xml`; const robotsTxtPath = path.join(__dirname, 'robots.txt'); require('fs').writeFileSync(robotsTxtPath, robotsTxtContent); // Use the robots.txt middleware // The middleware will read robots.txt from the specified path once at startup. app.use(robots(robotsTxtPath)); // Example route for a sitemap (often linked from robots.txt) app.get('/sitemap.xml', (req, res) => { res.type('application/xml'); res.send('<urlset><url><loc>http://localhost:3000/</loc></url></urlset>'); }); // Example protected route app.get('/admin', (req, res) => { res.send('Admin area - disallowed by robots.txt'); }); // Other routes app.get('/', (req, res) => { res.send('Hello World! Check http://localhost:3000/robots.txt'); }); const PORT = 3000; app.listen(PORT, () => { console.log(`Server running on http://localhost:${PORT}`); console.log(`View robots.txt at http://localhost:${PORT}/robots.txt`); });
Debug
Known issues
breakingThis package relies on synchronous file reading at startup. While it ensures the file exists, it can block the event loop during application initialization if the file is large or on a slow filesystem.
fix
No direct fix within the package. For highly performant applications, consider serving static robots.txt via a web server (e.g., Nginx) or using custom asynchronous middleware.
affects: >=1.0.0
breakingThe package is effectively unmaintained, with its last publish being over 12 years ago. It may have compatibility issues with modern Node.js versions, Express versions (beyond 3.x/4.x), or lack crucial security updates.
fix
Consider migrating to a manually served static file using `app.get('/robots.txt', ...)` or a more modern, actively maintained alternative like `express-robots.txt`.
affects: >=1.1.0
gotchaThis package only serves a static `robots.txt` file. It does not provide any parsing, validation, or dynamic generation capabilities. For parsing, other packages like `robots-txt-parser` or `@chronocide/robots-txt` are available.
fix
If dynamic `robots.txt` generation or parsing is required, use a different library explicitly designed for those purposes.
affects: >=1.0.0
gotchaThe package does not support TypeScript out-of-the-box and does not have official `@types/robots.txt` definitions. Modern projects using TypeScript will need to declare their own module or resort to `require` with `@ts-ignore`.
fix
Create a `d.ts` declaration file (e.g., `declare module 'robots.txt';`) or use a different, actively maintained solution that provides TypeScript support.
affects: >=1.0.0
Errors
Common errors & fixes
Error: ENOENT: no such file or directory, open 'path/to/your/robots.txt'
The path provided to the `robots()` middleware function does not point to an existing `robots.txt` file.
fix
Ensure the path passed to `robots()` is an absolute path to a valid `robots.txt` file. Use `path.join(__dirname, 'robots.txt')` for relative paths within your project.
TypeError: app.use() requires middleware functions but got a [object Undefined]
This error typically occurs if `robots.txt` fails to load or if `robots()` is called without a valid path, causing it to return `undefined`.
fix
Verify that `robots.txt` is correctly installed and that the path passed to the `robots()` function is a non-empty string and resolves to a readable file.
Cannot GET /robots.txt
The `robots.txt` middleware is either not mounted correctly, or another middleware is intercepting the `/robots.txt` request before this middleware can handle it.
fix
Ensure `app.use(robots(pathToRobotsTxt))` is placed early in your middleware stack, before any general static file servers or route handlers that might capture `/robots.txt`.
Upgrade
Version history
1.1.0latest on npm
Audit
Dependencies
expressoptionalThis package is designed as middleware for Express.js applications.
connectoptionalThis package is also compatible with Connect-based applications.
Agent activity
6 hits · last 30 days
node
4
Resources
packagehttps://www.npmjs.com/package/robots.txt ↗
robots.txt — npm install robots.txt · libregistry