Registry / web-framework / express-robots-middleware

express-robots-middleware

library1.0.1jsnpmunverified

Middleware for Express.js applications to dynamically generate a robots.txt file. Version 1.0.1 is the current stable release, with infrequent updates since 2016. It supports multiple user agents, allow/disallow directives, and crawl-delay configuration. Differentiators include simple integration as route middleware and no external dependencies beyond Express.

web-framework

npm install express-robots-middleware

github ↗package ↗

INSTALL

IMPORT

SIG · EXPRESS-ROBOTS-MID

express-robots-middleware

web-frameworkjavascriptv1.0.1

Install

—

Import

—

Disk

—

Pass rate

0/ 6

Env Coverage0 / 6

glibc

18–22

musl

18–22

Install & Compatibility

Where this runs

tested against v? · npm install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

node 18–226 runs

build_error

glibc

node 18–226 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

expressRobotsMiddleware

✓ import expressRobotsMiddleware from 'express-robots-middleware'

✗ const expressRobotsMiddleware = require('express-robots-middleware').default

Default export; no named export.

Creates an Express app that serves a robots.txt with a disallow list and crawl delay.

import express from 'express'; import expressRobotsMiddleware from 'express-robots-middleware'; const app = express(); const robotsMiddleware = expressRobotsMiddleware({ UserAgent: '*', Disallow: ['/private', '/tmp'], Allow: '/', CrawlDelay: '10' }); app.get('/robots.txt', robotsMiddleware); app.listen(3000, () => console.log('Server running on port 3000'));

Debug

Known issues

gotchaOptions must be passed as an object with capitalized keys: UserAgent, Disallow, Allow, CrawlDelay. Using lowercase keys will be silently ignored.

fix

Use exactly { UserAgent: string, Disallow: string[], Allow: string, CrawlDelay: string }

affects: *

breakingIn version 1.0.0, the middleware returned a function that directly sent the response; in 1.0.1, it uses next() for compatibility with error handling.

fix

Upgrade to 1.0.1 to avoid unexpected behavior with Express error middleware.

affects: 1.0.0

gotchaThe middleware does not validate the options. Invalid or missing fields will result in an empty robots.txt or runtime errors.

fix

Always provide all required fields: UserAgent, Disallow array, Allow, CrawlDelay.

affects: *

Errors

Common errors & fixes

TypeError: Cannot read property 'set' of undefined

Middleware used without being bound to a route (e.g., app.use(robotsMiddleware) instead of app.get('/robots.txt', robotsMiddleware)).

fix

Bind the middleware to the '/robots.txt' route using app.get('/robots.txt', robotsMiddleware).

TypeError: expressRobotsMiddleware is not a function

Default import used incorrectly in CommonJS environment without the .default property.

fix

Use const expressRobotsMiddleware = require('express-robots-middleware').default;

Upgrade

Version history

1.0.1latest on npm

Audit

Dependencies

expressrequiredpeer dependency for providing request/response objects

Agent activity

2 hits · last 30 days

node

Resources

githubhttps://github.com/chriswalkr/express-robots-middleware ↗

packagehttps://www.npmjs.com/package/express-robots-middleware ↗