How AI describes robots-parser

Robots Parser is an npm package that provides a robots.txt parser designed to be compliant with the draft specification for interpreting crawl directives. It supports directives like User-agent, Allow, Disallow, Sitemap, Crawl-delay, and Host, and exposes methods such as isAllowed, isDisallowed, getCrawlDelay, getSitemaps, getPreferredHost, and getMatchingLineNumber to inspect applicable rules. Install with npm install robots-parser (or yarn add robots-parser) and initialize it with a robots.txt URL and directives to determine whether crawling a URL is permitted for a given user-agent.