Data as of Jul 25, 2026 · Based on 2,718,867 AI responses across 9,511 prompts · See how Parse measures this
Protegopypi.orgProtego is a pure-Python library that parses robots.txt files and supports modern conventions. It exposes a Protego class with methods such as can_fetch(url, user_agent), crawl_delay(user_agent), and request_rate(user_agent), and includes sitemaps and a preferred_host derived from robots.txt. It is compatible with Python 3.10+ and installable via pip (pip install protego).
Parse Score
