I have been working on a tool to access the sitemap of a news website to try to create feeds when the site does not provide feeds. The tool starts by reading the robots.txt file, which usually has a list of things that are “disallowed”. On one of the sites, it specifically included “Claude Code” […]
Andy Sylvester's Web
• Andy