Sitemap & robots crawl challenge

Start from discovery files, not a hard-coded URL list.

Challenge

  1. Fetch /robots.txt and find the Sitemap line
  2. Fetch /sitemap.xml and list all <loc> URLs
  3. Filter URLs under /playground/ and crawl titles
  4. Bonus: read /llms.txt for human-oriented map