// LEARN / CRAWL

Robots.txt Best Practices — Every Directive, Every Pitfall

Disallow vs noindex, wildcards, crawl-delay, sitemap reference, and the 6 robots.txt rules that accidentally deindex production sites.

By Mike Technical SEO Consultant
2026-05-21 · 10 min

Full article publishes in the next 1-2 weeks. Below is the outline this article will cover.

Overview

Disallow vs noindex, wildcards, crawl-delay, sitemap reference, and the 6 robots.txt rules that accidentally deindex production sites.

Topics covered

  • robots.txt examples
  • robots.txt syntax
  • disallow vs noindex
  • robots.txt for seo
  • robots.txt wildcard
// FREE-AUDIT

Get a free 30-minute technical SEO audit

Send your URL and one symptom. You'll get a Loom walkthrough back within 48 hours — no sales call.

Request a free audit
In plain English: Deep-dive on robots txt best practices. If you need this fixed on your own site, send the URL and you'll get a Loom walkthrough back within 48 hours.