robots.txt errors: misspelled directives, rules before User-agent, blocked pages
Last reviewed 2026-09-19
robots.txt tells crawlers what they may fetch, and crawlers quietly ignore lines they cannot understand. A typo in a directive means the rule is skipped, and one rule in the wrong place can block your whole site.
Messages you might see
- “Unknown directive / robots.txt has errors”
- “Blocked by robots.txt (unexpectedly)”
- “Sitemap URL in robots.txt is not a full URL”
Why it happens
- Misspelled directives such as Useragent or Dissallow.
- Rules placed before any User-agent line.
- Paths missing their leading slash, or a Sitemap given as a relative path.
- Allow and Disallow rules that contradict each other.
Try this first (free, no tool needed)
- Fetch yourdomain.com/robots.txt in a browser and read it as a crawler would.
- Use Search Console's robots.txt report and the URL Inspection tool to see whether a page is blocked.
- Be careful with a bare `Disallow: /`, which blocks everything.
If that didn't work
These tools run in your browser. Your file is read and repaired on your own device and is never uploaded.
Malformed robots.txt Fixer
Corrects misspelled directives, missing colons, rules before any User-agent and paths missing their leading slash. It merges duplicate groups, removes duplicate rules, turns relative Sitemap paths into full URLs and flags Allow/Disallow conflicts for you to decide.
Open Malformed robots.txt FixerCommon questions
Does it decide what I should block?
No. It repairs syntax and flags conflicts. Which parts of your site to allow is your decision.
Is my file uploaded anywhere?
No. The tool runs as a program inside your own browser, so the file is read and repaired on your device and never sent to a server.