Robots.txt and sitemap.xml are 2 files in the root of a site that govern the relationship with search engines. Robots.txt tells the crawler what it may visit, while sitemap.xml lists the addresses you want discovered. Robots.txt restricts, the sitemap suggests - and that is the crucial difference.
Robots.txt is a plain text file at the root of the domain. It states what each crawler is allowed to access, and it usually also names the address of the sitemap.
It is important to understand that robots.txt does not remove a page from the index. It prevents crawling. A blocked page with links pointing to it from other sites can still appear in results - only without a description, because the crawler was not allowed to read it.
To remove something from the index you use the noindex directive in the page head. And here the classic mistake appears: if you block a page in robots.txt, the crawler cannot read the noindex directive, so the page stays in the index indefinitely.
A sitemap is a list of addresses you want the search engine to find, with a note of when each was last modified. It is particularly useful for new sites, large sites and those with few external links pointing at them.
A sitemap does not guarantee indexing. It aids discovery, but the decision on whether a page enters the index is made by the system based on quality.
The modification date only means something if it is accurate. A sitemap where every page carries the same date from a year ago conveys no information at all. Google has ignored priority tags since 2023 and there is no reason to spend time on them.
The most serious is blocking the whole site. One line left over from a staging version is enough to make a site vanish from search. Check this first if traffic dropped sharply after launch.
The second is blocking CSS and JavaScript files. Google renders pages like a browser, so if it cannot load the styles it sees a broken page and may judge it poor on mobile.
The third is a sitemap containing addresses that return an error or a redirect. A sitemap should list only final addresses returning a clean response and not blocked anywhere.
The fourth is a forgotten sitemap. A site that keeps adding pages while the sitemap is generated by hand once a year loses most of the benefit of the file.
Open both files directly in a browser - they must load and return a clean response. Then submit the sitemap in Search Console and follow the indexing report.
The report shows how many addresses from the sitemap are indexed and which were rejected, with a reason. That is the fastest way to spot a problem otherwise noticed only when traffic falls.
On larger sites it is worth splitting the sitemap by content type - one for pages, one for articles - and linking them through a sitemap index. Then it is immediately clear which group of content has a problem.
The first check is the simplest: your domain, a slash, the file name - open both directly in the browser and read what they say. If robots.txt contains a rule blocking everything, the site will not be crawled regardless of anything else.
Then use the URL inspection tool in Search Console. It shows whether Googlebot may crawl that page and whether it is indexed - the fastest way to confirm a rule behaves as you think.
After submitting the sitemap, follow the report for a few days. It is normal for not every address to be indexed at once, but if a large share is rejected the reason is stated in the report.
Particular care goes with launching a new site. Staging versions are usually blocked entirely, and that rule must be removed before the site goes live. Check too that the modification date in the sitemap reflects reality, because a sitemap with the same date on every address conveys nothing.
Last updated: 17 August 2026
Send the address and we will tell you where you stand - no obligation.
Send an enquiry
