How to write and submit a robots.txt file
Quick summary A robots.txt file tells search engines which pages they can and cannot crawl. It is placed in the site's root folder and follows simple rules. Creating, uploading, and checking the file in Google Search Console helps optimize crawling, protect sensitive content, and prevent unnecessary pages from appearing in results. This guide explains step by step how to create, configure, test, and submit robots.txt to improve SEO.
Basic guidelines for creating robots.txt A robots.txt file is a plain text document that guides search engine bots. It is essential for blocking admin panels and login pages, preventing the crawling of duplicate or irrelevant pages, and directing bots toward important content. Key rules: there can only be one robots.txt file per domain, it must be placed in the domain's main folder, saved as robots.txt in lowercase, follow the Robots Exclusion Protocol, rules are case-sensitive, and it must be accessible at https://yourdomain.com/robots.txt
How to create the file Use a simple text editor like Notepad on Windows or TextEdit on Mac in plain text mode. Avoid processors that add invisible formatting. Save with UTF-8 encoding and name the file robots.txt. Example of basic content: User-agent: * Disallow: /admin/ Allow: / Sitemap: https://www.yourdomain.com/sitemap.xml This blocks all crawlers from the /admin/ folder while allowing the rest and points to the sitemap to facilitate indexing.
How to write the rules Main directives: User-agent defines which bot the rule applies to, for example User-agent: Googlebot or User-agent: * for all crawlers. Disallow blocks access to a page or directory, for example Disallow: /checkout/ Disallow: /search/ Allow allows exceptions within a blocked directory, for example Allow: /search/results/ Sitemap points to the XML sitemap so bots can find it. Useful wildcards: the asterisk * matches any sequence of characters and the dollar sign $ indicates the end of the URL. Example to block PDFs: Disallow: /*.pdf$
Practical examples Block all crawlers: User-agent: * Disallow: / Block a directory: User-agent: * Disallow: /private/ Disallow: /admin/ Block a specific page: User-agent: * Disallow: /confirmation.html Allow only Googlebot: User-agent: Googlebot Allow: / User-agent: * Disallow: / Block PDF files: User-agent: * Disallow: /*.pdf$ Allow everything except a bad bot: User-agent: BadBot Disallow: / User-agent: * Allow: / Include sitemap: Sitemap: https://www.yourdomain.com/sitemap.xml
Uploading the file Place robots.txt in the domain's root folder so it is accessible at https://example.com/robots.txt. On hosted platforms, look for the SEO section to upload or edit robots.txt. In frameworks like Next.js, put robots.txt in the public folder so it is served from the root.
Testing the file Step 1 visit https://yourdomain.com/robots.txt in a browser to check public access. Step 2 use Google Search Console and the robots.txt Tester tool to detect syntax errors. Google also provides an open source parser for advanced testing. Make sure the rules do not block critical resources like CSS or JS that affect rendering and indexing.
Submitting to Google Google usually discovers robots.txt automatically, but you can force a review and speed up the application of changes by signing in to Google Search Console, opening robots.txt Tester and clicking Submit, or testing the URL with the URL inspection tool to see how Google views the page.
SEO best practices Use robots.txt to optimize the crawl budget by blocking irrelevant pages, duplicate content, or search parameters. Do not use robots.txt to hide content that should disappear from results; for that, use meta robots noindex or X-Robots-Tag headers. Keep a backup copy of file versions and document important changes when you modify the site structure or sitemaps.
Common errors 1 accidentally blocking essential resources like CSS and JS that prevent rendering 2 relying on robots.txt to hide sensitive content instead of protecting it with authentication 3 using multiple files or placing them in subfolders 4 forgetting to update the sitemap path if it changes
Frequently asked questions What does robots.txt do This tells bots what they can and cannot crawl. Does it hide my pages from Google If you only use robots.txt, blocked pages can still appear in results if other sites link to them. For complete removal use noindex. Can I block scrapers and AI bots Yes, by blocking their user agents in robots.txt, although it is not an infallible measure. What happens if I don't have robots.txt Bots will assume they can crawl everything. How often should I update it Update when the site structure changes or when adding or removing sitemaps.
About Q2BSTUDIO Q2BSTUDIO is a custom software and application development company specialized in advanced technology solutions. We offer custom software, custom applications, artificial intelligence services, and cybersecurity solutions for businesses of all sizes. We also provide aws and azure cloud services, business intelligence services, and power bi consulting. Our team develops AI agents and AI projects for companies that integrate language models and secure automation. If you need to optimize your website's crawling, improve the security of aws and azure cloud services, or build a platform with artificial intelligence and power bi, Q2BSTUDIO can handle the design, development, and maintenance.
How Q2BSTUDIO can help We audit the robots.txt configuration as part of a complete technical review. We create, test, and maintain robots.txt files aligned with SEO strategies, develop custom software that respects indexing rules, and offer secure implementation on aws and azure cloud services. We also deploy artificial intelligence solutions and AI agents to automate tasks and extract business intelligence with power bi visualizations. Our offering includes cybersecurity to protect data and access, ensuring compliance and control over what search engines can see.
Conclusions A well-designed robots.txt guides crawlers, protects sensitive areas, improves indexing focus, and saves crawl budget. Integrate robots.txt with noindex, sitemaps, and a complete technical SEO strategy. If you want your website and custom applications to work with maximum efficiency and security, contact Q2BSTUDIO for a consultation that combines custom software, artificial intelligence, cybersecurity, aws and azure cloud services, business intelligence services, AI for companies, AI agents, and power bi.
