Jul 31, 2022
You’ll want to remove the disallow statement causing the block. How you do this varies depending on the technology you’re using. WordPress If the issue impacts your entire website, the most likely cause is that you checked a setting in WordPress to disallow indexing. This mistake is common on new websites and following website migrations. Follow these steps to check for it: Click ‘Settings’ Click ‘Reading’ Make sure ‘Search Engine Visibility’ is unchecked. 3-wordpress-search-engine-block.png WordPress with Yoast If you’re using the Yoast SEO plugin, you can directly edit the robots.txt file to remove the blocking statement. Click ‘Yoast SEO’ Click ‘Tools’ Click ‘File editor’ WordPress with Rank Math Similar to Yoast, Rank Math allows you to edit the robots.txt file directly. Click ‘Rank Math’ Click ‘General Settings’ Click ‘Edit robots.txt’ FTP or hosting If you have FTP access to the site, you can directly edit the robots.txt file to remove the disallow statement causing the issue. Your hosting provider may also give you access to a File Manager that allows you to access the robots.txt file directly. Check for intermittent blocks Intermittent issues can be more difficult to troubleshoot because the conditions causing the block may not always be present. The workload like this whatsapp number list allows both the vendor and the affiliate to focus on. Clicks are the number of clicks coming to your website’s URL from organic search results. What I’d recommend is checking the history of your robots.txt file. For instance, in the GSC robots.txt tester, if you click the dropdown, you’ll see past versions of the file that you can click and see what they contained. 4-historic-robots-txt.gif The Wayback Machine on archive.org also has a history of the robots.txt files for the websites they crawl. You can click on any of the dates they have data for and see what the file included on that particular day. 5-wayback-machine.png Or use the beta version of the Changes report, which lets you easily see content changes between two different versions. 6-wayback-machine.gif How to fix The process for fixing intermittent blocks will depend on what is causing the issue. For example, one possible cause would be a shared cache between a test environment and a live environment. When the cache from the test environment is active, the robots.txt file may include a blocking directive. And when the cache from the live environment is active, the site may be crawlable. In this case, you would want to split the cache or maybe exclude .txt files from the cache in the test environment.