Go to robots.txt settings by clicking on the menu and going to Configuration > robots.txt > Settings Step 2 – Ignore robots.txt What this does is makes the crawler effectively crawl everything it can and not stop even if robots.txt tells it not to. To go that one step further, we make our robot a little naughty and tell it to ignore robots.txt. However, this does not mean there isn’t a page linked somewhere on the site which might still contain the text you are looking for. Pro Tip: Robots.txt – This is an important oneĪs a good crawler should, Screaming Frog is set to obey robots.txt as a good robot should. The reason being is that there is a piece of code which noted something like ‘id_warranty’ in the HTML so this crawler has picked up on it. The reality of this particular website is that ‘warranty’ does not appear on the webpage when you land. You will see in the above screenshot that there are 27 instances of ‘warranty’. From there, you can check through the resulting URLs to then go through and update the relevant pages on your live website. The overview will show your search results and you can also select which filter you want to look at on the left hand side. You will find your search results under the ‘custom’ tab. I’ll explain later why this is important to know. What this means is it searches the coding of the page and not necessarily what you see as a user. Note: As you can see in the screenshot, this searches the source code of the internal HTML page. It would be much easier filtering through the pages where the code doesn’t appear rather than potentially hundreds of pages where the code does appear. Often this might be something like your chat tool or a tool like Hotjar, Google Tag Manager or Google Analytics. When would you use this? if you have a piece of code that needs to appear on many pages or all pages. Pro Tip: ‘Does Not Contain’ is a search option. In the screenshot below, you can see we searched ‘lifetime’, ‘lifetime warranty’ and ‘warranty’. Great if you need to change an email address and a phone number. You can enter multiple search terms or ‘filters’ so you only need to perform one crawl to get data for multiple searches. This is where you input the search terms you are looking for. You’ll then get to the ‘Custom Search’ screen. On the menu, go to Configuration > Custom > Search Solution: This is how to use Screaming Frog to find that Specific Text Load your Screaming Frog applicationĮnter in URL details as you normally would. All references to ‘lifetime warranty’ had previously been removed but our clients were concerned that there may have ben an instance of ‘lifetime’ lingering around on the website. We now offer a very lengthy warranty period but no longer advertise ‘lifetime’. A website we work with once advertised and offered a ‘lifetime warranty’ on its products. The example today where we use Screaming Frog to make what could be an arduous task easy is this. Case Study Scenario why we used Screaming Frog to find specific content For example, a company changes phone number or email address and we need to find any page that this information exists. This can be so useful in so many scenarios. In context of todays topic, we can also use it to find specific text on a website. After scraping your website, it can help you to visualise the structure of the website, finds broken links and is full of useful information for anything from H1s, Metas, Canonicals. Screaming frog is one of the many SEO tools available with plenty of great features for SEOs.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |