SEO COCKTAILS
Robots.txt on the Beach (Sex on the Beach)
What is robots.txt? Let's get some knowledge on the search engine file that says yes or no to crawling your web pages.
SEO BARTENDER
Lance Beaudry
Lance doesn't have time for the spacesinhissentences. He's always going, curious, and extremely competitive (mostly with himself). He loves to investigate a website to identify opportunities for more traffic and conversions. Spend a little time with him and he'll give you a treasure chest full of website marketing ideas to help you get more results.
Why This Recipe Works
Your robots.txt file tells search engine crawlers which pages or files the crawler can or can’t request from your site.
Ingredients
- One website
- One robots.txt file
- One nerd to edit the file
Steps
- Make sure you have a website.
- Get access to your robots.txt file (multiple options available).
- Modify that thing to make sure search engine crawlers crawl what they should and don’t crawl what they shouldn’t.
