A huge responsibility of every site owner, and our clients to make sure Google does not crawl certain pages of your website such as your cart or checkout page. Sometimes also, the Robots.TXT Page may not be setup correctly. The main reason to have a Robots.TXT Page is Google will penalize you if they can access some personally identifiable information. That makes a site insecure or without security to the consumer and vulnerable to threats or hacking. Its recommended we check and test that your Robots.TXT Page is setup properly.
Issues on why you will want to use a Robots.TXT file :
- There are certain links or images you will want blocked from the search engine
- Some Links you have have special instructions for the robots
- You are making a website and it's not complete yet, so you want to use Robots. TXT to stop Google from indexing at first until you are ready