google alert.

Ever since I started monitoring what search engine robots indexed on my web sites, I have also been using Google Alert. Google sends me reports (html email or plain text, your choice) of any web site that has been indexed with keywords that you specify. Not only can you make sure Google is following the rules set up in your robots.txt document, but you can also see what other web sites are being picked up by certain keywords that you use to attract users. Google Alert is free and very useful and I would recommend using it if you want to monitor your web presence as well as your competitor’s.