IIS Search Engine Optimization Toolkit
Can the user-agent of the SEO crawler be changed?
Last post Jun 12, 2013 08:13 PM by wksears
Jul 11, 2011 10:21 AM|adamhems|LINK
The site I own that wish to check is hosted for me, and I do not control the robots.txt page it serves up. My problem is that it only allows the common web search crawlers and disallows everything else:
Can I override the user-agent that the SEO crawler behaves as in order for it to index my stuff?
Jul 18, 2011 12:33 AM|CarlosAg|LINK
Unfortunately no, we decided not to allow it since it could be used to disguise the crawler and potentially try to bypass logic that servers would not be able to identify. So even though you could do that with others, we decided to "be good citizens" and
not enable hiding it.
That means that unfortunately in valid cases like yours it won't be possible.
Oct 27, 2011 03:51 AM|jackchong|LINK
Hi seoblogging, do we really need to edit the pages html for that matter? I'm using wordpress with SEO plugin that allows me to select which page to be index and which not. Is this the same as compare to modifying the html code?
Nov 08, 2011 02:25 AM|nitinslash|LINK
I agree with Calos that, cannot control the robots.txt then it is not possible to change the behavior of crawlers
Nov 21, 2011 11:04 AM|adamhems|LINK
Thanks everyone - I do use Webmaster Tools on both Bing and Google to see what is being indexed but I have so many pages (30k+) which are created automagically by my hosting provider (hence I don't know what they all are), that I was hoping to use the SEO
toolkit to do do a thorough analysis of what my site actually looks like to a crawler like Bing and Google so that I can exlude the duplicate pages via the Webmaster tools. Alas, I cannot do so as my hosting provider has one robots.txt for every single site
they host so I just have to do it manually :-P
- Adam Hems
Jan 14, 2012 02:01 AM|RachaelJetkins|LINK
There is an add-on that can be installed, after installing that, go to Firefox option and you will come across the add-on user agent switcher added to the list of the existing add-ons. This add-on comprises of a Google bot user agent that you can select.Sometimes
the existing user agents get outdated when Google changes the user agent; therefore in case you dont want the existing agent, it is recommended that you edit and add your own user agent.
Jan 14, 2012 02:04 AM|RachaelJetkins|LINK
You can install an add on if you want that provides with the present Google bot user agent. You can edit or add your own user agents there. This is the user agent switcher provided by Firefox.
Jan 20, 2012 06:18 AM|jhonybravo222|LINK
yes you can change it using webmaster tool
Jun 12, 2013 08:13 PM|wksears|LINK
I would like to use IIS SEO Toolkit to crawl our site with a mobile user agent. Our site displays differnet pages to desktop and mobile users. Ideally, I'd like to be able to set the useragent to emulate Google's smartphone bot:
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)