We are excited to announce that the IIS.NET Forums are moving to the new Microsoft Q&A experience. Learn more >

Can the user-agent of the SEO crawler be changed?RSS

8 replies

Last post Jun 12, 2013 08:13 PM by wksears

  • Can the user-agent of the SEO crawler be changed?

    Jul 11, 2011 10:21 AM|adamhems|LINK

    The site I own that wish to check is hosted for me, and I do not control the robots.txt page it serves up. My problem is that it only allows the common web search crawlers and disallows everything else: http://photos.hemsfamily.com/robots.txt

    Can I override the user-agent that the SEO crawler behaves as in order for it to index my stuff?

    robots.txt crawlers user-agent

  • Re: Can the user-agent of the SEO crawler be changed?

    Jul 18, 2011 12:33 AM|CarlosAg|LINK

    Unfortunately no, we decided not to allow it since it could be used to disguise the crawler and potentially try to bypass logic that servers would not be able to identify. So even though you could do that with others, we decided to "be good citizens" and not enable hiding it.

    That means that unfortunately in valid cases like yours it won't be possible.

  • Re: Can the user-agent of the SEO crawler be changed?

    Oct 27, 2011 03:51 AM|jackchong|LINK

    Hi seoblogging, do we really need to edit the pages html for that matter? I'm using wordpress with SEO plugin that allows me to select which page to be index and which not. Is this the same as compare to modifying the html code?

  • Re: Can the user-agent of the SEO crawler be changed?

    Nov 08, 2011 02:25 AM|nitinslash|LINK

     I agree with Calos that, cannot control the robots.txt then it is not possible to change the behavior of crawlers

  • Re: Can the user-agent of the SEO crawler be changed?

    Nov 21, 2011 11:04 AM|adamhems|LINK

    Thanks everyone - I do use Webmaster Tools on both Bing and Google to see what is being indexed but I have so many pages (30k+) which are created automagically by my hosting provider (hence I don't know what they all are), that I was hoping to use the SEO toolkit to do do a thorough analysis of what my site actually looks like to a crawler like Bing and Google so that I can exlude the duplicate pages via the Webmaster tools. Alas, I cannot do so as my hosting provider has one robots.txt for every single site they host so I just have to do it manually :-P

    - Adam Hems

  • Re: Can the user-agent of the SEO crawler be changed?

    Jan 14, 2012 02:01 AM|RachaelJetkins|LINK

    There is an add-on that can be installed, after installing that, go to Firefox option and you will come across the add-on user agent switcher added to the list of the existing add-ons. This add-on comprises of a Google bot user agent that you can select.Sometimes the existing user agents get outdated when Google changes the user agent; therefore in case you dont want the existing agent, it is recommended that you edit and add your own user agent.

  • Re: Can the user-agent of the SEO crawler be changed?

    Jan 14, 2012 02:04 AM|RachaelJetkins|LINK

     You can install an add on if you want that provides with the present Google bot user agent. You can edit or add your own user agents there. This is the user agent switcher provided by Firefox.

  • Re: Can the user-agent of the SEO crawler be changed?

    Jan 20, 2012 06:18 AM|jhonybravo222|LINK

     yes you can change it using webmaster tool

  • Re: Can the user-agent of the SEO crawler be changed?

    Jun 12, 2013 08:13 PM|wksears|LINK

    I would like to use IIS SEO Toolkit to crawl our site with a mobile user agent. Our site displays differnet pages to desktop and mobile users. Ideally, I'd like to be able to set the useragent to emulate Google's smartphone bot:

    Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)