We are excited to announce that the IIS.NET Forums are moving to the new Microsoft Q&A experience. Learn more >

Finding wrong robots.txtRSS

2 replies

Last post Aug 13, 2012 05:26 AM by elinejoseph85

  • Finding wrong robots.txt

    Aug 01, 2012 05:37 PM|ben_jones|LINK

    On our server I've been trying to run a Site Analysis report but it doesn't get beyond the site root due to a Robots.txt rule not allowing it, however there is no Robots.txt.

    If I go to 'http://.../robots.txt' it does find a file, and it's as below,

    User-agent: *
    Disallow: /

    But again, this file does not exist anywhere on the server, I've done a full search across all drives, all folders to be sure and there isn't a single Robots.txt on there, creating one with the correct content in it and placing that one in site root doesn't change anything and it continues to show the above.

    I just can't figure out whats going on. Any ideas would be really appreciated.

    Thanks

    IIS robots.txt site analysis report

  • Re: Finding wrong robots.txt

    Aug 01, 2012 10:54 PM|terridonahue|LINK

    Have you verified that you are looking at the right server? What does http://network-tools.com show as the A record for that domain?
    Terri Donahue
    Microsoft MVP ASPNET/IIS

    Please 'Mark as Answer' if this post helps you.
  • Re: Finding wrong robots.txt

    Aug 13, 2012 05:26 AM|elinejoseph85|LINK

    First thing that your web site has robot.txt not Robot.txt (case sensitive issue). Try to generate URL list using free sitemap tool generate or else use Xenu links analyzer to find the right path of robots.txt. However check files status might be any hidden file over there on web server or else might your checking wrong server. Get full status of listed issue...! Wish you good luck!