We are excited to announce that the IIS.NET Forums are moving to the new Microsoft Q&A experience. Learn more >

Analysis is looping through my site recursivelyRSS

3 replies

Last post Jun 09, 2009 12:48 PM by CarlosAg

  • Analysis is looping through my site recursively

    Jun 04, 2009 10:05 AM|Alex J|LINK

    I have a problem with one of my sites where the analysis process is looping through my site recursively. In the report I see the following -

    e.g.

    • http://sitename/filename
    • http://sitename//filename
    • http://sitename///filename
    • http://sitename////filename etc

     

    So I guess my questions are -

    • How can I stop the analysis tool from doing this?
    • Is this a problem with my site or with the analysis tool?
    • How can I trace where the problem starts?
    • I'm guessing somewhere in one (or more) of my pages I have a bad link somewhere and the analysis tool just keeps spidering.  If I go to http://sitename////////default.aspx and view the source I don't see any multiple //////'s so I guess that's why it keeps going...

    Any help, much appreciated.

    Thanks

    Alex

  • Re: Analysis is looping through my site recursively

    Jun 05, 2009 12:46 PM|CarlosAg|LINK

    Could you find one of those "deep routes" and right-click and use "View Routes for this Page".

    It should show the way we got into this page (sort of like a call-stack starting at the bottom). Then double click each of the entries and see where the link is being used.

    I have a feeling that it might be a redirect (302) that generates an incorrect link in the markup, but please let us know what do you see.

     

  • Re: Analysis is looping through my site recursively

    Jun 09, 2009 05:40 AM|Alex J|LINK

    Thanks, yes it is a redirect that is causing the problem.

    The response headers - 302, Location: /page.aspx
    In the content body of the response is some HTML and a link to %2Fpage.aspx

  • Re: Analysis is looping through my site recursively

    Jun 09, 2009 12:48 PM|CarlosAg|LINK

    This is a bug in ASP.NET redirection logic that it incorrectly encodes the attribute in the markup causing it to look like a different URL.

    We will probably disable parsing the markup for 302 responses in future versions since by now most browsers will not even display the content and just follow the Location header.

    What you can try doing for now is add a robots.txt in the root of your site that includes something like:

    User-agent: iisbot

    Disallow: /*//

    This will tell us to not follow any of the URL's that contain // slashes together which hopefully will only be this case. Either way when you re-run the analysis you will see an informational entry for each URL we decided not to visit based on robots which will confirm that. This should prevent the infinite loop.

    Let us know if this helped.