We are excited to announce that the IIS.NET Forums are moving to the new Microsoft Q&A experience. Learn more >

SEO Toolkit only searching my home page [Answered]RSS

26 replies

Last post Oct 11, 2010 01:40 PM by CarlosAg

  • SEO Toolkit only searching my home page

    Jan 08, 2010 10:15 AM|tgakk|LINK

     I just installed the toolkit and ran an analysis.  I gave it my http://www.<mysite>.com address as the root folder.  It came back and told me that I had two violations.  It says the Redirection response results in another redirection.  What I'm doing is determining the visitors IP Address and setting the page culture appropriately based on where they are geolocated, so to set the page culture, it is redirecting the page to itself once.  I'm not sure how to fix the error and still have the page set the culture appropriately.  That's one issue.

     The second issue is the one in my thread title.  The only content it seems to have searched was my homepage.  I have lots of links on the homepage, so I'm not sure why it only pulled back results for the homepage, unless the original redirection error is making it so it can't search any further.

    Any thoughts on how to get it to search my whole site?  The report is pretty empty, like it didn't find any links, but links are there!  This is an asp.net 3.5 page running on iis 6.

    Thanks in advance

  • Re: SEO Toolkit only searching my home page

    Jan 08, 2010 02:14 PM|emumford|LINK

    It may be in your best interest to set redirection for crawlers (user-agent rather than IP) to a fixed culture, what ever is most appropriate (for your primary audiance perhaps?) and respond with a 301 Permanent Redirect rather than a 302, if you have to redirect at all and are not already using a 301... does the root of your site reflect the default culture, or must it be accessed via a sub path at all times? 

    I've honestly no experience with supporting multicultural sites, but I do have another idea... it may be wise to produce a landing page specifically for crawlers that links to the various cultures, allowing them to index all of the possibilities. Or, rather than relying on geolocation and redirection produce a homepage that allows the user to choose the region/culture... a majority of company websites I have visited take this approach, and it may indeed be directly related to SEO... but again, I'm not terribly familiar on the impact that multiple cultures may have on SEO in general.  Perhaps a combination of this option...

    I'd say it's obvious that from the get go you have issues hindering the SEO value of the site as it is, so it may be time to evaluate from that alone.

  • Re: SEO Toolkit only searching my home page

    Jan 08, 2010 02:24 PM|tgakk|LINK

     Thanks for the information.  I do have it set where they can select a culture, but I'm initially setting it based on their ip.  I'll definitely look into your suggestions above.  I don't have sub-paths for each culture.  All the pages are the same, they just hit different localization files for translation based on the culture of the page.

    I'm still not clear as to why the seo toolkit is only crawling my home page and not any other pages on my site.  Is the redirect violation preventing the toolkit from crawling any pages beyond my home page?

  • Re: SEO Toolkit only searching my home page

    Jan 08, 2010 03:04 PM|CarlosAg|LINK

    Could you see in the Details of the page that was crawled if it found any additional links?

    Could you also let us know what ContentType is shown for the code? I'm wondering if it is XHTML.

     

  • Re: SEO Toolkit only searching my home page

    Jan 08, 2010 03:23 PM|tgakk|LINK

     I doubleclicked the page in the summary of the report and clicked on Content--it says "<html><head><title>Object moved</title></head><body>
    <h2>Object moved to <a href="http://www.xxxx.com/index.aspx">here</a>.</h2>
    </body></html>

    Says 302 Found.

    Content type is text/HTML.

    Let me know if you need any other information and thanks for all of the assistance!

  • Re: SEO Toolkit only searching my home page

    Jan 08, 2010 04:35 PM|emumford|LINK

    302 = "Found" redirect and the search engines will not follow those.

    You need to use a 301 = "Permanent" redirect... now, that's not possible when using ASP.NET 3.5 or below, as they dont allow you to specify the type of redirect (coming in .net 4.0 though)...

    So for now, I'd make sure not to redirect any user agent that is a cralwler and just default the culture to your "primary" and allow the page to just load as normal.

  • Re: SEO Toolkit only searching my home page

    Jan 08, 2010 04:52 PM|tgakk|LINK

     How do I tell if it is a crawler or a normal user in asp.net?

  • Re: SEO Toolkit only searching my home page

    Jan 08, 2010 07:46 PM|CarlosAg|LINK

    Actually that is not true, Search Engines will follow 302 so that should work.

    As a separate note, it is possible to generate a 301 using ASP.NET 2.0 by setting the StatusCode to 301 and adding the location header, but you are certainly right that .net 4.0 has now a Permanent Redirect feature that makes this simpler.

    In the details (in the Links tab) of that page do you see the links listed? you show that the page says: http://www.xxxx.com/index.aspx  are you crawling http://www.xxxx.com/ ?

    In the main report dashboard page, Could you do a "Query->New Query", and click Execute. Add the column "Is External" to the report and see if which URLs are reports and if they are treated as external or not.

     

     

  • Re: SEO Toolkit only searching my home page

    Jan 08, 2010 08:48 PM|tgakk|LINK

     I'm trying to have it crawl http://www.xxxxx.com.  In my web host, I only have index.aspx as my default doc so it should just use index.aspx as my default page when http://www.xxxx.com is hit. 

     If I doubleclick the www.xxxxx.com page it finds, the links tab shows no links on the page. In pages linked by this page in the bottom part of the Links tab, I have http://www.xxxxxx.com/ listed as Linking-url .   Linked-URL says http://www.xxxxxx.com/index.aspx.   Linked-Status says Found.  Link Type is Redirect.  Link Title is Object Moved.

    If I run the query you suggest above, it pulls http://www.xxxxxxx.com and http://www.xxxxxx.com/index.aspx.  Adding the Is External as a filter and executing the query pulls back no results.  However, adding just the column to the report states False for both links.

    I promise the main index.aspx page has links ;).

    Let me know if there is something else I can try to get results for all of my pages.  I'm not clear why it is only crawling my home page.  I'd put my real URL up for you to test on to see if you could see why it wasn't pulling results, but it is still a work in progress and I don't want to post it in this forum.

  • Re: SEO Toolkit only searching my home page

    Jan 11, 2010 02:21 PM|drmadcow|LINK

    @emumford

    Search Engines support both 301 & 302 redirects however there are some major differences.  If you do a 301 redirect the search engines will index the new location and drop the old location from their index.  If you do a 302 redirection it will pick up the content from the new location but display it at the URL of the original.  This is the reason you will quite often see www.domain.com ranking but when you click on the link you get redirected to www.domain.com/en/default.aspx without seeing the aspx location in the rankings.

  • Re: SEO Toolkit only searching my home page

    Jan 11, 2010 04:30 PM|emumford|LINK

    Thanks for the correction Carlos, I totally inteded that to specify when using "Response.Redirect"... you are absolutly correct.

  • Re: SEO Toolkit only searching my home page

    Jan 11, 2010 05:01 PM|emumford|LINK

    Thanks for the clarification.  After the previous statement, I attempted some short lived research to see where my misunderstanding lay... this however is the best, short, to the point description.  Thank you!
  • Re: SEO Toolkit only searching my home page

    Jan 11, 2010 07:05 PM|CarlosAg|LINK

    What is the content type shown for http://www.xxxxxx.com/index.aspx ?

    At this point my guess is that something is happening that is making us not parse correctly the index.aspx, it could be that we do not recognize the ContentType returned as HTML so the HTML parser is not run and we dont find any links inside. It could also be that something in the markup really breaks us from parsing it.

    Do you have any violations listed in the details of http://www.xxxxxx.com/index.aspx ? What about the details for example Title, Keywords, HEading, Description, etc? are they populated correctly in the details dialog or empty?

     

  • Re: SEO Toolkit only searching my home page

    Jan 11, 2010 08:42 PM|tgakk|LINK

     Content type is text/html.  The only violation says "The redirection response results in another redirection".  Content just has some html that says "Object Moved".  The header looks like:

    Set-Cookie: ASP.NET_SessionId=vwdgdya30qlisinige45wm45; path=/; HttpOnly,clickLocalization=Language=en-US&Currency=US Dollars&Units=US; expires=Sat, 08-Jan-2011 14:43:05 GMT; path=/,clickLocalization=Language=en-US&Currency=US Dollars&Units=US; expires=Sat, 08-Jan-2011 14:43:05 GMT; path=/,clickLocalization=Language=en-US&Currency=US Dollars&Units=US; expires=Sat, 08-Jan-2011 14:43:05 GMT; path=/
    Server: Microsoft-IIS/6.0
    X-AspNet-Version: 2.0.50727
    X-Powered-By: ASP.NET
    Cache-Control: private
    Content-Type: text/html; charset=utf-8
    Content-Length: 157

    Title just says "Object Moved".  It seems like the redirection is causing it to only pull the words "Object Moved" and not actually follow the redirection to the new page to finish crawling the links on the real page.  If I go directly to my page in the browser, it pulls up correctly--I don't get the "Object Moved" HTML page like the SEO Toolkit is finding.

    What is causing the Toolkit to not actually follow the redirection to the new "real" page?

  • Re: SEO Toolkit only searching my home page

    Jan 12, 2010 12:26 AM|CarlosAg|LINK

    Interesting, do you see a "Location" header in the Headers tab in the Details dialog?

    If it is returning a redirect it should probably send the path that is redirecting to, this is probably what is confusing it. If I may ask, how is the redirect generated? Is it using the standard Response.Redirect() api in asp.net?

    I'm assuming you do not see any links in the Links tab, right? The HTML content returned by a Redirect (301 or 302) is not processed which explains why we would not see any links inside that HTML, however it should have a Location header that tells the browser which URL to redirect to, which in theory should be followed by SEO Toolkit

  • Re: SEO Toolkit only searching my home page

    Jan 15, 2010 10:34 AM|tgakk|LINK

     Sorry for the delay in responding.  The header for the page is included in my previous post.  I am merely doing a response.redirect in asp.net to set the appropriate page culture.  The two inbound links I have are http://www.xxxxx.com and http://www.xxxxx.com/index.aspx.  Outbound links are http://www.xxxxx.com/index.aspx and as mentioned, the content type is text/html.

  • Re: SEO Toolkit only searching my home page

    Jan 15, 2010 02:45 PM|CarlosAg|LINK

    That is really weird, in the headers you posted I could not see any Location header which means AFAIK there is nowhere to redirect the request to.

    Is there any way you could provide more details on how the Redirect is being performed and the scenario, based on the headers my guess is that this is trying to check for Session state.

    Are you by any chance detecting requests that do not have sometihng Session state and redirecting them to the home page if that is the case?

  • Re: SEO Toolkit only searching my home page

    Jan 15, 2010 03:19 PM|tgakk|LINK

     I have a pageculture class that every page inherits from.  Here is the code:

    <code>

    Public Class pageculture : Inherits Page
        Public theuser As User = populateLinqUser(User.Identity.Name)
        Protected Overrides Sub InitializeCulture()
            'redirect to www page if not localhost.

            If Request.Url.ToString.ToUpper.Contains("LOCALHOST") = False And Request.Url.ToString.ToUpper.Contains("WWW.XXXXXXXXXX.COM") = False Then
                'redirect
                Response.Redirect(Request.Url.ToString.Replace("xxxxxxxxxxxx.com", "www.xxxxxxxxx.com"))
            End If

            Dim myCookie As HttpCookie = DirectCast(Request.Cookies("clickLocalization"), HttpCookie)
            setPageCulture(Page.Culture, myCookie, Request, Response)
            MyBase.InitializeCulture()

            If User.Identity.IsAuthenticated Then
                'see if their session var for loginguid is the same as the other one or not
                If Session("loginguid") <> theuser.loginguid Then
                    'logout User
                    FormsAuthentication.SignOut()

                  
                    If Request.Url.AbsoluteUri.ToLower.Contains("/content/") Then
                        Response.Redirect("~/login.aspx")
                    Else
                        'redirect to same page
                        Response.Redirect(Request.Url.AbsoluteUri)
                    End If

                End If
            End If
               

        End Sub

    </code>

    In the setpageculture sub, it checks to see if there is a cookie value set for the culture and if so, sets the page culture appropriately and redirects to the same page.  If no value is set for the culture in the cookie, it detects the users IP and sets the culture appropriately and redirects.  If the culture is the same as what was in the cookie, it doesn't redirect the page to itself.  The redirect is:

    response.Redirect(request.Url.AbsoluteUri)

    Maybe there is a better way to set the page culture, but this is how I've seen it done elsewhere.


  • Re: SEO Toolkit only searching my home page

    Jan 15, 2010 04:06 PM|CarlosAg|LINK

    My guess is that the problem is in the SetPageCulture method (not sure without seeing the code) but what the most important think to keep in mind is that Robots (Search Engine Crawlers like Bing or others) including SEO Toolkit does not support Cookies which means the cookie you are trying to read will never be set and the thing that is happening is that you probably end up redirecting to the original URL and it of course this means no other pages will ever be seen.

     

     

  • Re: SEO Toolkit only searching my home page

    Jan 15, 2010 04:24 PM|tgakk|LINK

     Here is my code.  If you have any suggestions on how to do what I'm trying to do, yet make it more search engine friendly, I'd appreciate it.  I'm just getting in to SEO.  I edited a few minor things of course before posting to generalize some pieces, but the core code is there.

    <code>

    Public Shared Sub setPageCulture(ByVal theculture As String, ByVal mycookie As HttpCookie, ByVal request As HttpRequest, ByVal response As HttpResponse)
            'see if cookie set first.  If so, then use that value, otherwise use it based on language

            If mycookie IsNot Nothing Then
                If mycookie.Values("Language") = "xxxxxx" Then

                    System.Threading.Thread.CurrentThread.CurrentUICulture = New System.Globalization.CultureInfo("xxxxxx")
                    System.Threading.Thread.CurrentThread.CurrentCulture = New System.Globalization.CultureInfo("xxxxxx")

                ElseIf mycookie.Values("Language") = "en-US" Then

                    System.Threading.Thread.CurrentThread.CurrentUICulture = New System.Globalization.CultureInfo("en-US")
                    System.Threading.Thread.CurrentThread.CurrentCulture = New System.Globalization.CultureInfo("en-US")
                 
                End If

            Else
                'no cookie, so set one based on ip address and change values accordingly
                Dim theip As String = request.UserHostAddress
                Dim thecountryname As String = ""
                If theip <> "127.0.0.1" Then
                    Dim ls As New geomappingdll.LookupService("c:\domains\xxxxxxxx.com\wwwroot\GeoLiteCity.dat", geomappingdll.LookupService.GEOIP_MEMORY_CACHE)
                    'get country of the ip address
                    Dim c As geomappingdll.Country = ls.getCountry(theip)
                    'Dim mystr As String = (" code: " + c.getCode())
                    'Dim mystr2 As String = (" name: " + c.getName())
                    thecountryname = c.getName()
                Else
                    thecountryname = "United States"
                End If
             
                If Not String.IsNullOrEmpty(thecountryname) Then
                    If thecountryname = "xxxxx" Then
                        System.Threading.Thread.CurrentThread.CurrentUICulture = New System.Globalization.CultureInfo("xxxxx")
                        System.Threading.Thread.CurrentThread.CurrentCulture = System.Globalization.CultureInfo.CreateSpecificCulture("xxxxxx")
                        addLanguageCookie("xxxxxx", request, response)
                        addCurrencyCookie("xxxxxx", request, response)
                        addUnitsCookie("Metric", request, response)

                    Else
                        'some other country so set to english
                        System.Threading.Thread.CurrentThread.CurrentUICulture = New System.Globalization.CultureInfo("en-US")
                        System.Threading.Thread.CurrentThread.CurrentCulture = System.Globalization.CultureInfo.CreateSpecificCulture("en-US") 'System.Globalization.CultureInfo.CreateSpecificCulture("xxxx")

                        addLanguageCookie("English", request, response)
                        addCurrencyCookie("US Dollars", request, response)
                        addUnitsCookie("US", request, response)
                    End If


                Else
                    'didn't get a value so set to english
                    System.Threading.Thread.CurrentThread.CurrentUICulture = New System.Globalization.CultureInfo("en-US")
                    System.Threading.Thread.CurrentThread.CurrentCulture = System.Globalization.CultureInfo.CreateSpecificCulture("en-US")

                    addLanguageCookie("English", request, response)
                    addCurrencyCookie("US Dollars", request, response)
                    addUnitsCookie("US", request, response)
                End If
            
                response.Redirect(request.Url.AbsoluteUri)
            End If

        End Sub
     

    </code>

  • Re: SEO Toolkit only searching my home page

    Jan 15, 2010 04:57 PM|CarlosAg|LINK

    I think bottomline you want to choose the right locale you want for the Search Engine to use.

    My guess is that your code in many pages will try to find the value for Language, Currency and Units from the cookie. Assuming you could abstract the logic from reading it from the cookie, you could check in the code where these settings are loaded from the cookie and react accordingly if the cookie is not set. You could just special check the UserAgent and see if its msnbot, or iisbot, or whatever user agent you detect as a robot and not do the redirection logic.

    So as a simple test in your SetPageCulture just before you do the Response.Redirect just add a check for UserAgent to see if it is iisbot, something like:

                If Not IsBot(request) Then
                    response.Redirect(request.Url.AbsoluteUri)
                End If

     And then a "VERY simple" (oversimplified) check could be something like:

         Shared Function IsBot(ByVal Request As HttpRequest) As Boolean
            Return Not String.IsNullOrEmpty(Request.UserAgent) AndAlso _
                (Request.UserAgent.Contains("iisbot") OrElse _
                 Request.UserAgent.Contains("msnbot") OrElse _
                 Request.UserAgent.Contains("googlebot"))
        End Function

    Of course you also want to handle similarly when loading the values from the cookie and choose a default when you detect it is a Bot.

    Please let us know if by doing something like this the problem indeed goes away.

  • Re: SEO Toolkit only searching my home page

    Jan 15, 2010 05:13 PM|tgakk|LINK

     Interesting--I'll try out the code over the next few days and see how it goes.  Thanks a lot for the assistance.  Hopefully it will make a difference.  I'll report back once I get a little bit of time to implement it.

  • Re: SEO Toolkit only searching my home page

    Jan 16, 2010 12:41 AM|tgakk|LINK

     I implemented the code you posted and the toolkit was able to crawl my site.  I haven't fully looked through the data but it seems pretty complete.  Thank you so much for your assistance!  I'm sure I'll have more questions as I look through all of the violations, but I'll post them in a different thread if I can't find the answer.

     Thanks again!

  • Re: SEO Toolkit only searching my home page

    Oct 08, 2010 05:36 AM|picaleo|LINK

    I am facing the same problem.
    Firstly, it's working. And I have seen all the warnings.
    I have fixed some problems and want to run test again.
    But it only read home page and says that there is one page.
    Same URL, I didn't select any options.
    Please see the screen shot below.. Thanks.
    SEO Toolkit Problem

  • Re: SEO Toolkit only searching my home page

    Oct 08, 2010 03:02 PM|CarlosAg|LINK

    The reason it is not working for your site is because it is not returning the right Content Type. Browsers (and SEO Toolkit) use that header to understand what kind of content is being served (note that file extensions are not used), and since your site is not returning that header it is not possible for SEO Toolkit to determine that is HTML and so it is not parsing the content to find links, etc.

    If you use a Content Management System (CMS) make sure that it returns the right content, or if it is your own code make sure to specify the Response.ContentType.

  • Re: SEO Toolkit only searching my home page

    Oct 11, 2010 02:33 AM|picaleo|LINK

    Hi CarlosAg, Thanks for your reply. I have added in ContentType to my pages and still no luck. Please advise what could be the problem... Thanks

  • Re: SEO Toolkit only searching my home page

    Oct 11, 2010 01:40 PM|CarlosAg|LINK

    From what I can see (assuming we are talkiing about the site in the image) your site is not sending the Content Type header. It should be set to text/html.

     Please use Fiddler or some other networking trace tool and you will see that it is not sending that:

    Headers that I get are:

    HTTP/1.1 200 OK
    Proxy-Connection: Keep-Alive
    Connection: Keep-Alive
    Transfer-Encoding: chunked
    Date: Mon, 11 Oct 2010 17:36:58 GMT
    Server: Microsoft-IIS/7.0
    Cache-Control: private
    X-Powered-By: UrlRewriter.NET 2.0.0
    Set-Cookie: ASP.NET_SessionId=***; path=/; HttpOnly
    X-AspNet-Version: 2.0.50727
    X-Powered-By: ASP.NET

     As you can see there is no Content-Type, in contrast the headers from a correct site:

    ... 

    Proxy-Connection: Keep-Alive
    Connection: Keep-Alive
    Content-Length: 142
    ...

    Content-Type: text/html; charset=utf-8
    Server: Microsoft-IIS/7.5
    Cache-Control: private
    X-AspNet-Version: 2.0.50727
    X-Powered-By: ASP.NET