I have a number of inbound links that contain entities which result in 404 errors. I would like to create some url rewrite rules (IIS7) to take care of this. These rules need to work on any page since some of the links that contain entities could be several
folders deep and be a file other that index.php.
For example the rewrite rules should be able to handle :
Any help I can get on this I would appreciate as my knowledge in IIS URL Rewrite is pretty limited. I did find this post (http://forums.iis.net/t/1185938.aspx/1) which may help a bit.
I found a solution for Apache URL rewrite but it fails in the import included with IIS. It says none of the rules were converted due to the control flow flags (C,S,N) not being supported. Here is the Apache version of what I need:
# If THE_REQUEST contains a URL-path with a percent-encoded "?" and/or a query string with one
# or more specific percent-encoded characters, and we're not already in the process of fixing
# it, then copy the client-requested URL-path-plus-query-string into the "MyURI" variable.
RewriteCond %{ENV:MyURI}>%{THE_REQUEST} ^>[A-Z]+\ /([^\ ]+)\ HTTP/
RewriteCond %1 ^([^?]*\?([^%]*(\%(25)*([^3].|.[^D]))*)*\%(25)*3D.*)$ [NC,OR]
RewriteCond %1 ^([^?]*\?([^%]*(\%(25)*([^2].|.[^6]))*)*\%(25)*26.*)$ [OR]
RewriteCond %1 ^(([^%]*(\%(25)*([^3].|.[^F]))*)*\%(25)*3F.*)$ [NC]
RewriteRule ^. - [NE,E=MyURI:%1]
#
# If any encoded question mark is present in the client-requested URI, and
# no unencoded question mark is present, replace the first encoded question
# mark, queue up a redirect, and then re-start mod_rewrite processing
RewriteCond %{ENV:MyURI} ^[^?]+$
RewriteCond %{ENV:MyURI} ^(([^%]*(\%(25)*([^3].|.[^F]))*)*)\%(25)*3F(.*)$ [NC]
RewriteRule ^. - [NE,E=MyURI:%1?%7,E=QRedir:Yes,N]
#
# If any encoded "=" sign follows the "?", replace it, queue
# up a redirect, and re-start mod_rewrite processing
RewriteCond %{ENV:MyURI} ^([^?]*\?([^%]*(\%(25)*([^3].|.[^D]))*)*)\%(25)*3D(.*)$ [NC]
RewriteRule ^. - [NE,E=MyURI:%1=%7,E=QRedir:Yes,N]
#
# If any encoded ampersand follows the "?", replace it, queue
# up a redirect, and then re-start mod_rewrite processing
RewriteCond %{ENV:MyURI} ^([^?]*\?([^%]*(\%(25)*([^2].|.[^6]))*)*)\%(25)*26(.*)$
RewriteRule ^. - [NE,E=MyURI:%1&%7,E=QRedir:Yes,N]
#
# If we get here, there are no more percent-encoded characters which can
# and should be replaced by the rules above, so do the external redirect
RewriteCond %{ENV:QRedir} =Yes [NC]
RewriteRule ^. http://www.example.com/%{ENV:MyURI} [NE,R=301,L]
With no one answering this questions, is this not really possible to do with the url rewrite module?
I am also surprised this question hasn't been asked before. Other people must have inbound links (as reported in google webmaster tools) that contain entities.
IIS should take care of that already. For example, I setup a binding on a test site to localtest.me and visited the following page (that doesn't exist).
http://localtest.me/index.htm%3Fpage%3Dtest
The error message shows that the requested URL is http://localtest.me:80/index.htm?page=test
So as far as URL Rewrite and IIS are concerned, it's already good. Is it possible that you have double encoding? Can you provide a further example of what's occuring in your situation?
On Firefox I get a 404 error with my url that contains entities. In the Detailed Error Information it does show the Requested URL and it is converted correctly (http://www.mysite.com:80/index.php?page=test)
and if I copy and paste it into the browser it works.
Interestingly enough when I tried the url with entites in IE I got the following message:
Web server.
The URL contains a typographical error.
A custom filter or module, such as URLScan, restricts access to the file.
Things you can try:
Create the content on the Web server.
Review the browser URL.
Create a tracing rule to track failed requests for this HTTP status code and see which module is calling SetStatus. For more information about creating a tracing rule for failed requests, click here.
Links and More Information
This error means that the file or directory does not exist on the server. Create the file or directory and try the request again.
I haven’t changed much with the existing IIS7 install besides adding PHP. I also added the URL rewrite module (it currently just rewrites any
http://mysite.com to
http://www.mysite.com. Could something like URLScan (which I haven’t touched) be causing my problems? Could you point me in the right direction to find out?
It sounds like something is wrong that needs to be fixed, but I would need to be able to repro it to be able to track it down. When I paste
http://localtest.me/index.php%3Fpage%3Dtest in the address bar of IE, Firefox, or Chrome, it performs like what happens for you in Firefox. Is that exactly what you did for IE when it gave that error,
or did you do another step, like follow a link from another page?
If you haven't installed URLScan then it shouldn't be in play, but possibly request filtering is kicking in.
What happens if you disable your non-www to www redirect rule to see if it's on the 2nd redirect that this occurs? (just as a test to track down the cause)
Just a quick note to help you here. The reason you are getting a 404 is because IIS does not properly map your .php extension to the PHP handler mapping you have set up.
While IIS does decode your URL (and shows it correctly to you in the 404 detailed error or the FRT trace if you got one), it seems that this is not sufficient for the handler mapping to occurr.
One possible suggestion: intercept such URLs using a custom module (or Begin_Request handler in global.asax), fix and TransferRequest there. E.g. something like:
Just a quick note to help you here. The reason you are getting a 404 is because IIS does not properly map your .php extension to the PHP handler mapping you have set up.
So checking further the detailed error report shows (it looks like you are right):
I also got the failed request log file for it. Any place I should look in particular in the log for more information on this?
When I installed PHP 5 (FASTCGI) a year or so ago I let the install create the mappings. The error indicates that the static file mapping is being used instead of the *.php mapping? Does anyone know why this could be happening?
What happens if you disable your non-www to www redirect rule to see if it's on the 2nd redirect that this occurs? (just as a test to track down the cause)
I have now tried this on Microsoft Server 2012 and I still get a 404 error. I guess it is not an isolated case.
OWScott can you confirm it actually works with a real page? On my test with 2 different servers (one with IIS7 and the other with IIS8) the urls never get handled by php.
Is this possible to overcome with URL rewriting in IIS?
JesterMagic
9 Posts
Need help transforming %3F into ? and %3D into =
Nov 26, 2012 06:09 PM|LINK
I have a number of inbound links that contain entities which result in 404 errors. I would like to create some url rewrite rules (IIS7) to take care of this. These rules need to work on any page since some of the links that contain entities could be several folders deep and be a file other that index.php.
For example the rewrite rules should be able to handle :
http://www.mysite.com/index.php%3Fpage%3Dtest
http://www.mysite.com/folder/somepage.php%3Fvariable%3Dtest
and convert them into:
http://www.mysite.com/index.php?page=test
http://www.mysite.com/folder/somepage.php?variable=test
Any help I can get on this I would appreciate as my knowledge in IIS URL Rewrite is pretty limited. I did find this post (http://forums.iis.net/t/1185938.aspx/1) which may help a bit.
Thanks for any replies
JesterMagic
9 Posts
Re: Need help transforming %3F into ? and %3D into =
Nov 27, 2012 01:31 PM|LINK
I found a solution for Apache URL rewrite but it fails in the import included with IIS. It says none of the rules were converted due to the control flow flags (C,S,N) not being supported. Here is the Apache version of what I need:
# If THE_REQUEST contains a URL-path with a percent-encoded "?" and/or a query string with one # or more specific percent-encoded characters, and we're not already in the process of fixing # it, then copy the client-requested URL-path-plus-query-string into the "MyURI" variable. RewriteCond %{ENV:MyURI}>%{THE_REQUEST} ^>[A-Z]+\ /([^\ ]+)\ HTTP/ RewriteCond %1 ^([^?]*\?([^%]*(\%(25)*([^3].|.[^D]))*)*\%(25)*3D.*)$ [NC,OR] RewriteCond %1 ^([^?]*\?([^%]*(\%(25)*([^2].|.[^6]))*)*\%(25)*26.*)$ [OR] RewriteCond %1 ^(([^%]*(\%(25)*([^3].|.[^F]))*)*\%(25)*3F.*)$ [NC] RewriteRule ^. - [NE,E=MyURI:%1] # # If any encoded question mark is present in the client-requested URI, and # no unencoded question mark is present, replace the first encoded question # mark, queue up a redirect, and then re-start mod_rewrite processing RewriteCond %{ENV:MyURI} ^[^?]+$ RewriteCond %{ENV:MyURI} ^(([^%]*(\%(25)*([^3].|.[^F]))*)*)\%(25)*3F(.*)$ [NC] RewriteRule ^. - [NE,E=MyURI:%1?%7,E=QRedir:Yes,N] # # If any encoded "=" sign follows the "?", replace it, queue # up a redirect, and re-start mod_rewrite processing RewriteCond %{ENV:MyURI} ^([^?]*\?([^%]*(\%(25)*([^3].|.[^D]))*)*)\%(25)*3D(.*)$ [NC] RewriteRule ^. - [NE,E=MyURI:%1=%7,E=QRedir:Yes,N] # # If any encoded ampersand follows the "?", replace it, queue # up a redirect, and then re-start mod_rewrite processing RewriteCond %{ENV:MyURI} ^([^?]*\?([^%]*(\%(25)*([^2].|.[^6]))*)*)\%(25)*26(.*)$ RewriteRule ^. - [NE,E=MyURI:%1&%7,E=QRedir:Yes,N] # # If we get here, there are no more percent-encoded characters which can # and should be replaced by the rules above, so do the external redirect RewriteCond %{ENV:QRedir} =Yes [NC] RewriteRule ^. http://www.example.com/%{ENV:MyURI} [NE,R=301,L]JesterMagic
9 Posts
Re: Need help transforming %3F into ? and %3D into =
Nov 28, 2012 02:00 PM|LINK
With no one answering this questions, is this not really possible to do with the url rewrite module?
I am also surprised this question hasn't been asked before. Other people must have inbound links (as reported in google webmaster tools) that contain entities.
OWScott
286 Posts
Re: Need help transforming %3F into ? and %3D into =
Nov 28, 2012 02:12 PM|LINK
IIS should take care of that already. For example, I setup a binding on a test site to localtest.me and visited the following page (that doesn't exist).
http://localtest.me/index.htm%3Fpage%3Dtest
The error message shows that the requested URL is http://localtest.me:80/index.htm?page=test
So as far as URL Rewrite and IIS are concerned, it's already good. Is it possible that you have double encoding? Can you provide a further example of what's occuring in your situation?
Microsoft MVP - IIS
Vaasnet
www.vaasnet.com
JesterMagic
9 Posts
Re: Need help transforming %3F into ? and %3D into =
Nov 28, 2012 03:54 PM|LINK
Hi Scott,
Thanks for replying.
On Firefox I get a 404 error with my url that contains entities. In the Detailed Error Information it does show the Requested URL and it is converted correctly (http://www.mysite.com:80/index.php?page=test) and if I copy and paste it into the browser it works.
Interestingly enough when I tried the url with entites in IE I got the following message:
Web server.
The URL contains a typographical error.
A custom filter or module, such as URLScan, restricts access to the file.
Things you can try:
Create the content on the Web server.
Review the browser URL.
Create a tracing rule to track failed requests for this HTTP status code and see which module is calling SetStatus. For more information about creating a tracing rule for failed requests, click here.
Links and More Information
This error means that the file or directory does not exist on the server. Create the file or directory and try the request again.
I haven’t changed much with the existing IIS7 install besides adding PHP. I also added the URL rewrite module (it currently just rewrites any http://mysite.com to http://www.mysite.com. Could something like URLScan (which I haven’t touched) be causing my problems? Could you point me in the right direction to find out?
Thanks
Tom
OWScott
286 Posts
Re: Need help transforming %3F into ? and %3D into =
Nov 28, 2012 04:09 PM|LINK
Hi Tom,
It sounds like something is wrong that needs to be fixed, but I would need to be able to repro it to be able to track it down. When I paste http://localtest.me/index.php%3Fpage%3Dtest in the address bar of IE, Firefox, or Chrome, it performs like what happens for you in Firefox. Is that exactly what you did for IE when it gave that error, or did you do another step, like follow a link from another page?
If you haven't installed URLScan then it shouldn't be in play, but possibly request filtering is kicking in.
What happens if you disable your non-www to www redirect rule to see if it's on the 2nd redirect that this occurs? (just as a test to track down the cause)
Scott
Microsoft MVP - IIS
Vaasnet
www.vaasnet.com
mvolo
629 Posts
Re: Need help transforming %3F into ? and %3D into =
Nov 28, 2012 04:32 PM|LINK
Hi JesterMagic,
Just a quick note to help you here. The reason you are getting a 404 is because IIS does not properly map your .php extension to the PHP handler mapping you have set up.
While IIS does decode your URL (and shows it correctly to you in the 404 detailed error or the FRT trace if you got one), it seems that this is not sufficient for the handler mapping to occurr.
One possible suggestion: intercept such URLs using a custom module (or Begin_Request handler in global.asax), fix and TransferRequest there. E.g. something like:
web.config:
<system.webServer> <modules runAllManagedModulesForAllRequests="true" /> </system.webServer>global.asax:
<%@ Application Language="C#" %> <%@ Import Namespace="System.Diagnostics" %> <script runat="server"> void Application_BeginRequest(object sender, EventArgs e) { HttpApplication app = (HttpApplication)sender; HttpContext context = app.Context; // Detect and rewrite here } </script>The problem with this approach is that ASP.NET will also refuse to serve this URL as is because it contains invalid characters.
You can attempt to work around this using info here: http://adorr.net/2010/01/configure-iis-to-accept-url-with-special-characters.html.
Let us know if this works out, I am curious myself.
Best,
CTO at LeanSentry
Former IIS/ASP.NET PM
Want to become an expert at monitoring and troubleshooting your IIS applications?
See the demo at www.leansentry.com!
JesterMagic
9 Posts
Re: Need help transforming %3F into ? and %3D into =
Nov 28, 2012 06:17 PM|LINK
So checking further the detailed error report shows (it looks like you are right):
Module IIS Web Core
Notification MapRequestHandler
Handler StaticFile
Error Code 0x80070002
I also got the failed request log file for it. Any place I should look in particular in the log for more information on this?
When I installed PHP 5 (FASTCGI) a year or so ago I let the install create the mappings. The error indicates that the static file mapping is being used instead of the *.php mapping? Does anyone know why this could be happening?
The same thing happens.
murtaza_t
484 Posts
Re: Need help transforming %3F into ? and %3D into =
Dec 03, 2012 09:53 AM|LINK
Hello jester,
I just replied your post at:
http://forums.iis.net/p/1193377/2036269.aspx
Regards,
Murtaza
Web Hosting UK | VPS Hosting
JesterMagic
9 Posts
Re: Need help transforming %3F into ? and %3D into =
Dec 05, 2012 05:51 PM|LINK
Thanks murtaza_t, I replied to your other post.
I have now tried this on Microsoft Server 2012 and I still get a 404 error. I guess it is not an isolated case.
OWScott can you confirm it actually works with a real page? On my test with 2 different servers (one with IIS7 and the other with IIS8) the urls never get handled by php.
Is this possible to overcome with URL rewriting in IIS?