Hi,
If you selected URL, the custom error URL must exist in the application pool that directs the request to the custom error URL. If the page your requested encounter a 401 or 403 error, the custom error page also encounter the same error. Then, there would be a loop. Hence, you can set only File or Default for these specific errors.
Since you set custom error page as File, the custom error page will be served as a static file. In this scenario, the .aspx page was processed as an XML file. The workaround is set simple html file for the specific errors and set URL for the others.