I'm having the exact same problem, but it seems to be limited to virtual sites that are using host headers as opposed to listening on a unique IP address. Such as...
1. In IIS Manager, right-click Web Sites -> New -> Web Site
2. Fill out info & listen on unique IP address (already configured in TCP/IP network properties). Point to a new root web directory..
3. All Programs -> Administrative Tools -> Microsoft SharePoint Administrator (http://localhost:6109/fpadmdll.dll)
4. Click Extend to create FrontPage Server Extensions 2002 on the new virtual server.
5. Set an admin user (HOSTNAME\Administrator).
6. Click the new virtual server via SharePoint admin page.. links to:
http://<website-url>/_vti_bin/_vti_adm/fpadmdll.dll?page=webadmin.htm
This page prompts for UserName/Password. Enter HOSTNAME\Administrator & password. Displays website administration page (Works fine).
When I repeat the same steps using host header instead of a unique IP at Step 2, instead of properly displaying the admin config page at step 6, I keep getting the prompt to enter Username/Password 3 times until it finally results in:
HTTP Error 401.1 - Unauthorized: Access is denied due to invalid
credentials.
----------
NOTE: This only happens if I attempt to run the SharePoint Administration from the the host the web sites are installed on. Accessing the virtual web site administration from a remote host works fine:
http://<website-url>/_vti_bin/_vti_adm/fpadmdll.dll?page=webadmin.htm
I can only guess the SharePoint is having trouble resolving the use of the host headers.