I verified all my settings were still in place, then performed an iisreset. After ther reset, I was indeed receiving the same 401 error denying access. I verified this in a new browser process as well. I re-added "NT Authority\Authenticated Users" back to the Users group and iisreset again, but I was still being denied access.
I performed a few more iisreset's and test hits to the site in confusion, and suddenly I was receiving a generic "page not found" error. I checked IIS and my test Web site was gone. applicationHost.config had no entries pertaining to the site.
I rebooted the machine, re-created the Web site from scratch, including a new physical home directory, and performed these tests again from step 1.
Now I'm not able to reproduce the behavior you mentioned. I am granted access to the site even after removing "NT Authority\Authenticated Users" from the Users group and performing an iisreset. Removing the Users group from the home directory ACL still denies access without an iisreset, and can be undone also without an iisreset. I verified the authentication is still anonymous/"IUSR" on the site.
Any ideas what's going on?