I have a few suggestions that may help.
As stated in the first section of our FAQ, http://www.rtr.com/fpse/faq1.htm#users, the FrontPage Server Extensions must be installed and executed using the native Administrator account on both Windows Server 2008 and Vista. The domain administrator and local administrator accounts may not be used to install the FrontPage Server Extensions.
A simple workaround to that would be to uninstall FPSE, open a command prompt using the "Run as administrator" option, then install FPSE from the command-line using the msiexec /I <fpse MSI package> syntax.
Also, in the 64 bit environment, AppPool must be configured for 32-bit operation.
I hope that this helps you.
If you would like or need further assistance, we would be happy to open a support contract with you. I refer you to our Technical Support page at http://www.rtr.com/fpse/technicalsupport.htm for additional information concerning types of support and support rates.
Thank you for your use of the FrontPage Server Extensions.
FrontPage Server Extensions Support Team
Ready-to-Run Software, Inc.