This nasty problem confused the heck out of me. Luckily I wasn't the only one. James Saull ran into it on a machine with SharePoint installed (same as me). Removing the virtual directory (or whole Virtual Server) from the SharePoint Application Pool got me 1/2 way there. Last trick (as James mentioned in his blog), was to add the following to my web.config file:
<trust level="Full" originUrl="" />
Wacked!
