If you are creating a setup for your Visual Studio add-in, chances are that you don’t want running instances of Visual Studio when the add-in is going to be installed or uninstalled. For my MZ-Tools add-in, I use the free InnoSetup, which is script-based and uses Pascal for custom code, and I wrote an article some time ago about how to create a setup for a Visual Studio add-in using InnoSetup.

A couple of weeks ago I noticed something strange: when running the setup on Windows Vista, it detected running instances of Visual Studio .NET 2002, 2003 and 2005, but not of Visual Studio 2008. Actually, I detected the problem on Windows 2008 Server with the Visual Studio 2010 CTP, but I was able to reproduce the problem with Windows Vista and Visual Studio 2008.

The setup detects running instances of Visual Studio by using the GetActiveOleObject function for the ProgID of each Visual Studio version (“VisualStudio.DTE.7”, etc.):

function IsVSRunningByProgID(ProgID: String):Boolean;
var
   IDE: Variant;
begin

   try
      IDE := GetActiveOleObject(ProgID);
   except
   end;

   if VarIsEmpty(IDE) then
      Result := False
   else
      Result := True
end;

The GetActiveOleObject function returns a running (if any) COM (ActiveX) server given a ProgId. Somehow it was failing for Visual Studio 2008 whose ProgId is “VisualStudio.DTE.9.0”. I tried to reproduce the problem with a VB6 program using the GetObject function, and it worked, but I soon realized that the GetObject function of VB6 always return an instance: if none is running, it creates one.

Incidentally, the setup worked as expected on Windows XP so I suspected that the User Account Control (UAC) of Windows Vista (and Windows Server 2008) could be the culprit. And the final detail was that while Visual Studio .NET 2002, 2003 and 2005 require by default admin rights (so you get a “consent prompt” from the UAC of Windows Vista), Visual Studio 2008 was designed to run without admin rights, so it launches without “consent prompts” on Windows Vista, running as a standard user even if you are an administrator. Searching on the InnoSetup forums, I confirmed that on Windows Vista a process running with elevated priviledges (such as my setup) cannot interact with a COM process running with low priviledges (such as Visual Studio 2008), so the GetActiveOleObject call fails (“Operation unavailable”) and no instance is returned, as if no instance was executing. In fact, the setup worked as expected if I run Visual Studio 2008 as an administrator (using the context menu) and getting the “consent prompt” of the UAC.

So, which is the solution for this problem? The forums mentioned something about mutexes or something that I am not willing to investigate, because sometimes when you devote a lot of time to a tough problem you realize that you don’t have a problem actually, or that it is not so severe. For example, an add-in can be installed perfectly even with Visual Studio running, it just needs to copy some files and add some registry entries. Its toolbar and menus won’t appear in the running instances of Visual Studio but if the user goes to the Add-In Manager, it will be there. There is only a problem when uninstalling the add-in, because removing the commands (and permanent UI if used) requires to persist the changes and if there is a running instance of Visual Studio the changes will be overwritten when this other instance is closed. But I guess I’ll have to live with that.

For applications that are not setups, such as applications automating the IDE (or for setups that allow you to execute .NET code), you can use System.Diagnostics.Process.GetProcessByName(“devenv”) and it will work.