I have updated the post of yesterday about the strange case of Visual Studio 2008 severely crashing loading an add-in with more information, since finally I was able to reproduce it.

Some months ago a customer of my MZ-Tools add-in reported a crash in Visual Studio 2008 Team Suite when loading it. When an add-in crashes loading it, usually Visual Studio captures the exception or COM error and shows a semi-friendly messagebox indicating that the add-in has caused an exception or failed to load, and give you the chance of removing it. While sometimes the error is <Unknown Error> and you get only the error number (specially if the problem is found by the CLR loading the add-in and not executing the code), this crash was more severe: “Visual Studio needs to close”, and your only options are to close or to restart. MZ-Tools doesn’t have instrumentation yet, but after a couple of custom builds with traces sent to the customer I was able to determine that the problem happened creating the very first toolbar. Somehow the customer changed to Visual Studio 2008 Professional Edition and the problem disappeared so I closed the incident (falsely).

Last week it was a prospect who experienced the severe crash. It is not the best experience for a prospect to download a trial version of your product, install it and get such a crash. I knew that for such severe crashes the problem is not really my add-in but the prospect would think otherwise… Since the add-in is marked to load on startup, it crashes VS every time so chances are that he will uninstall and forget the product. But fortunately he reported it to me, so I troubleshooted it. After a couple of days verifying that the problem was actually MZ-Tools and not other extensions, that the referenced assemblies were present and that the versions were correct, I recalled this problem, that I had almost totally forgotten. I retreived the e-mails of the previous case, and the problem happened with VS 2008 Team Suite also this time. And when I was about to send him a minimal add-in that creates a toolbar to reproduce the problem, I searched the web and I found this post by Jamie Cansdale, former MVP colleague and author of TestDriven.NET (formerly NUnitAddIn):

VS 2008 crashes on startup when ‘Code Analysis Tools’ feature is not installed
http://weblogs.asp.net/nunitaddin/archive/2007/11/29/vs-2008-crashes-on-startup-when-code-analysis-tools-feature-is-not-installed.aspx

So, the problem happens in Visual Studio 2008 Team Suite if you don’t install the “Code Analysis Tools”. Since in the first case the user switched to Visual Studio 2008 Professional Edition, which doesn’t include the Code Analysis Tools (only Visual Studio 2008 Development Edition or Visual Studio 2008 Team Suite), the problem disappeared. In the second case, the prospect run the setup again and certainly the Code Analysis Tools were not installed. After installing them, MZ-Tools loaded fine.

Now, I have tried to reproduce the problem to report it to Microsoft to no avail:

My computer has Visual Studio 2008 Team Suite with the Code Analysis Tools and SP1. I have removed the Code Analysis Tools and the problem doesn’t appear. I have removed the SP1 and nothing. I have removed VS 2008 completely and I have installed it from scratch without the Code Analysis Tools and the problem doesn’t appear. So, I don’t know yet how to reproduce but at least I am glad that I know how to solve it and hopefully I won’t forget it again (I’ll force myself to check the “The strange case of …” series for those rare problems…)

I have also updated this article:

HOWTO: Troubleshooting Visual Studio and Office add-ins
http://www.mztools.com/articles/2007/MZ2007009.aspx

UPDATE (Dec 11, 2008): Finally I have reproduced the problem with a clean virtual machine and Visual Studio 2008 Development Edition without the “Code Analysis Tools” and without installing SP1. The problem happens when an add-in performs certain operations with commandbars of DTE such as:

• Trying to create a toolbar
• Trying to retrieve an nonexistent command bar
• Trying to set the Name property of a Commandbar
• Maybe others

The problem can be solved through two ways:

• Installing the missing “Code Analysis Tools” component
• Installing Service Pack 1

While not many users are affected by this bug since most of them perform a full installation of Visual Studio 2008 and/or install SP1, the problem is so nasty that you may want to detect this circumstance and show a warning when loading your add-in. Something like the following in the OnConnection method before manipulating CommandBars:

If IsVS2008TeamSuite() Or IsVS2008DevelopmentEdition() Then

   If Not IsCodeAnalysisToolsComponentInstalled() Then

      If Not IsVS2008SP1Installed() Then

         MessageBox.Show("This add-in is going to crash :-(")

      End If

   End If

End If

Some details to implement those functions:

• You can detect if VS 2008 SP1 is installed using the registry key:

– HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DevDiv\VSServicing\9.0\VSTS\<locale>, name “SP” for VS 2008 Team Suite

– HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DevDiv\VSServicing\9.0\VSTD<locale>, name “SP” for VS 2008 Development Edition

Where <locale> is the language of Visual Studio, such as 1033 for English, 3082 for Spanish, etc. Your add-in can get the locale of IDE using the DTE.LocaleID property.

The SP value can be 0 (no SP installed) or 1 (SP1 installed).

• You can detect if VS 2008 Team Suite installed if the registry key “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\Setup\VS\VSTS” exists.
• You can detect if VS 2008 Development Edition installed if the registry key “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\Setup\VS\VSTD” exists.
• You can detect if the “Code Analysis Tools” component is installed getting the folder of VS 2008 (reading the “ProductDir” name under the registry key above), appending “Team ToolsStatic Analysis ToolsFxCop” and checking if the file FxCopSdk.dll exists or not.

Notes:

• VS 2008 without SP1 lacks the folder “Team ToolsStatic Analysis ToolsFxCop” if Code Analysis Tools are not installed (as expected).
• VS 2008 SP1 creates that folder even if you don’t have Code Analysis Tools installed, but it doesn’t add DLLs, just some RepositoryCompatibility folder with some XML files.

Visual Studio preserves the visibility state of its own toolwindows (Solution Explorer, etc.) but not of toolwindows created by add-ins. It is up to the add-in to store the state of its toolwindows when unloaded, and restore them when it is loaded again. So, some add-ins can show their toolwindows when loaded and not only when the menu to show them is invoked. However, some developers are noticing that in some cases toolwindows don’t show when the add-in is loaded on startup, but they show if the add-in is loaded by hand using the Add-In Manager. If this is the case, chances are that it is because you are not using the OnConnection method correctly and the IDE is not fully initialized when the add-in is trying to show its toolwindow:

HOWTO: Use correctly the OnConnection method of a Visual Studio add-in
http://www.mztools.com/articles/2008/MZ2008004.aspx

This is just one of the many side effects of a badly designed IDTExtensibility2 interface that populates the MSDN forum with related questions every week…

If you are creating a setup for your Visual Studio add-in, chances are that you don’t want running instances of Visual Studio when the add-in is going to be installed or uninstalled. For my MZ-Tools add-in, I use the free InnoSetup, which is script-based and uses Pascal for custom code, and I wrote an article some time ago about how to create a setup for a Visual Studio add-in using InnoSetup.

A couple of weeks ago I noticed something strange: when running the setup on Windows Vista, it detected running instances of Visual Studio .NET 2002, 2003 and 2005, but not of Visual Studio 2008. Actually, I detected the problem on Windows 2008 Server with the Visual Studio 2010 CTP, but I was able to reproduce the problem with Windows Vista and Visual Studio 2008.

The setup detects running instances of Visual Studio by using the GetActiveOleObject function for the ProgID of each Visual Studio version (“VisualStudio.DTE.7”, etc.):

function IsVSRunningByProgID(ProgID: String):Boolean;
var
   IDE: Variant;
begin

   try
      IDE := GetActiveOleObject(ProgID);
   except
   end;

   if VarIsEmpty(IDE) then
      Result := False
   else
      Result := True
end;

The GetActiveOleObject function returns a running (if any) COM (ActiveX) server given a ProgId. Somehow it was failing for Visual Studio 2008 whose ProgId is “VisualStudio.DTE.9.0”. I tried to reproduce the problem with a VB6 program using the GetObject function, and it worked, but I soon realized that the GetObject function of VB6 always return an instance: if none is running, it creates one.

Incidentally, the setup worked as expected on Windows XP so I suspected that the User Account Control (UAC) of Windows Vista (and Windows Server 2008) could be the culprit. And the final detail was that while Visual Studio .NET 2002, 2003 and 2005 require by default admin rights (so you get a “consent prompt” from the UAC of Windows Vista), Visual Studio 2008 was designed to run without admin rights, so it launches without “consent prompts” on Windows Vista, running as a standard user even if you are an administrator. Searching on the InnoSetup forums, I confirmed that on Windows Vista a process running with elevated priviledges (such as my setup) cannot interact with a COM process running with low priviledges (such as Visual Studio 2008), so the GetActiveOleObject call fails (“Operation unavailable”) and no instance is returned, as if no instance was executing. In fact, the setup worked as expected if I run Visual Studio 2008 as an administrator (using the context menu) and getting the “consent prompt” of the UAC.

So, which is the solution for this problem? The forums mentioned something about mutexes or something that I am not willing to investigate, because sometimes when you devote a lot of time to a tough problem you realize that you don’t have a problem actually, or that it is not so severe. For example, an add-in can be installed perfectly even with Visual Studio running, it just needs to copy some files and add some registry entries. Its toolbar and menus won’t appear in the running instances of Visual Studio but if the user goes to the Add-In Manager, it will be there. There is only a problem when uninstalling the add-in, because removing the commands (and permanent UI if used) requires to persist the changes and if there is a running instance of Visual Studio the changes will be overwritten when this other instance is closed. But I guess I’ll have to live with that.

For applications that are not setups, such as applications automating the IDE (or for setups that allow you to execute .NET code), you can use System.Diagnostics.Process.GetProcessByName(“devenv”) and it will work.