Here is another frustration when working with the automation model of Visual Studio: suppose that you want to make an add-in that is loaded before a command-line build (for example using devenv.exe mysolution.sln /build Debug) to perform some checkings that, if failed, should abort the build. Quite reasonable, isn’t it?. So, how do you approach this? Any programmer familiar with the automation model knows that the add-in wizard offers an option to create command-line add-ins and that the OnConnection method has a connectMode = ext_ConnectMode.ext_cm_CommandLine value, so that seems the way to go. But it happens that:

• VS.NET 2002 does never receive that value, even when you launch a build from the command line. I reported this bug long time ago: BUG: ext_ConnectMode.ext_cm_CommandLine flag not passed to Visual Studio .NET 2002 add-ins.
• VS.NET 2003 fixed that bug, at least apparently. Further experimentation has shown that if the solution to build is very small (for example, the default ClassLibrary project created by VS) then the build is performed without the add-in loaded !!! If the solution is bigger so that the compilation is not so immediate, then the add-in is loaded (you can test all this if you create a simple add-in with a messagebox in the OnConnection and OnDisconnection methods). Apparently it seems that add-ins are not loaded before the build begins but some time later, maybe in some parallel fashion. This, of course, ruins this approach.
• VS 2005 has a somewhat different bug and the add-in is never loaded, no matter the size of the solution. I reported this here yesterday:

https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=303803

(UPDATE August 17, 2008: Microsoft released a hotfix for this problem after VS 2005 SP1: FIX: An command-line add-in does not load when you start the Visual Studio 2005 IDE)

• VS 2008 (Beta 2) seems to fix all the problems of all predecessors and it loads the add-in even for builds of small solutions (I have not tested it very much, so I could be wrong)
• Bets are open about the behavior in the future VS 2010 or whatever new version… 😉

Well, let’s assume that the behavior is correct and the add-in gets loaded. How do you write some results to the log file or the console output that Visual Studio uses (with the /out switch)? While I was able to write to the Output window when using the full IDE, I was unable to write to the log file. After some searching it seems that Console.Write should do the trick:

http://www.dotnet247.com/247reference/msgs/30/151588.aspx

But I was unable to succeed. Maybe someone from Microsoft can verify if this indeed works…

OK, the add-in can write to a different log file, no big deal. Now, how can the add-in cancel the build if it needs to? It happens that, in an apparent terrible oversight, the Build events lack a cancel parameter:

HOWTO: Canceling a build from a Visual Studio .NET add-in.
http://www.mztools.com/articles/2005/MZ2005010.aspx

No big deal because you can use DTE.ExecuteCommand(“Build.Cancel”)…. Alas, when the IDE is run for a command-line build, commands are not created.

Then you can try some ugly TerminateProcess API call, but it seems that it won’t work properly…

So, why on earth does the automation model provide a command-line connectMode that, when you are lucky enough that your VS version honors it, is basically useless? It would be nice if the Microsoft people could address all this in the automation model (I’m sure that an SDK package can handle all this OK)…

Well, after some relaxing time, I got a new idea. Forget the command-line add-in. Let’s try another approach. I don’t really care about the build. I just want some command-line utility to automate Visual Studio to perform a review before another command-line utility (devenv.exe) perform the build. If the first utility reports some error, my script does not execute the build. Sometime ago I wrote an article about how to automate Visual Studio from the outside:

HOWTO: Automating Visual Studio .NET from outside the IDE
http://www.mztools.com/articles/2005/MZ2005005.aspx

So, the idea is:

• Create an add-in that offers a public method Sub Review(ByVal logFileName As String) in the Connect class.
• Create a console application that receives a solution file name, examines its format (“7.00”, “8.00”, “9.00”, “10.00”) to guess the Visual Studio ProgID to use (VisualStudio.DTE.7, VisualStudio.DTE.7.1, VisualStudio.DTE.8.0, VisualStudio.DTE.9.0) (don’t ask me why the solution format numbers and the ProgID numbers don’t match), creates the DTE object as the article above explains, iterates its DTE.AddIns collection and locates the add-in by the AddIn.ProgId property, call its Connect = True property to load it, calls DTE.Solution.Open(solutionFileName) to load the solution, gets the AddIn.Object property to get the instance of the class that represents the add-in and then calls its Review(logFileName) method. Sounds feasible, right?

The first problem is that AddIn.Object returns a System.__ComObject that can not be converted to the .NET type of the Connect class of the add-in, even if your console application has a reference to the add-in dll. I am not a COM expert, but since this works with in-process DLLs, it can only be that such casts are not allowed from one process to another. Fortunately, you can call the Review method using late binding (Option Strict Off) in VB.NET, or using InvokeMethod of reflection using C# or VB.NET.

So, basically, I got this working…most of the time :-(. I noticed that from time to time, I get COMExceptions of the kind ‘Call was Rejected By Callee’ (that is, RPC_E_CALL_REJECTED). Ummmhh, more frustation. After some searching, it happens that that’s normal automating COM servers and fortunately the MSDN docs of Visual Studio report this problem and solution:

Fixing ‘Application is Busy’ and ‘Call was Rejected By Callee’ Errors
http://msdn2.microsoft.com/en-us/library/ms228772(VS.80).aspx

Kudos to the Microsoft guy that documented that. I didn’t know about that esoteric IMessageFilter::RetryRejectedCall stuff but it seems to work.

I am posting the code so other add-in developers facing the same problem can report problems or improvements:

The add-in (named CommandLineAddIn) VB.NET code is:

Imports System
Imports Microsoft.VisualStudio.CommandBars
Imports Extensibility
Imports EnvDTE
Imports EnvDTE80

Public Class Connect
   Implements IDTExtensibility2

   Private m_DTE As DTE

   Private Sub OnConnection(ByVal application As Object, ByVal connectMode As ext_ConnectMode, ByVal addInInst As Object, ByRef custom As Array) Implements IDTExtensibility2.OnConnection

      ' MessageBox.Show(connectMode.ToString)

      m_DTE = DirectCast(application, DTE)

   End Sub

   Private Sub OnDisconnection(ByVal disconnectMode As ext_DisconnectMode, ByRef custom As Array) Implements IDTExtensibility2.OnDisconnection

      ' MessageBox.Show(disconnectMode.ToString)

   End Sub

   Public Sub ExecuteReview(ByVal logFullFileName As String)

      Dim logStreamWriter As System.IO.StreamWriter = Nothing

      Try

         logStreamWriter = New System.IO.StreamWriter(logFullFileName, False, System.Text.Encoding.Default)

         If m_DTE.Solution.IsOpen = False Then
            logStreamWriter.WriteLine("No solution loaded!")
         Else
            logStreamWriter.WriteLine("Solution has " & m_DTE.Solution.Projects.Count.ToString & " projects.")
         End If

      Catch ex As Exception

         If Not (logStreamWriter Is Nothing) Then
            logStreamWriter.WriteLine(ex.ToString)
         End If

      Finally

         If Not (logStreamWriter Is Nothing) Then
            logStreamWriter.Close()
            logStreamWriter.Dispose()
         End If

      End Try

   End Sub

   Private Sub OnAddInsUpdate(ByRef custom As Array) Implements IDTExtensibility2.OnAddInsUpdate
   End Sub

   Private Sub OnStartupComplete(ByRef custom As Array) Implements IDTExtensibility2.OnStartupComplete
   End Sub

   Private Sub OnBeginShutdown(ByRef custom As Array) Implements IDTExtensibility2.OnBeginShutdown
   End Sub

End Class

And the command-line utility is:

'---------------------------------------------------------------------------------------------
' Module VSAutomator.vb
'---------------------------------------------------------------------------------------------

Module VSAutomator

   Private Enum VisualStudioVersion
      Unknown = 0
      VSNET2002 = 1
      VSNET2003 = 2
      VS2005 = 3
      VS2008 = 4
   End Enum

   <STAThread()> _
   Sub Main(ByVal args() As String)

      Const ADDIN_PROGID As String = "CommandLineAddIn.Connect"
      Const ADDIN_METHOD As String = "ExecuteReview"

      Dim dte As EnvDTE.DTE = Nothing
      Dim dteType As Type
      Dim commandLineAddIn As CommandLineAddIn.Connect = Nothing
      Dim solutionFullFileName As String
      Dim solutionFolder As String
      Dim solutionName As String
      Dim logFullFileName As String
      Dim connectObject As Object = Nothing
      Dim connectObjectType As Type
      Dim version As VisualStudioVersion
      Dim progID As String
      Dim executableName As String
      Dim addIn As EnvDTE.AddIn
      Dim msgFilter As MessageFilter = Nothing

      Try

         msgFilter = New MessageFilter

         If args.Length = 0 Then
            executableName = IO.Path.GetFileName(System.Reflection.Assembly.GetExecutingAssembly.Location)
            ReportError("Usage: " & executableName & " solution_file_name.sln")
         Else
            solutionFullFileName = args(0)

            If Not IO.File.Exists(solutionFullFileName) Then
               ReportError("Solution file '" & solutionFullFileName & "' does not exist.")
            Else

               solutionFolder = IO.Path.GetDirectoryName(solutionFullFileName)
               solutionName = IO.Path.GetFileNameWithoutExtension(solutionFullFileName)
               logFullFileName = IO.Path.Combine(solutionFolder, solutionName & ".log")

               If IO.File.Exists(logFullFileName) Then
                  IO.File.Delete(logFullFileName)
               End If

               version = GetSolutionVersion(solutionFullFileName)

               If version = VisualStudioVersion.Unknown Then
                  ReportError("The format version of the solution file is not supported.")
               Else

                  progID = GetVisualStudioProgID(version)

                  dteType = System.Type.GetTypeFromProgID(progID)

                  If dteType Is Nothing Then
                     ReportError("Could not find the ActiveX Server for ProgID '" & progID & "'. Likely the proper version of Visual Studio is not installed.")
                  Else

                     dte = DirectCast(System.Activator.CreateInstance(dteType), EnvDTE.DTE)
                     dte.SuppressUI = True
                     dte.UserControl = False

                     addIn = GetAddInByProgID(dte, ADDIN_PROGID)

                     If addIn Is Nothing Then
                        ReportError("The Add-in " & ADDIN_PROGID & " was not found in Visual Studio.")
                     Else

                        addIn.Connected = True

                        connectObject = addIn.Object

                        dte.Solution.Open(solutionFullFileName)

                        connectObjectType = connectObject.GetType

                        connectObjectType.InvokeMember(ADDIN_METHOD, Reflection.BindingFlags.InvokeMethod Or Reflection.BindingFlags.Instance Or Reflection.BindingFlags.Public, Nothing, connectObject, New String() {logFullFileName})

                     End If

                  End If

               End If

            End If

         End If

      Catch ex As Exception
         ReportError(ex.ToString)
      Finally

         If Not (dte Is Nothing) Then
            Try
               dte.Quit()
            Catch ex As Exception
            End Try
         End If

         If Not (msgFilter Is Nothing) Then
            msgFilter.Dispose()
         End If

      End Try

   End Sub

   Private Sub ReportError(ByVal msg As String)

#If DEBUG Then
      MsgBox(msg)
#End If

      Console.WriteLine(msg)

   End Sub

   Private Function GetAddInByProgID(ByVal dte As EnvDTE.DTE, ByVal addinProgID As String) As EnvDTE.AddIn

      Dim addinResult As EnvDTE.AddIn = Nothing
      Dim addin As EnvDTE.AddIn

      For Each addin In dte.AddIns

         If addin.ProgID = addinProgID Then

            addinResult = addin
            Exit For

         End If

      Next

      Return addinResult

   End Function

   Private Function GetSolutionVersion(ByVal solutionFullFileName As String) As VisualStudioVersion

      Dim version As VisualStudioVersion = VisualStudioVersion.Unknown
      Dim solutionStreamReader As IO.StreamReader = Nothing
      Dim firstLine As String
      Dim format As String

      Try

         solutionStreamReader = New IO.StreamReader(solutionFullFileName)
         firstLine = solutionStreamReader.ReadLine()

         format = firstLine.Substring(firstLine.LastIndexOf(" ")).Trim

         Select Case format
            Case "7.00"
               version = VisualStudioVersion.VSNET2002
            Case "8.00"
               version = VisualStudioVersion.VSNET2003
            Case "9.00"
               version = VisualStudioVersion.VS2005
            Case "10.00"
               version = VisualStudioVersion.VS2008
         End Select

      Finally

         If Not (solutionStreamReader Is Nothing) Then
            solutionStreamReader.Close()
         End If

      End Try

      Return version

   End Function

   Private Function GetVisualStudioProgID(ByVal version As VisualStudioVersion) As String

      Dim progID As String = ""

      Select Case version
         Case VisualStudioVersion.VSNET2002
            progID = "VisualStudio.DTE.7"
         Case VisualStudioVersion.VSNET2003
            progID = "VisualStudio.DTE.7.1"
         Case VisualStudioVersion.VS2005
            progID = "VisualStudio.DTE.8.0"
         Case VisualStudioVersion.VS2008
            progID = "VisualStudio.DTE.9.0"
      End Select

      Return progID

   End Function

End Module

And the message filter is:

'---------------------------------------------------------------------------------------------
' Module MessageFilter.vb
'---------------------------------------------------------------------------------------------

Imports System.Runtime.InteropServices

' See: Fixing 'Application is Busy' and 'Call was Rejected By Callee' Errors
' http://msdn2.microsoft.com/en-us/library/ms228772(VS.80).aspx

<ComImport(), Guid("00000016-0000-0000-C000-000000000046"), InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)> _
Interface IOleMessageFilter
   <PreserveSig()> _
   Function HandleInComingCall(ByVal dwCallType As Integer, ByVal hTaskCaller As IntPtr, ByVal dwTickCount As Integer, ByVal lpInterfaceInfo As IntPtr) As Integer

   <PreserveSig()> _
   Function RetryRejectedCall(ByVal hTaskCallee As IntPtr, ByVal dwTickCount As Integer, ByVal dwRejectType As Integer) As Integer

   <PreserveSig()> _
   Function MessagePending(ByVal hTaskCallee As IntPtr, ByVal dwTickCount As Integer, ByVal dwPendingType As Integer) As Integer
End Interface

Public Class MessageFilter
   Implements IOleMessageFilter, IDisposable

   <DllImport("Ole32.dll")> _
   Private Shared Function CoRegisterMessageFilter(ByVal newFilter As IOleMessageFilter, ByRef oldFilter As IOleMessageFilter) As Integer
   End Function

   ' Class containing the IOleMessageFilter thread error-handling functions.

   Private Enum SERVERCALL
      SERVERCALL_ISHANDLED = 0
      SERVERCALL_REJECTED = 1
      SERVERCALL_RETRYLATER = 2
   End Enum

   Private Enum PENDINGMSG
      PENDINGMSG_CANCELCALL = 0
      PENDINGMSG_WAITNOPROCESS = 1
      PENDINGMSG_WAITDEFPROCESS = 2
   End Enum

   Private m_oldFilter As IOleMessageFilter
   Private m_disposedValue As Boolean = False

   Public Sub New()

      Dim hr As Integer

      m_oldFilter = Nothing

      hr = CoRegisterMessageFilter(Me, m_oldFilter)
      System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(hr)

   End Sub

   Private Function HandleInComingCall(ByVal dwCallType As Integer, ByVal hTaskCaller As System.IntPtr, ByVal dwTickCount As Integer, ByVal lpInterfaceInfo As System.IntPtr) As Integer Implements IOleMessageFilter.HandleInComingCall

      ' Return the ole default (don't let the call through)
      Return SERVERCALL.SERVERCALL_ISHANDLED

   End Function

   Private Function MessagePending(ByVal hTaskCallee As System.IntPtr, ByVal dwTickCount As Integer, ByVal dwPendingType As Integer) As Integer Implements IOleMessageFilter.MessagePending

      Return PENDINGMSG.PENDINGMSG_WAITDEFPROCESS

   End Function

   Private Function RetryRejectedCall(ByVal hTaskCallee As System.IntPtr, ByVal dwTickCount As Integer, ByVal dwRejectType As Integer) As Integer Implements IOleMessageFilter.RetryRejectedCall

      Dim iResult As Integer

      ' See: IMessageFilter::RetryRejectedCall
      ' http://msdn2.microsoft.com/en-us/library/ms680739.aspx

      ' Return values:
      ' -1: The call should be canceled. COM then returns RPC_E_CALL_REJECTED from the original method call.
      ' Value >= 0 and <100: The call is to be retried immediately.
      ' Value >= 100: COM will wait for this many milliseconds and then retry the call.

      If dwRejectType = SERVERCALL.SERVERCALL_RETRYLATER Then ' Thread call was rejected, so try again.
         iResult = 99 ' Retry immediately
      Else
         ' Too busy; cancel call.
         iResult = -1
      End If

      Return iResult

   End Function

   Protected Overridable Sub Dispose(ByVal disposing As Boolean)

      Dim dummyFilter As IOleMessageFilter
      Dim hr As Integer

      dummyFilter = Nothing

      If Not m_disposedValue Then

         hr = CoRegisterMessageFilter(m_oldFilter, dummyFilter)
         System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(hr)

         m_disposedValue = True

      End If

   End Sub

   Public Sub Dispose() Implements IDisposable.Dispose

      ' Do not change this code.  Put cleanup code in Dispose(ByVal disposing As Boolean) above.
      Dispose(True)
      GC.SuppressFinalize(Me)

   End Sub

End Class