Here is another frustration when working with the automation model of Visual Studio: suppose that you want to make an add-in that is loaded before a command-line build (for example using devenv.exe mysolution.sln /build Debug) to perform some checkings that, if failed, should abort the build. Quite reasonable, isn’t it?. So, how do you approach this? Any programmer familiar with the automation model knows that the add-in wizard offers an option to create command-line add-ins and that the OnConnection method has a connectMode = ext_ConnectMode.ext_cm_CommandLine value, so that seems the way to go. But it happens that:
- VS.NET 2002 does never receive that value, even when you launch a build from the command line. I reported this bug long time ago: BUG: ext_ConnectMode.ext_cm_CommandLine flag not passed to Visual Studio .NET 2002 add-ins.
- VS.NET 2003 fixed that bug, at least apparently. Further experimentation has shown that if the solution to build is very small (for example, the default ClassLibrary project created by VS) then the build is performed without the add-in loaded !!! If the solution is bigger so that the compilation is not so immediate, then the add-in is loaded (you can test all this if you create a simple add-in with a messagebox in the OnConnection and OnDisconnection methods). Apparently it seems that add-ins are not loaded before the build begins but some time later, maybe in some parallel fashion. This, of course, ruins this approach.
- VS 2005 has a somewhat different bug and the add-in is never loaded, no matter the size of the solution. I reported this here yesterday:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=303803
(UPDATE August 17, 2008: Microsoft released a hotfix for this problem after VS 2005 SP1: FIX: An command-line add-in does not load when you start the Visual Studio 2005 IDE)
- VS 2008 (Beta 2) seems to fix all the problems of all predecessors and it loads the add-in even for builds of small solutions (I have not tested it very much, so I could be wrong)
- Bets are open about the behavior in the future VS 2010 or whatever new version… 😉
Well, let’s assume that the behavior is correct and the add-in gets loaded. How do you write some results to the log file or the console output that Visual Studio uses (with the /out switch)? While I was able to write to the Output window when using the full IDE, I was unable to write to the log file. After some searching it seems that Console.Write should do the trick:
http://www.dotnet247.com/247reference/msgs/30/151588.aspx
But I was unable to succeed. Maybe someone from Microsoft can verify if this indeed works…
OK, the add-in can write to a different log file, no big deal. Now, how can the add-in cancel the build if it needs to? It happens that, in an apparent terrible oversight, the Build events lack a cancel parameter:
HOWTO: Canceling a build from a Visual Studio .NET add-in.
http://www.mztools.com/articles/2005/MZ2005010.aspx
No big deal because you can use DTE.ExecuteCommand(“Build.Cancel”)…. Alas, when the IDE is run for a command-line build, commands are not created.
Then you can try some ugly TerminateProcess API call, but it seems that it won’t work properly…
So, why on earth does the automation model provide a command-line connectMode that, when you are lucky enough that your VS version honors it, is basically useless? It would be nice if the Microsoft people could address all this in the automation model (I’m sure that an SDK package can handle all this OK)…
Well, after some relaxing time, I got a new idea. Forget the command-line add-in. Let’s try another approach. I don’t really care about the build. I just want some command-line utility to automate Visual Studio to perform a review before another command-line utility (devenv.exe) perform the build. If the first utility reports some error, my script does not execute the build. Sometime ago I wrote an article about how to automate Visual Studio from the outside:
HOWTO: Automating Visual Studio .NET from outside the IDE
http://www.mztools.com/articles/2005/MZ2005005.aspx
So, the idea is:
- Create an add-in that offers a public method Sub Review(ByVal logFileName As String) in the Connect class.
- Create a console application that receives a solution file name, examines its format (“7.00”, “8.00”, “9.00”, “10.00”) to guess the Visual Studio ProgID to use (VisualStudio.DTE.7, VisualStudio.DTE.7.1, VisualStudio.DTE.8.0, VisualStudio.DTE.9.0) (don’t ask me why the solution format numbers and the ProgID numbers don’t match), creates the DTE object as the article above explains, iterates its DTE.AddIns collection and locates the add-in by the AddIn.ProgId property, call its Connect = True property to load it, calls DTE.Solution.Open(solutionFileName) to load the solution, gets the AddIn.Object property to get the instance of the class that represents the add-in and then calls its Review(logFileName) method. Sounds feasible, right?
The first problem is that AddIn.Object returns a System.__ComObject that can not be converted to the .NET type of the Connect class of the add-in, even if your console application has a reference to the add-in dll. I am not a COM expert, but since this works with in-process DLLs, it can only be that such casts are not allowed from one process to another. Fortunately, you can call the Review method using late binding (Option Strict Off) in VB.NET, or using InvokeMethod of reflection using C# or VB.NET.
So, basically, I got this working…most of the time :-(. I noticed that from time to time, I get COMExceptions of the kind ‘Call was Rejected By Callee’ (that is, RPC_E_CALL_REJECTED). Ummmhh, more frustation. After some searching, it happens that that’s normal automating COM servers and fortunately the MSDN docs of Visual Studio report this problem and solution:
Fixing ‘Application is Busy’ and ‘Call was Rejected By Callee’ Errors
http://msdn2.microsoft.com/en-us/library/ms228772(VS.80).aspx
Kudos to the Microsoft guy that documented that. I didn’t know about that esoteric IMessageFilter::RetryRejectedCall stuff but it seems to work.
I am posting the code so other add-in developers facing the same problem can report problems or improvements:
The add-in (named CommandLineAddIn) VB.NET code is:
Imports System
Imports Microsoft.VisualStudio.CommandBars
Imports Extensibility
Imports EnvDTE
Imports EnvDTE80
Public Class Connect
Implements IDTExtensibility2
Private m_DTE As DTE
Private Sub OnConnection(ByVal application As Object, ByVal connectMode As ext_ConnectMode, ByVal addInInst As Object, ByRef custom As Array) Implements IDTExtensibility2.OnConnection
' MessageBox.Show(connectMode.ToString)
m_DTE = DirectCast(application, DTE)
End Sub
Private Sub OnDisconnection(ByVal disconnectMode As ext_DisconnectMode, ByRef custom As Array) Implements IDTExtensibility2.OnDisconnection
' MessageBox.Show(disconnectMode.ToString)
End Sub
Public Sub ExecuteReview(ByVal logFullFileName As String)
Dim logStreamWriter As System.IO.StreamWriter = Nothing
Try
logStreamWriter = New System.IO.StreamWriter(logFullFileName, False, System.Text.Encoding.Default)
If m_DTE.Solution.IsOpen = False Then
logStreamWriter.WriteLine("No solution loaded!")
Else
logStreamWriter.WriteLine("Solution has " & m_DTE.Solution.Projects.Count.ToString & " projects.")
End If
Catch ex As Exception
If Not (logStreamWriter Is Nothing) Then
logStreamWriter.WriteLine(ex.ToString)
End If
Finally
If Not (logStreamWriter Is Nothing) Then
logStreamWriter.Close()
logStreamWriter.Dispose()
End If
End Try
End Sub
Private Sub OnAddInsUpdate(ByRef custom As Array) Implements IDTExtensibility2.OnAddInsUpdate
End Sub
Private Sub OnStartupComplete(ByRef custom As Array) Implements IDTExtensibility2.OnStartupComplete
End Sub
Private Sub OnBeginShutdown(ByRef custom As Array) Implements IDTExtensibility2.OnBeginShutdown
End Sub
End Class
And the command-line utility is:
'---------------------------------------------------------------------------------------------
' Module VSAutomator.vb
'---------------------------------------------------------------------------------------------
Module VSAutomator
Private Enum VisualStudioVersion
Unknown = 0
VSNET2002 = 1
VSNET2003 = 2
VS2005 = 3
VS2008 = 4
End Enum
<STAThread()> _
Sub Main(ByVal args() As String)
Const ADDIN_PROGID As String = "CommandLineAddIn.Connect"
Const ADDIN_METHOD As String = "ExecuteReview"
Dim dte As EnvDTE.DTE = Nothing
Dim dteType As Type
Dim commandLineAddIn As CommandLineAddIn.Connect = Nothing
Dim solutionFullFileName As String
Dim solutionFolder As String
Dim solutionName As String
Dim logFullFileName As String
Dim connectObject As Object = Nothing
Dim connectObjectType As Type
Dim version As VisualStudioVersion
Dim progID As String
Dim executableName As String
Dim addIn As EnvDTE.AddIn
Dim msgFilter As MessageFilter = Nothing
Try
msgFilter = New MessageFilter
If args.Length = 0 Then
executableName = IO.Path.GetFileName(System.Reflection.Assembly.GetExecutingAssembly.Location)
ReportError("Usage: " & executableName & " solution_file_name.sln")
Else
solutionFullFileName = args(0)
If Not IO.File.Exists(solutionFullFileName) Then
ReportError("Solution file '" & solutionFullFileName & "' does not exist.")
Else
solutionFolder = IO.Path.GetDirectoryName(solutionFullFileName)
solutionName = IO.Path.GetFileNameWithoutExtension(solutionFullFileName)
logFullFileName = IO.Path.Combine(solutionFolder, solutionName & ".log")
If IO.File.Exists(logFullFileName) Then
IO.File.Delete(logFullFileName)
End If
version = GetSolutionVersion(solutionFullFileName)
If version = VisualStudioVersion.Unknown Then
ReportError("The format version of the solution file is not supported.")
Else
progID = GetVisualStudioProgID(version)
dteType = System.Type.GetTypeFromProgID(progID)
If dteType Is Nothing Then
ReportError("Could not find the ActiveX Server for ProgID '" & progID & "'. Likely the proper version of Visual Studio is not installed.")
Else
dte = DirectCast(System.Activator.CreateInstance(dteType), EnvDTE.DTE)
dte.SuppressUI = True
dte.UserControl = False
addIn = GetAddInByProgID(dte, ADDIN_PROGID)
If addIn Is Nothing Then
ReportError("The Add-in " & ADDIN_PROGID & " was not found in Visual Studio.")
Else
addIn.Connected = True
connectObject = addIn.Object
dte.Solution.Open(solutionFullFileName)
connectObjectType = connectObject.GetType
connectObjectType.InvokeMember(ADDIN_METHOD, Reflection.BindingFlags.InvokeMethod Or Reflection.BindingFlags.Instance Or Reflection.BindingFlags.Public, Nothing, connectObject, New String() {logFullFileName})
End If
End If
End If
End If
End If
Catch ex As Exception
ReportError(ex.ToString)
Finally
If Not (dte Is Nothing) Then
Try
dte.Quit()
Catch ex As Exception
End Try
End If
If Not (msgFilter Is Nothing) Then
msgFilter.Dispose()
End If
End Try
End Sub
Private Sub ReportError(ByVal msg As String)
#If DEBUG Then
MsgBox(msg)
#End If
Console.WriteLine(msg)
End Sub
Private Function GetAddInByProgID(ByVal dte As EnvDTE.DTE, ByVal addinProgID As String) As EnvDTE.AddIn
Dim addinResult As EnvDTE.AddIn = Nothing
Dim addin As EnvDTE.AddIn
For Each addin In dte.AddIns
If addin.ProgID = addinProgID Then
addinResult = addin
Exit For
End If
Next
Return addinResult
End Function
Private Function GetSolutionVersion(ByVal solutionFullFileName As String) As VisualStudioVersion
Dim version As VisualStudioVersion = VisualStudioVersion.Unknown
Dim solutionStreamReader As IO.StreamReader = Nothing
Dim firstLine As String
Dim format As String
Try
solutionStreamReader = New IO.StreamReader(solutionFullFileName)
firstLine = solutionStreamReader.ReadLine()
format = firstLine.Substring(firstLine.LastIndexOf(" ")).Trim
Select Case format
Case "7.00"
version = VisualStudioVersion.VSNET2002
Case "8.00"
version = VisualStudioVersion.VSNET2003
Case "9.00"
version = VisualStudioVersion.VS2005
Case "10.00"
version = VisualStudioVersion.VS2008
End Select
Finally
If Not (solutionStreamReader Is Nothing) Then
solutionStreamReader.Close()
End If
End Try
Return version
End Function
Private Function GetVisualStudioProgID(ByVal version As VisualStudioVersion) As String
Dim progID As String = ""
Select Case version
Case VisualStudioVersion.VSNET2002
progID = "VisualStudio.DTE.7"
Case VisualStudioVersion.VSNET2003
progID = "VisualStudio.DTE.7.1"
Case VisualStudioVersion.VS2005
progID = "VisualStudio.DTE.8.0"
Case VisualStudioVersion.VS2008
progID = "VisualStudio.DTE.9.0"
End Select
Return progID
End Function
End Module
And the message filter is:
'---------------------------------------------------------------------------------------------
' Module MessageFilter.vb
'---------------------------------------------------------------------------------------------
Imports System.Runtime.InteropServices
' See: Fixing 'Application is Busy' and 'Call was Rejected By Callee' Errors
' http://msdn2.microsoft.com/en-us/library/ms228772(VS.80).aspx
<ComImport(), Guid("00000016-0000-0000-C000-000000000046"), InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)> _
Interface IOleMessageFilter
<PreserveSig()> _
Function HandleInComingCall(ByVal dwCallType As Integer, ByVal hTaskCaller As IntPtr, ByVal dwTickCount As Integer, ByVal lpInterfaceInfo As IntPtr) As Integer
<PreserveSig()> _
Function RetryRejectedCall(ByVal hTaskCallee As IntPtr, ByVal dwTickCount As Integer, ByVal dwRejectType As Integer) As Integer
<PreserveSig()> _
Function MessagePending(ByVal hTaskCallee As IntPtr, ByVal dwTickCount As Integer, ByVal dwPendingType As Integer) As Integer
End Interface
Public Class MessageFilter
Implements IOleMessageFilter, IDisposable
<DllImport("Ole32.dll")> _
Private Shared Function CoRegisterMessageFilter(ByVal newFilter As IOleMessageFilter, ByRef oldFilter As IOleMessageFilter) As Integer
End Function
' Class containing the IOleMessageFilter thread error-handling functions.
Private Enum SERVERCALL
SERVERCALL_ISHANDLED = 0
SERVERCALL_REJECTED = 1
SERVERCALL_RETRYLATER = 2
End Enum
Private Enum PENDINGMSG
PENDINGMSG_CANCELCALL = 0
PENDINGMSG_WAITNOPROCESS = 1
PENDINGMSG_WAITDEFPROCESS = 2
End Enum
Private m_oldFilter As IOleMessageFilter
Private m_disposedValue As Boolean = False
Public Sub New()
Dim hr As Integer
m_oldFilter = Nothing
hr = CoRegisterMessageFilter(Me, m_oldFilter)
System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(hr)
End Sub
Private Function HandleInComingCall(ByVal dwCallType As Integer, ByVal hTaskCaller As System.IntPtr, ByVal dwTickCount As Integer, ByVal lpInterfaceInfo As System.IntPtr) As Integer Implements IOleMessageFilter.HandleInComingCall
' Return the ole default (don't let the call through)
Return SERVERCALL.SERVERCALL_ISHANDLED
End Function
Private Function MessagePending(ByVal hTaskCallee As System.IntPtr, ByVal dwTickCount As Integer, ByVal dwPendingType As Integer) As Integer Implements IOleMessageFilter.MessagePending
Return PENDINGMSG.PENDINGMSG_WAITDEFPROCESS
End Function
Private Function RetryRejectedCall(ByVal hTaskCallee As System.IntPtr, ByVal dwTickCount As Integer, ByVal dwRejectType As Integer) As Integer Implements IOleMessageFilter.RetryRejectedCall
Dim iResult As Integer
' See: IMessageFilter::RetryRejectedCall
' http://msdn2.microsoft.com/en-us/library/ms680739.aspx
' Return values:
' -1: The call should be canceled. COM then returns RPC_E_CALL_REJECTED from the original method call.
' Value >= 0 and <100: The call is to be retried immediately.
' Value >= 100: COM will wait for this many milliseconds and then retry the call.
If dwRejectType = SERVERCALL.SERVERCALL_RETRYLATER Then ' Thread call was rejected, so try again.
iResult = 99 ' Retry immediately
Else
' Too busy; cancel call.
iResult = -1
End If
Return iResult
End Function
Protected Overridable Sub Dispose(ByVal disposing As Boolean)
Dim dummyFilter As IOleMessageFilter
Dim hr As Integer
dummyFilter = Nothing
If Not m_disposedValue Then
hr = CoRegisterMessageFilter(m_oldFilter, dummyFilter)
System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(hr)
m_disposedValue = True
End If
End Sub
Public Sub Dispose() Implements IDisposable.Dispose
' Do not change this code. Put cleanup code in Dispose(ByVal disposing As Boolean) above.
Dispose(True)
GC.SuppressFinalize(Me)
End Sub
End Class