I have been very stressed the last weeks in my spare time (nights and weekends) to ensure that the upcoming version 8.0 of my MZ-Tools extension works with Visual Studio 2015 CTPs and RC. This has been complicated because of these factors:
- MZ-Tools 7.0 was an add-in, and MZ-Tools 8.0 will be a package.
- MZ-Tools 8.0 for VS shares 80% of code with the also upcoming MZ-Tools 8.0 for VBA / VB, which is not yet finished.
- VS 2015 belongs to my definition of “painful” category for extensions, because of the Roslyn-based implementation of the file code model.
Today, VS 2015 RC has passed for the first time the 3,300+ automated tests of MZ-Tools. And they are integration tests (not unit tests), each one testing a whole user-simulated scenario: clicking the options window of MZ-Tools, setting some options, clicking the button of a feature, capturing the result (either contents of an output toolwindow or generated code) and comparing with the expected result. They are able to detect subtle changes in the behavior of VS 2015 such as different automatic formatting, automatic removal of line continuation character “_” in VB.NET, automatic addition of parenthesis “()” to parameter-less VB.NET methods if the test omitted them, namespaces omitted from types in the Error List after a build, etc. These are not really bugs but required to tweak the tests, in some cases maintaining backward compatibility with previous VS versions and in some cases detecting the VS version and using different code.
The tests also detected problems in the automation file code model (EnvDTE.ProjectItem.FileCodeModel), which I patiently isolated to root causes resulting in a total of 18 bugs that I have reported on GitHub, the last one, quite obscure, yesterday. At the time of this writing, at least 15 are fixed (or will be fixed for VS 2015 RTM) and the remaining three ones have workarounds that I have coded to pass the tests.
So, at this point, VS 2015 RC is good enough from my point of view.
I was very interested to read that you have over 3,000 automated integration tests for your VS extension. That is a significant achievement!
Which automated test framework do you use to execute your tests? I have some experience with FitNesse.
Hi Richard,
For integration tests, Visual Studio Extensions need to be hosted in a second Visual Studio instance, and Microsoft provides a MS Test VS IDE Host adapter. However, it didn’t worked well for me some years ago so I ended developing my own test framework that runs in the second instance rather that in the first instance.