Monthly Archives: January 2018

VS Packages

Code signing a VSIX package targeting multiple Visual Studio versions

You know that I am such a big fan of targeting multiple Visual Studio versions with the same VSIX file that I wrote an article for MSDN Magazine explaining the approach to target  from Visual Studio 2012 to Visual Studio 2017. In that article I didn’t explain how to code sign such VSIX file to work with multiple Visual Studio versions but this week I have got a (personal) code signing certificate.

First things first, if you don’t code sign your VSIX file, the VSIX installer shows that the digital signature is none:

To code sign a vsix file, you can’t use the regular signtool.exe tool of the Windows SDK. You need to use the vsixsigntool.exe that you can get from NuGet:

And the official documentation to sign VSIX packages is here:

Signing VSIX Packages

In the samples that follow, I will assume that you have the vsixsigntool.exe file and a code signing certificate named CodeSigningCertificate.pfx in the same bin\debug folder that contains a vsix file named MyVSIXProject.vsix.

If you specify in the source.extension.vsixmanifest file that your VSIX package only targets Visual Studio 2012 (11.0) and 2013 (12.0):

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[11.0,12.0]" />
 </Installation>

and you sign the VSIX file with with the /fd sha1 option to use the SHA1 algorithm (do not confuse with the /sha1 option, which is used to select the correct certificate):

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha1 MyVSIXProject.vsix

then, although initially you can get this “invalid certificate” error:

after a few attempts while connected  to internet then it works as expected (I don’t know yet why the initial error, does anyone know?):

However, if you sign the VSIX file with with the /fd sha256 option to use the recommendable SHA256 algorithm:

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha256 MyVSIXProject.vsix

then you get an error because the algorithm is not supported:

Now, if you specify in the source.extension.vsixmanifest file that your VSIX package only targets Visual Studio 2015 (14.0) and 2017 (15.0):

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[14.0,15.0]" />
 </Installation>

and you sign the assembly with with the /fd sha1 option to use the SHA1 algorithm:

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha1 MyVSIXProject.vsix

then the VSIX installer shows “Invalid signature”:

That is because Visual Studio 2015 and higher no longer support the SHA1 algorithm, only the SHA256 algorithm. If you sign the VSIX file with with the /fd sha256 option to use the SHA256 algorithm:

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha256 MyVSIXProject.vsix

then the VSIX installer shows the correct signature:

But, what if you specify in the source.extension.vsixmanifest file that your VSIX package targets from Visual Studio 2012 (11.0) to Visual Studio 2017 (15.0)?:

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[11.0,15.0]" />
 </Installation>

Of course, using the SHA256 algorithm would case the “invalid signature” warning when installing the VSIX package for Visual Studio 2012 or 2013. But if you use the SHA1 algorithm:

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha1 MyVSIXProject.vsix

then, because the VSIX installer notices that the VSIX package is also valid for Visual Studio 2012 and 2013, it no longer shows “Invalid signature” for Visual Studio 2015 and 2017:

Note: according to Ed Dore in this post, this approach wouldn’t work with older versions of the VSIX Installer if the manifest states explicitly the higher version:

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[11.0,15.0]" />
 </Installation>

and the workaround was not to specify the higher version:

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[11.0,)" />
 </Installation>

But that bug is already fixed and you can specify the higher version.

VS Extensibility

The VisualStudio-TestHost project to execute interactive UI tests in Visual Studio

3 Comments

I started with automated tests for my MZ-Tools extension early in the development of version 7.0 (then an add-in, not a package), ten years ago, around the year 2008 or so. At that time Visual Studio 2008 provided a Visual Studio TestHost dll (even with source code, if I remember correctly) to run automated tests of Visual Studio packages using the own Visual Studio. I remember that it was so painful to use that approach (crashes, hangings, etc.) that after months of investment I threw all the stuff and started to build my own test framework infrastructure and test runner:

(yep, I have 3,354 automated integration tests for my extension for Visual Studio)

It was perfect in most aspects because I owned the code and could adapt it to my needs but it had one important inconvenient: it didn’t allow Continuous Integration (CI). Instead, I ran the tests manually before each (monthly) build. It had another inconvenient: I learned automated testing by myself without formal training and I ended with tons of integration tests (which use the real Visual Studio to host the extension), no system tests (that would mock Visual Studio) and no unit tests (that just test a single method or class). I know now that, for performance reasons, it is much better to have have a pyramid with tons of unit tests, lots of system tests and few integration tests.

Last year I started to focus on DevOps (even if I am a solo developer) and continuous integration, and decided to make my test framework infrastructure compatible with VS Test / MSTest, so I could keep using my own test runner for integration tests but use them 100% as system tests mocking Visual Studio with stubs:

As you can see, my progress is quite modest at this point (only 133 system tests vs 3,354 integration tests) because I need to mock the full EnvDTE.FileCodeModel for most tests. But it is a feasible approach because I have fully mocked with stubs the whole VBA / VB6 IDEs that I use for MZ-Tools 8.0 for VBA / VB6. So, for example I have 1,822 integration tests for VB6 with my own test runner (an add-in for VB6):

And the same 1,822 tests as system tests using the test runner of Visual Studio:

Once I finish the system tests for Visual Studio, the next step is to give another try to run the integration tests using the test runner of Visual Studio, rather than my own test runner, which will allow me Continuous Integration not only for system tests, but also for integration tests. For such approach, I learned in the Two videos about building High Performance Extensions by Omer Raviv about the Microsoft/VisualStudio-TestHost project on GitHub. I have still to read and learn about it, but I think it is based on the code of the VS TestHost of 2008 and hopefully these 10 years have been used to solve the problems that caused me to abandon that approach.

Update (July 29, 2018): yesterday I finished the last stubs for Visual Studio and now all the integration tests for the real Visual Studio are also system tests with stubs that simulate the behavior of Visual Studio:

VS Extensibility

Two videos about building High Performance Extensions by Omer Raviv

Omer Raviv, author of the OzCode extension for Visual Studio, has recorded two videos with Robert Green on Channel 9 about building high performance extensions for Visual Studio. If creating extensions for Visual Studio is already tricky, creating high performance extensions is quite difficult and Omer has some advices and techniques about it.

The first video is this one:

Building High Performance Extensions Part 1

And the second one is this:

Building High Performance Extensions Part 2