Monthly Archives: April 2017

VS 2017, VS Extensibility

Sample code and utilities to get installed Visual Studio 2017 editions programmatically

As I explained in the post Some implications of the new modular setup of Visual Studio 2017 for VSX developers, Visual Studio 2017 has changed all that you knew about installations of Visual Studio. In this episode of Channel 9, Art Leonard explains to Robert Green the internals of this re-architecture of Visual Studio:

The use of a private registry file causes that if you want to know programmatically the installed editions of Visual Studio 2017, the old approaches don’t work. For example, my article HOWTO: Detect installed Visual Studio editions, packages or service packs is now obsolete.

Fortunately, Microsoft provides a new Setup API to query the installed editions of Visual Studio 2017 or the highest VSIXInstaller.exe, along with sample code and utilities:

Visual Studio Setup Configuration Samples
Microsoft/vs-setup-samples
“This is a sample in various programming languages that demonstrates how developers can use the new Visual Studio setup query API. The included samples show how to use the new setup configuration API for discovering instances of Visual Studio 2017”.

Visual Studio Locator
Microsoft/vswhere
“Over the years Visual Studio could be discovered using registry keys, but with recent changes to the deployment and extensibility models a new method is needed to discover possibly more than once installed instance. These changes facilitate a smaller, faster default install complimented by on-demand install of other workloads and components. vswhere is designed to be a redistributable, single-file executable that can be used in build or deployment scripts to find where Visual Studio – or other products in the Visual Studio family – is located.”

Visual Studio Setup PowerShell Module
Microsoft/vssetup.powershell
“This PowerShell module contains cmdlets to query instances of Visual Studio 2017 and newer. It also serves as a more useful sample of using the Setup Configuration APIs than the previously published samples though those also have samples using VB and VC++.”

VSIX Installer Bootstrapper
Microsoft/vsixbootstrapper
“An installer that can be chained with other packages to locate the latest VSIXInstaller.exe to use for installing VSIX extensions. One of the great new features of Visual Studio 2017 is an initial smaller and fast install. To compliment a smaller – but powerful – initial feature set, installing additional workloads and components on-demand is supported for both end users and package developers. Package developers can install their VSIX extensions for Visual Studio using this bootstrapper to find the latest version of VSIXInstaller.exe and install their extension(s). This may be preferable for extensions that support Visual Studio 2017 or newer than installing extensions in Windows Installer .msi packages, since MSIs cannot run concurrently in separate processes. Other deployments may also benefit since they no longer have to find where VSIXInstaller.exe is installed. The command line passed to VSIXBootstrapper.exe is passed through to VSIXInstaller.exe.”

VS 2017, VS Extensibility, VS Packages

Building a VSIX extension with the Visual Studio 2017 Build Tools

As I explained in the post Migrating the build of a VSIX project to a build server if you are a solo developer, I am taking the steps to build my MZ-Tools extension on a build/release server. As part of that process, I realized than rather than installing Visual Studio 2017 Community edition on the server, I could use the Visual Studio 2017 Build Tools that were thought, well, for build servers that don’t need the overhead of a Visual Studio 2017 installation. They are a lightweight version of Visual Studio 2017 without the IDE (devenv.exe executable). They can be used to build either managed (C#, VB.NET, etc.) projects or native (C++) projects. Incidentally my MZ-Tools solution has both type of projects.

The Visual 2017 Build Tools can be downloaded from here. Once you install them on a clean machine, you will notice that they provide only the following:

  • A built-in (non-optional) set of components to build MSBuild-based projects (for example managed projects).
  • An optional workload “Visual C++ build tools”.
  • An optional workload “Web development build tools”.

There are also optional individual components to install .NET Framework 4.6.1 support Windows SDKs, ATL support, etc.:

In my case my extension needs to use .NET Framework 2.0 for some projects (I still support Visual Studio 2005). Since that version is not installed by default on modern versions of the Windows OS, I need to install it going to “Control Panel”, “Programs and Features” item, “Turn Windows Features on or off” link:

My obfuscator tool needs the .NET Framework 3.5 SDK (or the .NET Framework 2.0 SDK). While the Visual Studio 2017 Community installer provides the optional individual component “.NET Framework 3.5 developments tools”, the installer of Build Tools 2017 doesn’t. That is not only a pity but also causes a bug if you install yourself the Windows 7.0 SDK that contains the .NET Framework 3.5 SDK that I reported here: the resource .resx files of a .NET project targeting .NET Framework 2.0 are compiled using the Assembly Linker (al.exe tool) of the .NET Framework 4.0, which will cause them to fail silently at run-time. Microsoft fixed the bug just in time for RTM in Visual Studio 2017, but the Build Tools 2017 still has the bug due to the lack of the “.NET Framework 3.5 developments tools”. There is a fix that I explained in the bug report if you find this problem. I have also requested to Microsoft to include the “.NET Framework 3.5 developments tools” in the installer of Build Tools 2017.

My extension for Visual Studio targets version 2012, so I need to stick to .NET Framework 4.5, not some higher version. Since that version is not provided by the Build Tools 2017, I need to install the Windows 8.0 SDK, that contains the .NET Framework 4.5 assemblies and SDK.

For the C++ projects, I needed to install:

  • “Visual C++ ATL Support”: required to get files such as atlbase.h, etc.
  • “Windows 8.1 SDK”: I could upgrade to some Windows 10 SDK version but they occupy much more space on disk.
  • “UCRT SDK”: the Universal Common RunTime that provides files such errno.h and other files in the folder “C:\Program Files (x86)\Windows Kits\10\Include\10.0.10240.0\ucrt”.
  • “.NET Framework 4.6.1 SDK”: to get files such as mscoree.h / mscoree.lib in the folder “C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\”.

My build script uses tf.exe to set a workspace and download the latest sources. Alas, tf.exe is not installed with Build Tools 2017 (Visual Studio 2017 installs the Team Explorer extension, that contains that file in the folder “C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer”). Now there is a standalone installer for Team Explorer (that includes tf.exe) announced here. Since that’s a bit overkill, I just copied tf.exe and the required dlls.

The build script needs also nuget.exe, but that’s easy to get.

So, when I thought that I had all the required components installed, I tried to build the extension. I got the following problems:

First, the error “Error’MSB4226: The imported project “(…)\VSSDK\Microsoft.VsSDK.targets” was not found.” Initially I thought it was a bug, that I reported to Microsoft here, but I discovered that the problem was solved setting the “VisualStudioVersion” MSBuild property, something that a machine with the full Visual Studio 2017 does and that a machine with the Build Tools 2017 does if you open a developer command prompt. Since I was not using it, I passed it as a parameter to the MSBuild script. It can be defined too inside the .csproj file, something that previous Visual Studio versions did automatically but recent versions don’t.

Then I got an error about a missing Microsoft.VisualStudio.Settings.15.0 file. How is that a file from Visual Studio is required by the NuGet package that provides the Visual Studio SDK?. It happens that Microsoft.VsSDK.Build.Tasks.dll, the file that contains the MSBuild tasks needed when creating a VSIX file, references it. But on a machine without Visual Studio, that dll is not present. I discovered that despite the error, the VSIX file was generated correctly, so that DLL was required for a task after generating the VSIX file. That task is to deploy the generated VSIX file to the experimental instance for debugging. Since on a release server that step is not needed, I knew how to instruct the .csproj project file to avoid it:

<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
   ... 
   <!-- Do not deploy the extension in Release configuration -->
   <DeployExtension>False</DeployExtension>
</PropertyGroup>

Finally, when I thought that all obstacles were solved, I got an error about the Microsoft.Visualstudio.Shell.Interop.dll being delay-signed rather than strong-signed. Since my development machine has tons of Visual Studio versions and SDKs, I discovered that all the Microsoft.Visualstudio.Shell.Interop.dll files were strong-named except an old one, which somehow Microsoft shipped delay-signed, and that was the one that I was referencing. So, I only need to change it by the correct version. And finally, my extension generates a VSIX file on a release server with only the Build Tools 2017 plus some additional components, but without Visual Studio 2017 installed.

The next step is to install some agent to connect to the Build/Release Management of Visual Studio Team Services.

VS Extensibility

Microsoft, the Visual Studio Shells and the old versions

2 Comments

Visual Studio 2008 started to offer a new form of extensibility: a whole IDE for your app! That is, reusing the core of the Visual Studio IDE for your own tool instead of reinventing the wheel creating a new IDE from scratch. This is named Visual Studio Shell and there are two flavors: the “Isolated Shell“, where your app has its own instance of Visual Studio Shell even if the Visual Studio IDE is already installed (because the user is a developer); and the “Integrated Shell“, where if the Visual Studio IDE is already installed, your Visual Studio Shell integrates with it (otherwise is installed anyway).

I have never worked with the Visual Studio Shell, nor I have any knowledge, but I provide a page on this site for the download links. Alas, it seems that Microsoft doesn’t pay much attention to keep the links working, so I get questions on forums and by email about their availability. If the links of that page fail, you can try the following:

It would be nice if Microsoft provided a centralized, public page with the downloads that VSX developers may require to develop and test extensions (Visual Studio SDKs, .NET Framework SDKs, Visual Studio Shells, Community editions, etc.) and, very important, from the current Visual Studio version going back to… Visual Studio 2005 and .NET Framework 2.0. Not everyone wants or can use the latest bits. I keep a couple of hard disks at home and at my basement with all the required Visual Studio versions, Visual Studio SDKs, Windows SDKs, .NET Framework SDKs, Windows OS versions,  etc. fearing that they would disappear from the web some day (which certainly is happening with the shells). And now that Microsoft uses web-based installers for Visual Studio with no .iso file provided, I am even afraid that my Visual Studio 2017 installer won’t work in a few years, so I have to create an offline installer.

The strange case of...

The strange case of VB6 getting “access denied” building ActiveX DLL running with Registry virtualization

This is one of those posts that I write mainly for myself in the future, but that maybe can be useful for others developers that want or need to run Visual Basic 6.0 without admin rights on Windows 7 or Windows 10.

One of the flavors of my MZ-Tools add-in is for Visual Basic 6.0 (yes, there is still quite a few people using it). Visual Basic 6.0 was created and used heavily in a time (Windows 98-Windows XP) when every Windows user was an administrator, so it was not a problem that when building an ActiveX DLL project some registry entries were added to the HKEY_CLASSES_ROOT (HKCR) registry hive, actually to HKEY_LOCAL_MACHINE (HKLM) internally. On Windows Vista and higher with the User Account Control (UAC) that changed, so if you run Visual Basic 6.0 without admin rights you get this error when building an ActiveX DLL:

A solution is of course to run Visual Basic 6.0 with admin rights, and I have been doing it so for some years. But that requires Visual Studio running also with admin rights, to be able to debug an add-in project loaded on Visual Basic 6.0 (if Visual Studio is not running with admin rights, it prompts you to restart with admin rights, but that’s quite annoying). But running Visual Studio all the time with admin rights only because of Visual Basic 6.0, when all the IDEs (Visual Studio, VBA) of my other add-ins wouldn’t require it, is somewhat overkill, and I really wanted to avoid it.

So, I was aware of the Registry virtualization mechanism of UAC, which Microsoft invented precisely for this scenario: an application that required admin rights to write in some keys of the HKLM\Software key of the Registry but that was not allowed to run with admin rights. Since Registry virtualization was not working for Visual Basic 6.0, my first thought was that it no longer existed on Windows 10 (documents say that apps cannot rely on this mechanism forever since it will be removed in some Windows version), or that it needed to be activated at global level. But no, if you open the Local Security Policy and go to Local Policies > Security Options, the setting exists and file and registry write failures are virtualized by default:

My second thought was that it needed to be activated per-application. But no, all the documents say the opposite, applications can disable registry or file virtualization by adding a .manifest, which signals to the Windows OS that the application is UAC-compliant (and therefore virtualization is not required). And indeed Process Explorer revealed that vb6.exe runs with virtualization:

So, what was happening? Process Monitor to the rescue. I noticed that Visual Basic 6.0 was trying to write to HKCR\TypeLib, getting “access denied”, not directly to HKLM:

But HKCR is actually a merged view of some keys of HKLM and some keys of HKCU, and clearly the failure was due to the HKLM side.

Finally, a closer reading of the page Registry Virtualization on MSDN, section Controlling Registry Virtualization, gave me the clue:  it happens that not all the keys of HKLM\Software are virtualized by default. In particular, the keys and subkeys of HKLM\Software\Classes, which are used when registering an ActiveX component, are not virtualized by default. Fortunately, you can use the reg.exe tool to query and virtualize them.

So I launched an elevated (with admin rights) command prompt and queried the HCKR\Classes\TypeLib (actually HKLM\Software\Classes\TypeLib) in 32-bit Registry (Visual Basic 6.0 is a 32-bit executable) with this command:

reg.exe FLAGS HKLM\Software\Classes\TypeLib QUERY /reg:32

Note: the fact that Visual Basic 6.0 is a 32-bit executable and that Process Monitor showed HKCR\TypeLib (that would belong to 64-bit) and not HKCR\WOW6432Node\TypeLib (that would belong to 32-bit) means that both keys (32-bit and 64-bit) are the same.

I got that it is not virtualized (REG_KEY_DONT_VIRTUALIZED: SET):

To clear that flag I executed the following, that clears the DONT_VIRTUALIZED flag but keeps set the RECURSE_FLAG as before:

reg.exe FLAGS HKLM\Software\Classes\TypeLib SET RECURSE_FLAG /reg:32

Trying to build again the ActiveX DLL causes the same “Error accessing the system registry”, but this time Process Monitor revealed that the problem was in HCKR\WOW6432Node\Interface:

Notice that in this case HCKR\WOW6432Node\Interface is shown, instead of HCKR\Interface, which means that the 32-bit and the 64-bit keys are different.

So I queried the flags with this command:

reg.exe FLAGS HKLM\Software\Classes\Interface QUERY /reg:32

And I got that certainly is not virtualized:

Again, to clear that flag I executed the following, that clears the DONT_VIRTUALIZED flag but keeps set the RECURSE_FLAG as before:

reg.exe FLAGS HKLM\Software\Classes\Interface SET RECURSE_FLAG /reg:32

Trying to build again the ActiveX DLL caused the same “Error accessing the system registry”, but this time Process Monitor revealed that the problem was in HCKR\Interface, so I executed:

reg.exe FLAGS HKLM\Software\Classes\Interface SET RECURSE_FLAG /reg:64

And finally building the ActiveX didn’t cause any error.

VS 2017, VS Extensibility, VS Packages

Migrating the build of a VSIX project to a build server if you are a solo developer

2 Comments

Some months ago I started a long and slow journey to migrate the build process of my MZ-Tools extension from a custom .NET-based builder that ran on my development computer to Visual Studio Team Services, leveraging its Build management and Release management capabilities. My goals are to learn those capabilities and, well, to use them as if I were a team. I haven’t reached yet that destination but I have made significant progress and I am quite close now. In the process I have realized somewhat ashamed that I wasn’t following the best practices in a lot of places. I say “somewhat” because there are some mitigating circumstances:

  • Being a solo developer it is too easy to arrange things in such a way that only works on your development machine. You don’t have a team to warn you that it doesn’t work outside your development machine.
  • Even if you change your development machine to a new one from time to time, or use two development computers, you use the same username and tend to install and configure the software in the same way.
  • My Visual Studio solution and some projects were born in the year 2002, with the first Visual Studio .NET. At that time Team Foundation Server, NuGet, MSBuild, etc. didn’t even exist on paper.
  • I have all the Visual Studio versions and VS SDKs from 2005 to 2017 installed on my development machine. That causes that you don’t think carefully where a DLL is referenced from.
  • I have also all the .NET Frameworks and SDKs from .NET Framework 2.0 installed on my development machine. Another source for undocumented hidden dependencies.
  • My Visual Studio solution has become quite complex over the years with several projects and technologies:
    • One project that uses .NET Framework 2.0 and C# for the core plug-in, that is reused at binary level for many Microsoft’s IDEs.
    • Four projects with host adapters for Visual Studio (2005, 2008, 2010) as add-in, VBA (Microsoft Office 2000 or higher, even on Windows XP), VB 6.0 and VB 5.0, using .NET Framework 2.0, C# and 3rd party tool for the setups.
    • One project with a host adapter for Visual Studio (2012, 2013, 2015 and 2017) as package, using .NET Framework 4.5, C#, and VSIX.
    • Four projects with COM Shims for VBA (32-bit and 64-bit), VB 6.0 and VB 5.0 using Visual C++, ATL, Windows 8.1 SDK and the .NET CLR loader APIs.
    • One project for a portable version for VBA, using .NET Framework 2.0 and C#.
    • Help file and online help generated by a 3rd party tool.
    • Two projects with integration tests.
    • Obfuscation performed by a 3rd party tool, that requires delay signing.

Microsoft tends to create a new VSIX with each new Visual Studio release but most of us want to create a single package and single VSIX for as many Visual Studio versions as possible, using always the latest Visual Studio version for development (Visual Studio 2017 at the time of this writing). If this is your case, even if your solution is not as complex as mine, ask yourself these questions:

In the process that I have followed first I envisioned the final result:

  • I would use the hosted agent of the Build management of Visual Studio Team Services to provide the following benefits:
    • Gated check-ins to prevent code that breaks the build.
    • Gated check-ins to prevent code that violates code analysis rules.
    • Integration tests. This will require a major effort because I use my own test runner and testing framework instead of Visual Studio Test with the MSTest framework.
  • I would use a private agent on the Release server with the Release management of Visual Studio Team Services to provide the following benefits:
    • Tracking of releases deployed to the test environment, to the pre-production environment and to the production environment.
    • Maybe automated releases.

For these purposes I decided that I would use a new “Build” configuration of the solution for the Build server. In this configuration the obfuscation, help file and online help, setups, etc. are not created. For the Release server I would use the “Release” solution configuration that performs all those additional steps.

For the Build server it is quite easy and I can use the hosted agent of Visual Studio Team Services since I don’t need any 3rd party tools.

For the Release server I cannot use the hosted agent because I need the 3rd party tools that I use to obfuscate, create the help, setups, etc. In the process I have also removed the need for admin rights that my custom builder required previously (the hosted agent of Visual Studio Team Services doesn’t allow admin rights either).

The milestones would be:

  • Migrate the custom .NET-based builder to a MSBuild script. This took me weeks but it’s done.
  • The solution and projects, when built in “Release” configuration, would auto-increment version numbers and would obfuscate the output assemblies, build the help, setups, etc. with their own MSBuild targets without requiring external steps.
  • A master MSBuild script would perform some additional steps before building the solution (such as creating a workspace, getting latest files, restoring NuGet packages, etc.) and some other steps afterwards (such as publishing and archiving binaries).
  • Build on a separate workspace on my development machine. Previously I was building on the same workspace used for development. I know, I know, a horrible bad practice.
  • Create a separate temporary virtual machine to act as Release server with only Visual Studio 2017 Community edition installed, without previous Visual Studio versions to identify missing dependencies and build with the master MSBuild script.
  • Discard the virtual machine of the previous step and create a new virtual machine with the Build Tools 2017 but without Visual Studio 2017, identify and install missing dependencies, build with the master MSBuild script and document everything. I have reached this point!
  • Install a private Visual Studio Team Services agent on the virtual machine that acts as Release server and launch the master build script from the Release management section of Visual Studio Team Services.
  • Create additional scripts to publish binaries to the pre-production and production environment rather than using manual FTP.
  • Repeat everything with the solution for my ASP.NET web site.

I encourage you to follow the same journey if you are a solo developer (and of course if you are a team). At the very least, the exercise of building on a separate build server with only Build Tools 2017 will unhide you hardcoded paths, references that are not provided by source control or supplied as NuGet packages, etc.