Testing Packages

Boxstarter can detect which packages have changed and test their installations on one or more deployment targets. Boxstarter can automate publishing successful packages to their feed.

The Boxstarter TestRunner module can test Chocolatey package installs on one or more deployment targets and evaluate whether the installation was successful. The TestRunner module can also be used to automate the publishing of Chocolatey packages to a NuGet feed. That may be the public Chocolatey community feed or a private feed.

The TestRunner module can be downloaded and installed via Chocolatey by running:

choco install Boxstarter.TestRunner

Configuring the Local Boxstarter Repository

Boxstarter needs to know where your repository of Chocolatey packages is located. The Boxstarter repository is a single directory on disk that contains one subdirectory per package. By default, Boxstarter maintains this directory in a directory called BuildPackages inside the same root directory where the Boxstarter modules reside. See Creating Packages for detailed information on using Boxstarter commands to simplify package creation. All Boxstarter commands that create and build packages act within this directory. The repository location can be found using the $boxstarter settings variable in $Boxstarter.LocalRepo.

This location can be changed using the Set-BoxstarterConfig command.

Set-BoxStarterConfig -LocalRepo "c:\dev\Chocolatey-Packages"

By changing the location of the boxstarter repository with Set-BoxstarterConfig all future interactions with the Boxstarter repository will act on this location. To temporarily change the repository location just for the current PowerShell session, one can change the value of $Boxstarter.LocalRepo directly.

$Boxstarter.LocalRepo = "c:\dev\Chocolatey-Packages"

Configuring Boxstarter Testing Options

Boxstarter tests Chocolatey packages by deploying and installing the package to one or more deployment targets. The deployment options include settings that control what computers to use to test the packages, the credentials to use, VM checkpoints to restore as well as NuGet feed and API key for publishing successful packages.

To read the current settings for these options, use Get-BoxstarterDeploymentOptions. To set these options use Set-BoxstarterDeploymentOptions.

$cred=Get-Credential Admin
Set-BoxstarterDeployOptions -DeploymentTargetCredentials $cred `
  -DeploymentTargetNames "testVM1","testVM2" `
  -DeploymentVMProvider Azure -DeploymentCloudServiceName ServiceName `
  -RestoreCheckpoint clean `
  -DefaultNugetFeed https://www.myget.org/F/mywackyfeed/api/v2 `
  -DefaultFeedAPIKey 5cbc38d9-1a94-430d-8361-685a9080a6b8

This configures package deployments for Azure VMs testVM1 and testVM2 hosted in the ServiceName cloud service using the Admin credential. Prior to testing a package install, the VM will be restored to the clean checkpoint. If packages are published using Publish-BoxstarterPackage and are not associated with a NuGet feed, they will publish to the mywackyfeed on myget.org using API Key 5cbc38d9-1a94-430d-8361-685a9080a6b8.

Currently the available Boxstarter VM providers are Azure and HyperV. Any VM regardless of hypervisor implementation can be used by passing in the names of the computers as the deployment targets. However, use of Boxstarter VM providers are necessary in order to take advantage of checkpoints.

The default deployment options deploy locally using the credentials of the current user and publish to the public Chocolatey community feed.

Individual packages in a Boxstarter repository can be published to different feeds. The NuGet feed and API Key configured using Set-BoxstarterDeploymentOptions are default settings used if a package has not been individually assigned to its own feed.

To assign a package to an individual feed, use Set-BoxstarterPackageNugetFeed.

Set-BoxstarterPackageNugetFeed -PackageName MyPackage -NugetFeed https://www.myget.org/F/myotherfeed/api/v2

When using multiple feeds, if one needs to use the Publish-BoxstarterPackage command to publish the package to its feed, an API key for each feed must be set. This can be accomplished by using the Set-BoxstarterFeedAPIKey.

Set-BoxstarterFeedAPIKey -NugetFeed https://www.myget.org/F/myotherfeed/api/v2 -APIKey 5cbc38d9-1a94-430d-8361-685a9080a6b8

Testing Chocolatey Packages

Test-BoxstarterPackage can be called with an array of packages.

Test-BoxstarterPackage -PackageName "MyPackage1","MyPackage2"

Boxstarter will build their .nupkg files and attempt to install them on the deployment targets specified with Set-BoxstarterDeploymentOptions. Boxstarter will use the credentials provided in the deployment options. You can provide several targets to Set-BoxstarterDeploymentOptions. One may wish to supply different machines running different versions of windows. If a package install runs to completion with no exceptions or returned error codes, Boxstarter considers the install a PASSED test. If Test-BoxstarterPackage is called with no packages specified, Boxstarter will iterate over each package in its local repository. It will build the nupkg and compare its version to the version on the package published feed. If the version in the repo is greater than the published version, Boxstarter will initiate a test on the deployment targets otherwise the package test will be skipped.

Publishing Successful Packages

Test-BoxstarterPackage will return a set of test results. One can then use Select-BoxstarterResultsToPublish to consume these results and return the package IDs of the packages who had all deployment targets pass the package install. These IDs can then be passed to Publish-BoxstarterPackage to publish those packages to their associated feeds.

Test-BoxstarterPackage | Select-BoxstarterResultsToPublish | Publish-BoxstarterPackage

Conserving Cloud Resources

If the Azure Boxstarter module is used as a VM provider and any of the Azure VMs are not running when the tests begin, Boxstarter will shut down each VM that was not initially running once the testing is completed.

Including Testing and Publishing in Continuous Integration

The Boxstarter TestRunner can be plugged into modern Build systems and respond to source control commits. The Boxstarter local repo can be setup under a source control system like GIT, Mercurial, TFS, SVN etc. Build servers like TeamCity, TFS, Jenkins and others can be configured to listen for changes to the Boxstarter repo. For instance, if the repository is under git source control and pushed to a remote server monitored by a build server, the build server can execute the Boxstarter TestRunner commands to test the packages that have been changed and publish them to their feeds if the tests pass.

The TestRunner module includes an Install-BoxstarterScripts command that embeds a MSBuild file, a bootstrapper and a PowerShell script into a repository that can run the tests and publish successful packages.

PS C:\dev> git clone https://github.com/mwrock/Chocolatey-Packages.git
Cloning into 'Chocolatey-Packages'...
remote: Reusing existing pack: 2963, done.
Receiving objects: 100% (2963/2963), 3.93 MiB |
Resolving deltas: 100% (961/961), done.
Checking connectivity... done.
Checking out files: 100% (2491/2491), done.
PS C:\dev> Install-BoxstarterScripts -RepoRootPath .\Chocolatey-Packages
Boxstarter: Copying Boxstarter TestRunner scripts to .\Chocolatey-Packages\BoxstarterScripts
PS C:\dev> dir .\Chocolatey-Packages\BoxstarterScripts
    Directory: C:\dev\Chocolatey-Packages\BoxstarterScripts
Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         3/30/2014  11:02 PM         32 .gitignore
-a---         3/30/2014  10:53 PM       3471 bootstrap.ps1
-a---         3/30/2014  10:53 PM        887 boxstarter.proj
-a---         3/30/2014  10:53 PM       1305 BoxstarterBuild.ps1

There are two primary usage patterns for running tests from a build server:

Dedicated Build Server

Here, tests are run in a build environment on a server that is under your direct ownership and full control. The key differentiator here is that you can set your deployment options on this machine and be confident that they will remain on every build from build to build.

Hosted or Shared Build Server

This is often a build server that you do not own or control outside of your individual builds. If you set your deployment options on the server, they may likely be gone on your next build. The Visual Studio Online build services would land in this category.

Configuring a Dedicated Build Server

If you administer your own build server, there is some one time setup that you need to perform to integrate Boxstarter TestRunner tests and automatic package publishing. This must be performed on the build machine that will run the builds (a remote PowerShell session is acceptable). In many build systems, this may be a different machine from the central server, in TFS it would be the build "controller" or in TeamCity the build "agent." Also, these steps must be performed using the credentials of the SAME USER ACCOUNT that your build server runs under.

Install the Boxstarter TestRunner module on the build machine. You can use Chocolatey and run choco install Boxstarter.TestRunner or launch https://boxstarter.org/package/nr/boxstarter.testrunner from IE.
Although many of the deployment options are stored in a file inside the repository and will be available when your build server gets the latest repository code, the "secret" options are not kept in source control to prevent this information from being publicly disclosed. These secrets include the deployment target credentials and feed API keys. So you can use Set-BoxstarterDeployOptions and Set-BoxstarterFeedAPIKey to set these secrets. The Credential passwords are encrypted and can only be unencrypted on the same build server by the same user account setting the options. You may also add these settings as msbuild parameters in step 4 described below.
```
$cred=Get-Credential Admin
Set-BoxstarterDeployOptions -DeploymentTargetCredentials $cred `
  -DefaultFeedAPIKey 5cbc38d9-1a94-430d-8361-685a9080a6b8
            
```
If you will be performing the package test installations on Azure VMs, you will need to configure your Azure subscription. Run
```
Get-AzurePublishSettingsFile
```
to download your subscription file from Azure and then
```
Import-AzurePublishSettingsFile -PublishSettingsFile "C:\saved\subscription\file.publishsettings"
```
You may also add the Azure management key as a MSBuild parameter described next. See these instructions for setting up an Azure subscription for more details.
Configure the build configuration of your build server to call the Boxstarter.proj MSBuild file as a step in your build. This file is located in the BoxstarterScripts directory of your repo which Boxstarter creates when you run Install-BoxstarterScripts. This script will invoke a bootstrapper which will install everything needed for the TestRunner to run including the TestRunner itself if not present. On dedicated build servers, where the TestRunner is preinstalled, the bootstrapper will be skipped.

The script can take the following parameters:
- PublishSuccesfulPackages - Set this to true if you would like successful packages published.
- DeploymentTargetUserName - Username to use when initiating connections to the deployment targets.
- DeploymentTargetPassword - Password to use when initiating connections to the deployment targets.
- FeedAPIKey - API key to use when publishing packages. AzureSubscriptionName - Name of the azure subscription to use when using Azure VMs as deployment targets.
- AzureSubscriptionId - The Azure subscription ID Guid to use when using Azure VMs as deployment targets.
- AzureSubscriptionCertificate - The Azure subscription certificate to use when using Azure VMs as deployment targets. This is the Base64 encoded content of the certificate and can be found in the ManagementCertificate attribute of your Azure publish settings file.
NOTE: All of these MSBuild script parameters are optional for dedicated builds as long as the values were provided in steps 2 and 3 above which may be preferable over having plain text parameter values fed to your build.

Configuring a Hosted or Shared Build Server

If your build server environment is shared and may be built from scratch on every build then you cannot preinstall the TestRunner and configure necessary settings up front. Instead the runner must be installed during the build process and the settings passed via MSBuild parameters. When you run Install-BoxstarterScripts on your Chocolatey repository, a bootstrapper is added to your repo that can take care of installing the Test Runner and its dependencies. Additionally, you can pass the following MSBuild parameters to the BoxstarterBuild.proj script:

PublishSuccesfulPackages - Set this to true if you would like successful packages published.
DeploymentTargetUserName - Username to use when initiating connections to the deployment targets.
DeploymentTargetPassword - Password to use when initiating connections to the deployment targets.
FeedAPIKey - API key to use when publishing packages. AzureSubscriptionName - Name of the azure subscription to use when using Azure VMs as deployment targets.
AzureSubscriptionId - The Azure subscription ID Guid to use when using Azure VMs as deployment targets.
AzureSubscriptionCertificate - The Azure subscription certificate to use when using Azure VMs as deployment targets. This is the Base64 encoded content of the certificate and can be found in the ManagementCertificate attribute of your Azure publish settings file.