How do fail a Packer pipeline upon powershell provisioner pester test failure?

Question

I'm building a packer pipeline and having an issue in getting the failure of a powershell provisioner step to fail the packer job itself. This is important as without failing the Packer job, the AMI would be published, despite the test failure.

How I'm proceeding:

1. Run provisioning setup steps
2. Copy Pester Script to Remote machine
3. Invoke inline powershell statement to Invoke-Pester
4. Download the test results from the Pester remote run
5. Next provisioning step parses the file to count errors and throws error upon issue count > 0

However, even with issue count being > 0 it is not causing the pipeline to fail. Additionally some errors thrown in other provisioner steps also fail to cause the Packer job to stop. I've set $ErrorActionPreference = 'Stop' on one of them and it didn't seem to make any difference.

Here's part of the provisioning workflow for packer.

...     {
            "type": "file",
            "source": "{{ template_dir }}/tests/windows-server.tests.ps1",
            "destination": "{{ user `TEMP_DIRECTORY` }}/"
        },
        {
            "type": "powershell",
            "environment_vars": [
                "TEMP_DIRECTORY={{ user `TEMP_DIRECTORY` }}/"
            ],
            "inline": [
                "if(-not (Get-InstalledModule Pester -MinimumVersion 4.10.1)){ Install-Module Pester -Scope AllUsers -AllowClobber -Force -verbose} ",
                "Import-Module Pester -MinimumVersion 4.10.1 -Force -DisableNameChecking",
                "$null = New-Item -Path $ENV:TEMP_DIRECTORY -ItemType Directory -Force -ErrorAction SilentlyContinue",
                "$testresult = Invoke-Pester -Script (Join-Path $ENV:TEMP_DIRECTORY 'windows-server.tests.ps1') -OutputFile (Join-Path $ENV:TEMP_DIRECTORY 'TEST-pester-results.xml') -OutputFormat 'NUnitXML' -PassThru"
            ]
        },
        {
            "type": "file",
            "direction": "download",
            "source": "{{ user `TEMP_DIRECTORY` }}/TEST-pester-results.xml",
            "destination": "{{ template_dir }}/artifacts/TEST-pester-results.xml"
        }, ...

Once this part is done it's followed by this step which is some logic from this blog post:

  {
            "environment_vars": [
                "TEMP_DIRECTORY={{ user `TEMP_DIRECTORY` }}/"
            ],
            "script": "./scripts/Get-TestResults.ps1",
            "type": "powershell"
        },

The Get-TestResults.ps1 has the following function thanks to this article: Testing Packer Images with Pester written originally by Sam Cogan and modified slightly by me.

[cmdletbinding()]
param()
$TestFile = Join-Path $ENV:TEMP_DIRECTORY 'TEST-pester-results.xml'

if (Test-Path -Path $TestFile -PathType Leaf)
{
    [xml]$testResults = Get-Content -Path $TestFile -Raw
    $totalIssues=[int]$($testResults.'test-results'.errors) + [int]$($testResults.'test-results'.failures) + [int]$($testResults.'test-results'.invalid)
    exit $totalIssues
}
else
{
    Write-Error "🔥🔥🔥 Failed to find `$TestFile: $TestFile"
    Write-Warning "$(Get-ChildItem $ENV:TEMP_DIRECTORY -Recurse | Format-Table -autosize -wrap | Out-String)"
    Write-Error "🔥🔥🔥 Failed to find `$TestFile: $TestFile"
    exit 1  #not a good exit :-)

}

Answer 1

After resolving some syntax issues, I found the true cause of failing to throw the error. In my prior usage I was doing this through Inspec provisioner, which handled the exit codes.

If you are using Pester and having to do the workaround I described above, then to ensure error's in a step are thrown you need to throw an exit code other than 0.

The two steps I took that seem to work fine (I like some emoji fire in my logs 😁)

$ErrorActionPreference = 'Stop'

try 
{
# Do code here
}
catch
{
   Write-Error "🔥🔥🔥 Failed to do something important `$TestFile: $TestFile"
  exit 1  # packer will recognize failure at this point
}

Note that you will want to make sure your Azure DevOps Pipeline has the following settings to publish test results regardless of failing:

# ℹ  Using Invoke-Build to call Packer. You can do this with a packer extension or any other way you prefer
- task: PowerShell@2
  displayName: Run Packer Configuration
  inputs:
      filePath: build/build.ps1
      arguments: '-Task PackerBuild -Configuration $(Configuration)'
      errorActionPreference: 'Continue'
      pwsh: true
  continueOnError: true #I want to publish the test results regardless

- task: PublishTestResults@2
  displayName: Publish Pester Tests
  inputs:
      testResultsFormat: 'NUnit'
      testResultsFiles: '**/TEST-*.xml'
  condition: succeededOrFailed()