Two identical Azure deployments; one always throws 404 - how to find out what the difference is?

Question

I am deploying a C# ASP.NET Core web service to Azure using Pulumi. I can deploy it in 3 ways:

1. Run it locally from Visual Studio, i.e., not using Azure at all.
2. Deploy it to Azure from my local developer computer.
3. Deploy it to Azure from Jenkins (whicn runs on a different computer).

I have this problem:

1. When I run it locally, I can call the service fine, e.g. from Postman or from a C# application. The web service returns what I expect.
2. When I deploy it to Azure from my local machine, I can also call it fine. The web service returns what I expect.
3. When I deploy it to Azure from Jenkins and then try to call the webservice, it returns "NotFound" to all calls no matter what I do. (This presumably means HTTP 404.)

The deployments in 2 and 3 should be exactly the same. My question is: How can I find out what the difference is between these two deployments in Azure?

The Jenkins-deployed webservice exhibits the following curious behaviour:

• It does not log any exceptions (even when I wait several minutes for them to show up).
• If I go to my resource group -> Application Insights -> Logs and search for "requests", it does list requests. Curiously, it says that it returned HTTP 200 to all the requests, even though what I get when calling them is 404.
• The above is true even for web service calls that should never return 200 (they should return 201).
• The above is true even for web service calls to methods that shouldn't even exist (i.e., when I deliberately corrupt the method URI before calling the service).

During deployment I authenticate with Azure using a service principal. My Jenkinsfile looks like this:

withVaultSecrets([                                    
    "path/to/secret/in/vault": [
        "sp_name", "application_id", "object_id", "sp_secret"
    ]
]){
    script {
        env.PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
        env.ARM_CLIENT_ID = "${application_id}"
        env.ARM_CLIENT_SECRET = "${sp_secret}"
        env.ARM_TENANT_ID = "${azure_dev_tenant_id}"
        env.ARM_SUBSCRIPTION_ID = "${azure_dev_subscription_id}"
        env.AZURE_CLIENT_ID = "${application_id}"
        env.AZURE_CLIENT_SECRET = "${sp_secret}"
        env.AZURE_TENANT_ID = "${azure_dev_tenant_id}"
    }//script
    dir("./src/deploy/KmsStack"){
        powershell "pulumi login --local";
        powershell "pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase"
        powershell "pulumi up --yes"
    }//dir
}//withVaultSecrets

The script which I use to deploy locally looks like this, with the same service principal credentials:

cd $PSScriptRoot
cd webapi
dotnet publish /p:DisableGitVersionTask=true
cd ../deploy/KmsStack
$env:PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
$env:ARM_CLIENT_ID = ...
$env:ARM_CLIENT_SECRET = ...
$env:ARM_TENANT_ID = ...
$env:ARM_SUBSCRIPTION_ID = ...
$env:AZURE_CLIENT_ID = ...
$env:AZURE_CLIENT_SECRET = ...
$env:AZURE_TENANT_ID = ...
pulumi logout
pulumi login --local
pulumi stack rm jenkinsfunctionaltest -y
pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase
pulumi stack select jenkinsfunctionaltest
pulumi up --yes

How can I find out why these two deployed services behave differently? The Azure portal GUI is rich and has lots of sections. Can you recommend me where to look? Might there be some security settings that differ? How can I find them?

Thanks in advance!

Answer 1

We found out what was wrong. It was not an Azure issue. The problem was that we were deploying a bad ZIP file. The ZIP file was missing web.config, which meant that the web application could not start up.

We were zipping our published web application by having this in the CSPROJ file:

<Target Name="ZipOutputPath" AfterTargets="Publish">
  <ZipDirectory SourceDirectory="$(OutputPath)\publish" DestinationFile="$(MSBuildProjectDirectory)\kmswebapp.zip" Overwrite="true" />
</Target>

This turned out not to work because the compiler does things in a different order than we expected. At the time when it generated the ZIP file, web.config was not generated yet, so web.config never got packed into the ZIP file. Hence Azure could not start the application.

When we deployed from our local machines, it worked because we didn't clean the publish directory before each run, so there would be a web.config left over from the previous run, and this old (but unchanged) web.config would get packed into the ZIP file and deployed to Azure, so Azure would know how to start the application.

We solved it by removing the above from our CSPROJ file and doing (roughly) this in our Jenkinsfile:

powershell "dotnet publish ./src/webapi/WebAPI.csproj"
powershell "if (!(Test-Path('${publishDirectoryPath}/web.config'))){throw 'We need web.config to exist in the publish directory'}"
powershell "Compress-Archive -Path '${publishDirectoryPath}/*' -DestinationPath './src/webapi/kmswebapp.zip' -Force"

This generates a proper ZIP file including web.config, and Azure can now start our application so it can respond properly to requests.