Azure Bicep Deployment Issues #26752

oZakari · 2024-11-23T01:46:02Z

Description

Bug Report: Azure PowerShell Version 13.0.0 Causes Deployment Failures in ALZ-Bicep

Description:
Using Azure PowerShell version 13.0.0 has led to multiple reports of ALZ-Bicep deployment failures in both Azure DevOps pipelines and GitHub Actions workflows, where the latest version is specified. The issue is also reproducible in local environments.

The problem appears to impact specific modules within the ALZ-Bicep framework. In particular, I have confirmed failures when deploying the ALZ Default Policy Assignments module.

The following error is encountered during deployment:
Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject.

Note: For the failing deployment, we do load in a json file using the Bicep function: loadJsonContent

Workaround:
Pinning Azure PowerShell to version 12.5.0 or 12.4.0 resolves the issue.

Impact:
This issue affects deployments in:

Azure DevOps pipelines
GitHub Actions workflows
Local environments

Initial Reports of Error Here: Azure/ALZ-Bicep#907

Issue script & Debug output

New-AzManagementGroupDeployment @inputObject
VERBOSE: Using Bicep v0.31.92
VERBOSE: Calling Bicep with arguments: build "C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\assignments\alzDefaults\alzDefaultPolicyAssignments.bicep" --stdout
WARNING: C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\CRML\customerUsageAttribution\cuaIdManagementGroup.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentManagementGroup.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\assignments\policyAssignmentManagementGroup.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\assignments\alzDefaults\alzDefaultPolicyAssignments.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\assignments\alzDefaults\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentResourceGroup.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\exemptions\policyExemptions.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentResourceGroupMany.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentManagementGroupMany.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentSubscriptionMany.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentSubscription.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\CRML\customerUsageAttribution\cuaIdSubscription.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).


WARNING: The DeploymentDebug setting has been enabled. This can potentially log secrets like passwords used in resource property or listKeys operations when you retrieve the deployment operations through Get-AzResourceGroupDeploymentOperation
New-AzManagementGroupDeployment: 
Line |
  11 |  New-AzManagementGroupDeployment @inputObject
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | 7:34:37 PM - Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject
New-AzManagementGroupDeployment: 
Line |
  11 |  New-AzManagementGroupDeployment @inputObject
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The deployment validation failed

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.6
PSEdition                      Core
GitCommitId                    7.4.6
OS                             Microsoft Windows 10.0.26100
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

Version    Name            Repository           Description
-------    ----                                ----------
13.0.0     Az              PSGallery            Microsoft Azure PowerShell - Cmdlets to …

Error output

Message        : The deployment validation failed
StackTrace     :    at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.ResourceManagerCmdletBase.HandleException(ExceptionDispatchInfo capturedException)     
                    at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.ResourceManagerCmdletBase.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : System.InvalidOperationException
InvocationInfo : {New-AzManagementGroupDeployment}
Line           : New-AzManagementGroupDeployment @inputObject
Position       : At line:9 char:1
                 + New-AzManagementGroupDeployment @inputObject
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 8

Message        : 2:35:29 PM - Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject

StackTrace     : 
Exception      : System.Exception
InvocationInfo : {New-AzManagementGroupDeployment}
Line           : New-AzManagementGroupDeployment @inputObject
Position       : At line:9 char:1
                 + New-AzManagementGroupDeployment @inputObject
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 8

The text was updated successfully, but these errors were encountered:

msJinLei · 2024-11-25T02:03:12Z

@oZakari Please set

$DebugPreference='Continue'

before you run the cmdlet and collect the debug

oZakari · 2024-11-25T15:30:08Z

Hi @msJinLei, here you go. Above this are just bunch of other similar types of warnings for the policy deployments

{
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-sandbox/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-zhov76caaytik",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-eh6uinipiyc62",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-vasi3f2dclqii",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-o3ippdiysur7o",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-p7acceslthje6",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-w7ujuawx46ppo",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      }
    ]
  }
}


DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [EnableErrorRecordsPersistence], Module = [], Cmdlet = []. Returning default value [False].
New-AzManagementGroupDeployment: 
Line |
  10 |  New-AzManagementGroupDeployment @inputObject
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | 8:58:31 AM - Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 8:58:32 AM - [ResourceManagerCmdletBase.ExecuteCmdlet] Caught unhandled exception: System.InvalidOperationException: The deployment validation failed
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkClient.NewResourceManagerSdkClient.RunDeploymentValidation(PSDeploymentCmdletParameters parameters, Deployment deployment)
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkClient.NewResourceManagerSdkClient.ExecuteDeploymentInternal(PSDeploymentCmdletParameters parameters)
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkClient.NewResourceManagerSdkClient.ExecuteDeployment(PSDeploymentCmdletParameters parameters)
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.CmdletBase.DeploymentCreateCmdlet.ExecuteDeployment()
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.CmdletBase.DeploymentCreateCmdlet.OnProcessRecord()
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.ResourceManagerCmdletBase.ExecuteCmdlet()
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [EnableErrorRecordsPersistence], Module = [], Cmdlet = []. Returning default value [False].
New-AzManagementGroupDeployment: 
Line |
  10 |  New-AzManagementGroupDeployment @inputObject
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The deployment validation failed
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: AzureQoSEvent:  Module: Az.Resources:7.7.0; CommandName: New-AzManagementGroupDeployment; PSVersion: 7.4.6; IsSuccess: False; Duration: 00:00:25.5317021; SanitizeDuration: 00:00:00; Exception: The deployment validation failed;
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 8:58:32 AM - NewAzureManagementGroupDeploymentCmdlet end processing.
DEBUG: Setting WindowTitle: Admin: ALZ-Bicep [update-known-issues] - PowerShell 7.4 (34216)

alex-frankel · 2024-11-26T16:35:50Z

@msJinLei - based on the discussion in this thread, it looks like the issue was introduced in Azure PowerShell 13. Is it possible this was introduced due to a change in some of the core components in Azure PowerShell? I am not sure that we have made many (or any) changes to the New-Az*Deployment commands recently.

cc @jeskew / @anthony-c-martin

anthony-c-martin · 2024-11-26T22:14:10Z

@tsmallig33 - I think this is related to #26732.

Looks like the error being returned in the error ("Received unexpected type Newtonsoft.Json.Linq.JObject") is coming from this code path, though it's not clear why:

azure-powershell/src/Resources/ResourceManager/SdkClient/NewResourceManagerSdkClient.cs

Line 498 in 1bbb74d

    
           throw new InvalidOperationException($"Received unexpected type {validationResult.GetType()}");

msJinLei · 2024-11-27T20:50:59Z

@tsmallig33 - I think this is related to #26732.

Looks like the error being returned in the error ("Received unexpected type Newtonsoft.Json.Linq.JObject") is coming from this code path, though it's not clear why:

azure-powershell/src/Resources/ResourceManager/SdkClient/NewResourceManagerSdkClient.cs

Line 498 in 1bbb74d

throw new InvalidOperationException($"Received unexpected type {validationResult.GetType()}");

I believe the PR is not released yet. Please correct me if anything incorrect @VeryEarly

tsmallig33 · 2024-11-27T22:11:52Z

@tsmallig33 - I think this is related to #26732.
Looks like the error being returned in the error ("Received unexpected type Newtonsoft.Json.Linq.JObject") is coming from this code path, though it's not clear why:
azure-powershell/src/Resources/ResourceManager/SdkClient/NewResourceManagerSdkClient.cs
Line 498 in 1bbb74d
throw new InvalidOperationException($"Received unexpected type {validationResult.GetType()}");

I believe the PR is not released yet. Please correct me if anything incorrect @VeryEarly

I think this was the PR where the deployments SDK was updated which looks to be related: #26507

msJinLei · 2024-11-28T06:26:49Z

@tsmallig33
I think this was the PR where the deployments SDK was updated which looks to be related: #26507

We don't upgrade newtonsoft json library recently. Could you take a look?

* Fix Deployment Type Issue #27652 * Update to JObject * Update to only deserialize DeploymentExtended * Remove unused method --------- Co-authored-by: Tate Smalligan <[email protected]>

msJinLei · 2024-12-01T22:30:03Z

Fixed by #26776

slavizh · 2024-12-03T14:02:40Z

+1

Jigar191089 · 2024-12-04T11:39:41Z

Hi,

We have a similar error message only difference being we are using template spec. Support ticket 2411250050002160 is also raised for this from last 10 days. I see that a PR is merged to fix this. The PR contains changes in Az.Resources and it is version is updated to 7.7.0. I see that our deployment is still failing even after using az.resources of 7.7.0.

sgoyal100 · 2024-12-04T16:39:27Z

We are also facing the same issue as of yesterday (12/3/2024) for APP services, SQL Server and Front door deployment. Sometime rerun works, but most of the time it fails.
For now we reverted our module az powershell version to 11.4 to bypass this issue.

anthony-c-martin · 2024-12-09T21:45:31Z

I apologize for the problems this is causing. The fix has been checked in, and will be available in the next release of Azure PowerShell. @Azure/azure-powershell-team should be able to share an ETA. Until then, the only workaround I can recommend is to downgrade to the previous working version (12.5.0).

The problem is innately non-deterministic; it may succeed always, sometimes or never, but chance of failure is likely higher with more complex deployments.

slavizh · 2024-12-10T06:46:18Z

note you do not have to downgrade to 12.5.0 fully. You could downgrade az.resources to 7.6.0 only and all other modules to be on the latest.

oZakari added arm-bicep Dealing with bicep issues bug This issue requires a change to an existing behavior in the product in order to be resolved. needs-triage This is a new issue that needs to be triaged to the appropriate team. labels Nov 23, 2024

microsoft-github-policy-service bot removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Nov 23, 2024

oZakari added needs-triage This is a new issue that needs to be triaged to the appropriate team. and removed arm-bicep Dealing with bicep issues labels Nov 23, 2024

oZakari mentioned this issue Nov 23, 2024

Deployment Issues with Azure PowerShell version 13.0.0 Azure/ALZ-Bicep#907

Open

1 task

msJinLei added the ARM - Templates label Nov 25, 2024

microsoft-github-policy-service bot removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Nov 25, 2024

msJinLei added the Service Attention This issue is responsible by Azure service team. label Nov 25, 2024

tsmallig33 mentioned this issue Nov 27, 2024

Fix Deployment Type Issue #26752 #26776

Merged

6 tasks

slavizh mentioned this issue Dec 3, 2024

New-AzDeployment: 15:06:40 - Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject Azure/bicep#15751

Closed

isra-fel added this to the Az 13.1.0 (January 2025, 2025-01-14) milestone Dec 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure Bicep Deployment Issues #26752

Azure Bicep Deployment Issues #26752

oZakari commented Nov 23, 2024 •

edited

Loading

msJinLei commented Nov 25, 2024

oZakari commented Nov 25, 2024 •

edited

Loading

alex-frankel commented Nov 26, 2024

anthony-c-martin commented Nov 26, 2024

msJinLei commented Nov 27, 2024

tsmallig33 commented Nov 27, 2024

msJinLei commented Nov 28, 2024

msJinLei commented Dec 1, 2024

slavizh commented Dec 3, 2024

Jigar191089 commented Dec 4, 2024 •

edited

Loading

sgoyal100 commented Dec 4, 2024

anthony-c-martin commented Dec 9, 2024

slavizh commented Dec 10, 2024

Azure Bicep Deployment Issues #26752

Azure Bicep Deployment Issues #26752

Comments

oZakari commented Nov 23, 2024 • edited Loading

Description

Issue script & Debug output

Environment data

Module versions

Error output

msJinLei commented Nov 25, 2024

oZakari commented Nov 25, 2024 • edited Loading

alex-frankel commented Nov 26, 2024

anthony-c-martin commented Nov 26, 2024

msJinLei commented Nov 27, 2024

tsmallig33 commented Nov 27, 2024

msJinLei commented Nov 28, 2024

msJinLei commented Dec 1, 2024

slavizh commented Dec 3, 2024

Jigar191089 commented Dec 4, 2024 • edited Loading

sgoyal100 commented Dec 4, 2024

anthony-c-martin commented Dec 9, 2024

slavizh commented Dec 10, 2024

oZakari commented Nov 23, 2024 •

edited

Loading

oZakari commented Nov 25, 2024 •

edited

Loading

Jigar191089 commented Dec 4, 2024 •

edited

Loading