Az.Accounts cannot be imported in parallel using ForEach-Object -Parallel

[sharpjs]

Description

When Az.Accounts is imported in parallel using ForEach-Object -Parallel, one or more of the imports fail intermittently with the error "AzKeyStore is locked. Please try again after a while."

This issue seems similar to closed issues #19624 and #18321, but still occurs in the latest Az (9.2.0).

Issue script & Debug output

PS> $DebugPreference='Continue'
PS> 0..9 | ForEach-Object -Parallel { Import-Module Az.Accounts }
Import-Module: AzKeyStore is locked. Please try again after a while.
Import-Module: AzKeyStore is locked. Please try again after a while.
Import-Module: AzKeyStore is locked. Please try again after a while.

Environment data

PS> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.3.1
PSEdition                      Core
GitCommitId                    7.3.1
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

PS> Get-Module -ListAvailable

    Directory: C:\Users\Jeff\Documents\PowerShell\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     9.2.0                 Az                                  Core,Desk
Script     2.10.4                Az.Accounts                         Core,Desk {Disable-AzDataCollec…
Script     2.0.0                 Az.Advisor                          Core,Desk {Disable-AzAdvisorRec…
Script     5.1.0                 Az.Aks                              Core,Desk {Get-AzAksCluster, Ne…
Script     1.1.4                 Az.AnalysisServices                 Core,Desk {Resume-AzAnalysisSer…
Script     4.0.1                 Az.ApiManagement                    Core,Desk {Add-AzApiManagementA…
Script     1.2.0                 Az.AppConfiguration                 Core,Desk {Get-AzAppConfigurati…
Script     2.2.0                 Az.ApplicationInsights              Core,Desk {Get-AzApplicationIns…
Script     2.0.0                 Az.Attestation                      Core,Desk {Get-AzAttestationPol…
Script     1.8.0                 Az.Automation                       Core,Desk {Move-AzAutomationHyb…
Script     3.2.1                 Az.Batch                            Core,Desk {Remove-AzBatchAccoun…
Script     2.0.0                 Az.Billing                          Core,Desk {Get-AzBillingInvoice…
Script     2.1.0                 Az.Cdn                              Core,Desk {Clear-AzCdnEndpointC…
Script     1.1.0                 Az.CloudService                     Core,Desk {Get-AzCloudService, …
Script     1.12.0                Az.CognitiveServices                Core,Desk {Get-AzCognitiveServi…
Script     5.2.0                 Az.Compute                          Core,Desk {Remove-AzAvailabilit…
Script     1.0.0                 Az.ConfidentialLedger               Core,Desk {Get-AzConfidentialLe…
Script     3.1.0                 Az.ContainerInstance                Core,Desk {Add-AzContainerInsta…
Script     3.0.0                 Az.ContainerRegistry                Core,Desk {New-AzContainerRegis…
Script     1.9.0                 Az.CosmosDB                         Core,Desk {Get-AzCosmosDBSqlCon…
Script     1.1.0                 Az.DataBoxEdge                      Core,Desk {Get-AzDataBoxEdgeJob…
Script     1.4.0                 Az.Databricks                       Core,Desk {Get-AzDatabricksAcce…
Script     1.16.11               Az.DataFactory                      Core,Desk {Set-AzDataFactoryV2,…
Script     1.0.2                 Az.DataLakeAnalytics                Core,Desk {Get-AzDataLakeAnalyt…
Script     1.3.0                 Az.DataLakeStore                    Core,Desk {Get-AzDataLakeStoreT…
Script     1.0.1                 Az.DataProtection                   Core,Desk {Backup-AzDataProtect…
Script     1.0.1                 Az.DataShare                        Core,Desk {New-AzDataShareAccou…
Script     1.1.0                 Az.DeploymentManager                Core,Desk {Get-AzDeploymentMana…
Script     3.1.1                 Az.DesktopVirtualization            Core,Desk {Disconnect-AzWvdUser…
Script     1.0.2                 Az.DevTestLabs                      Core,Desk {Get-AzDtlAllowedVMSi…
Script     1.1.2                 Az.Dns                              Core,Desk {Get-AzDnsRecordSet, …
Script     1.5.0                 Az.EventGrid                        Core,Desk {New-AzEventGridTopic…
Script     3.2.0                 Az.EventHub                         Core,Desk {New-AzEventHubNamesp…
Script     1.9.0                 Az.FrontDoor                        Core,Desk {New-AzFrontDoor, Get…
Script     4.0.6                 Az.Functions                        Core,Desk {Get-AzFunctionApp, G…
Script     5.0.1                 Az.HDInsight                        Core,Desk {Get-AzHDInsightJob, …
Script     2.0.0                 Az.HealthcareApis                   Core,Desk {Get-AzHealthcareApis…
Script     2.7.4                 Az.IotHub                           Core,Desk {Add-AzIotHubKey, Get…
Script     4.9.1                 Az.KeyVault                         Core,Desk {Add-AzKeyVaultCertif…
Script     2.1.0                 Az.Kusto                            Core,Desk {Add-AzKustoClusterLa…
Script     1.5.0                 Az.LogicApp                         Core,Desk {Get-AzIntegrationAcc…
Script     1.1.3                 Az.MachineLearning                  Core,Desk {Move-AzMlCommitmentA…
Script     1.2.1                 Az.Maintenance                      Core,Desk {Get-AzApplyUpdate, G…
Script     1.1.0                 Az.ManagedServiceIdentity           Core,Desk {Get-AzFederatedIdent…
Script     3.0.0                 Az.ManagedServices                  Core,Desk {Get-AzManagedService…
Script     2.0.0                 Az.MarketplaceOrdering              Core,Desk {Get-AzMarketplaceTer…
Script     1.1.1                 Az.Media                            Core,Desk {Sync-AzMediaServiceS…
Script     2.1.0                 Az.Migrate                          Core,Desk {Get-AzMigrateDiscove…
Script     4.3.0                 Az.Monitor                          Core,Desk {Get-AzMetricDefiniti…
Script     1.1.0                 Az.MySql                            Core,Desk {Get-AzMySqlConfigura…
Script     5.2.0                 Az.Network                          Core,Desk {Add-AzApplicationGat…
Script     1.1.1                 Az.NotificationHubs                 Core,Desk {Get-AzNotificationHu…
Script     3.2.0                 Az.OperationalInsights              Core,Desk {New-AzOperationalIns…
Script     1.5.1                 Az.PolicyInsights                   Core,Desk {Get-AzPolicyEvent, G…
Script     1.1.0                 Az.PostgreSql                       Core,Desk {Get-AzPostgreSqlConf…
Script     1.2.0                 Az.PowerBIEmbedded                  Core,Desk {Remove-AzPowerBIWork…
Script     1.0.3                 Az.PrivateDns                       Core,Desk {Get-AzPrivateDnsZone…
Script     6.1.2                 Az.RecoveryServices                 Core,Desk {Get-AzRecoveryServic…
Script     1.6.0                 Az.RedisCache                       Core,Desk {Remove-AzRedisCacheP…
Script     1.1.0                 Az.RedisEnterpriseCache             Core,Desk {Export-AzRedisEnterp…
Script     1.0.3                 Az.Relay                            Core,Desk {New-AzRelayNamespace…
Script     1.1.0                 Az.ResourceMover                    Core,Desk {Add-AzResourceMoverM…
Script     6.5.0                 Az.Resources                        Core,Desk {Get-AzProviderOperat…
Script     1.3.0                 Az.Security                         Core,Desk {Get-AzSecuritySoluti…
Script     3.0.0                 Az.SecurityInsights                 Core,Desk {Get-AzSentinelAlertR…
Script     2.1.0                 Az.ServiceBus                       Core,Desk {New-AzServiceBusName…
Script     3.1.0                 Az.ServiceFabric                    Core,Desk {Add-AzServiceFabricC…
Script     1.5.0                 Az.SignalR                          Core,Desk {New-AzSignalR, Get-A…
Script     4.1.0                 Az.Sql                              Core,Desk {Get-AzSqlDatabaseTra…
Script     1.1.0                 Az.SqlVirtualMachine                Core,Desk {New-AzSqlVM, Get-AzS…
Script     1.4.0                 Az.StackHCI                         Core,Desk {Add-AzStackHCIVMAtte…
Script     5.2.0                 Az.Storage                          Core,Desk {Get-AzStorageAccount…
Script     1.7.0                 Az.StorageSync                      Core,Desk {Invoke-AzStorageSync…
Script     2.0.0                 Az.StreamAnalytics                  Core,Desk {Get-AzStreamAnalytic…
Script     1.0.0                 Az.Support                          Core,Desk {Get-AzSupportService…
Script     2.2.0                 Az.Synapse                          Core,Desk {Get-AzSynapseSparkJo…
Script     1.1.0                 Az.TrafficManager                   Core,Desk {Add-AzTrafficManager…
Script     2.12.0                Az.Websites                         Core,Desk {Get-AzAppServicePlan…
Script     1.1.0                 posh-git                            Desk      {Add-PoshGitToProfile…
Binary     2.0.0                 PSConcurrent                        Core,Desk Invoke-Concurrent
Binary     2.0.1                 PSql                                Core      {New-SqlContext, Conn…
Script     2.0.0      pre07      PSql.Deploy                         Core      {Find-SqlMigrations, …

    Directory: C:\program files\powershell\7\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Manifest   7.0.0.0               CimCmdlets                          Core      {Get-CimAssociatedIns…
Manifest   1.2.5                 Microsoft.PowerShell.Archive        Desk      {Compress-Archive, Ex…
Manifest   7.0.0.0               Microsoft.PowerShell.Diagnostics    Core      {Get-WinEvent, New-Wi…
Manifest   7.0.0.0               Microsoft.PowerShell.Host           Core      {Start-Transcript, St…
Manifest   7.0.0.0               Microsoft.PowerShell.Management     Core      {Add-Content, Clear-C…
Manifest   7.0.0.0               Microsoft.PowerShell.Security       Core      {Get-Acl, Set-Acl, Ge…
Manifest   7.0.0.0               Microsoft.PowerShell.Utility        Core      {Export-Alias, Get-Al…
Manifest   7.0.0.0               Microsoft.WSMan.Management          Core      {Disable-WSManCredSSP…
Script     1.4.8.1               PackageManagement                   Desk      {Find-Package, Get-Pa…
Script     2.2.5                 PowerShellGet                       Desk      {Find-Command, Find-D…
Script     7.0.0.0               PSDiagnostics                       Core      {Disable-PSTrace, Dis…
Script     2.2.6                 PSReadLine                          Desk      {Get-PSReadLineKeyHan…
Binary     2.0.3                 ThreadJob                           Desk      Start-ThreadJob

    Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     1.0.1                 Microsoft.PowerShell.Operation.Val… Desk      {Get-OperationValidat…
Binary     1.0.0.1               PackageManagement                   Desk      {Find-Package, Get-Pa…
Script     3.4.0                 Pester                              Desk      {Describe, Context, I…
Script     1.0.0.1               PowerShellGet                       Desk      {Install-Module, Find…
Script     2.2.6                 PSReadLine                          Desk      {Get-PSReadLineKeyHan…
Script     2.0.0                 PSReadLine                          Desk      {Get-PSReadLineKeyHan…

    Directory: C:\Windows\system32\WindowsPowerShell\v1.0\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Manifest   1.0.0.0               AppBackgroundTask                   Core,Desk {Disable-AppBackgroun…
Script     1.0.0.0               AssignedAccess                      Core,Desk {Clear-AssignedAccess…
Manifest   1.0.0.0               BitLocker                           Core,Desk {Unlock-BitLocker, Su…
Script     2.0.0.0               BitsTransfer                        Core,Desk {Add-BitsFile, Comple…
Manifest   1.0.0.0               BranchCache                         Core,Desk {Add-BCDataCacheExten…
Manifest   1.0                   ConfigDefenderPerformance           Core,Desk {New-MpPerformanceRec…
Manifest   1.0                   DefenderPerformance                 Core,Desk {New-MpPerformanceRec…
Manifest   1.0.3.0               DeliveryOptimization                Core,Desk {Get-DeliveryOptimiza…
Manifest   1.0.0.0               DirectAccessClientComponents        Core,Desk {Disable-DAManualEntr…
Script     3.0                   Dism                                Core,Desk {Add-AppProvisionedSh…
Manifest   1.0.0.0               DnsClient                           Core,Desk {Resolve-DnsName, Cle…
Manifest   1.0.0.0               EventTracingManagement              Core,Desk {Start-EtwTraceSessio…
Script     2022.3.14…            Get-NetView                         Core,Desk Get-NetView
Manifest   2.1.0.0               International                       Core,Desk {Copy-UserInternation…
Manifest   1.0.0.0               Kds                                 Core,Desk {Add-KdsRootKey, Get-…
Manifest   1.0                   LanguagePackManagement              Core,Desk {Install-Language, Un…
Manifest   1.0.0.0               Microsoft.PowerShell.LocalAccounts  Core,Desk {Add-LocalGroupMember…
Manifest   1.0.0                 Microsoft.Windows.Bcd.Cmdlets       Core,Desk {Copy-BcdEntry, Disab…
Manifest   1.0                   MMAgent                             Core,Desk {Disable-MMAgent, Ena…
Manifest   2.0.0.0               NetAdapter                          Core,Desk {Disable-NetAdapter, …
Manifest   2.0.0.0               NetConnection                       Core,Desk {Get-NetConnectionPro…
Manifest   1.0.0.0               NetEventPacketCapture               Core,Desk {New-NetEventSession,…
Manifest   2.0.0.0               NetLbfo                             Core,Desk {Add-NetLbfoTeamMembe…
Manifest   1.0.0.0               NetNat                              Core,Desk {Get-NetNat, Get-NetN…
Manifest   2.0.0.0               NetQos                              Core,Desk {Get-NetQosPolicy, Se…
Manifest   2.0.0.0               NetSecurity                         Core,Desk {Get-DAPolicyChange, …
Manifest   1.0.0.0               NetSwitchTeam                       Core,Desk {New-NetSwitchTeam, R…
Manifest   1.0.0.0               NetTCPIP                            Core,Desk {Get-NetIPAddress, Ge…
Manifest   1.0.0.0               NetworkConnectivityStatus           Core,Desk {Get-DAConnectionStat…
Manifest   1.0.0.0               NetworkSwitchManager                Core,Desk {Disable-NetworkSwitc…
Manifest   1.0.0.0               NetworkTransition                   Core,Desk {Add-NetIPHttpsCertBi…
Manifest   1.0.0.0               PcsvDevice                          Core,Desk {Get-PcsvDevice, Star…
Manifest   1.0.0.0               PKI                                 Core,Desk {Add-CertificateEnrol…
Manifest   1.0.0.0               PnpDevice                           Core,Desk {Get-PnpDevice, Get-P…
Manifest   1.1                   PrintManagement                     Core,Desk {Add-Printer, Add-Pri…
Binary     1.0.12                ProcessMitigations                  Core,Desk {Get-ProcessMitigatio…
Script     3.0                   Provisioning                        Core,Desk {Install-Provisioning…
Manifest   1.0.0.0               ScheduledTasks                      Core,Desk {Get-ScheduledTask, S…
Manifest   2.0.0.0               SecureBoot                          Core,Desk {Confirm-SecureBootUE…
Manifest   2.0.0.0               SmbShare                            Core,Desk {Get-SmbShare, Remove…
Manifest   2.0.0.0               SmbWitness                          Core,Desk {Get-SmbWitnessClient…
Manifest   1.0.0.1               StartLayout                         Core,Desk {Export-StartLayout, …
Manifest   2.0.0.0               Storage                             Core,Desk {Add-InitiatorIdToMas…
Manifest   2.0.0.0               TLS                                 Core,Desk {New-TlsSessionTicket…
Manifest   1.0.0.0               TroubleshootingPack                 Core,Desk {Get-TroubleshootingP…
Manifest   2.0.0.0               TrustedPlatformModule               Core,Desk {Get-Tpm, Initialize-…
Binary     2.1.639.0             UEV                                 Core,Desk
Manifest   2.0.0.0               VpnClient                           Core,Desk {Add-VpnConnection, S…
Manifest   1.0.0.0               Wdac                                Core,Desk {Get-OdbcDriver, Set-…
Manifest   2.0.0.0               Whea                                Core,Desk {Get-WheaMemoryPolicy…
Manifest   1.0.0.0               WindowsDeveloperLicense             Core,Desk {Get-WindowsDeveloper…
Script     1.0                   WindowsErrorReporting               Core,Desk {Enable-WindowsErrorR…
Manifest   1.0.0.0               WindowsSearch                       Core,Desk {Get-WindowsSearchSet…
Manifest   1.0.0.0               WindowsUpdate                       Core,Desk Get-WindowsUpdateLog

    Directory: C:\Program Files (x86)\Microsoft SQL Server\150\Tools\PowerShell\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Manifest   15.0                  SQLPS                               Desk

Error output

PS> Resolve-AzError
DEBUG: Registering Az shared AssemblyLoadContext for path: 'C:\Users\Jeff\Documents\PowerShell\Modules\Az.Accounts\2.10.4\StartupScripts\..\AzSharedAlcAssemblies'.
DEBUG: AssemblyLoadContext registered.
WARNING: INITIALIZATION: Fallback context save mode to process because of error during checking token cache persistence: Persistence check fails due to unknown error.
WARNING: INITIALIZATION: Fallback context save mode to process because of error during checking token cache persistence: Persistence check fails due to unknown error.
DEBUG: Got version 0 of Az
DEBUG: Got version 0 of Az.Accounts
DEBUG: 10:50:09 AM - ResolveError begin processing with ParameterSet 'AnyErrorParameterSet'.
DEBUG: 10:50:09 AM - using account id '(redacted)@(redacted).com'...
DEBUG: 10:50:09 AM - [ConfigManager] Got [False] from [DisplayBreakingChangeWarning], Module = [], Cmdlet = [].

   HistoryId: 1

Message        : AzKeyStore is locked. Please try again after a while.
StackTrace     :    at Microsoft.Azure.Commands.ResourceManager.Common.StorageWrapper.Create()
                    at Microsoft.Azure.Commands.ResourceManager.Common.AzKeyStore..ctor(String
                 directory, String fileName, Boolean loadStorage, Boolean autoSaveEnabled, IStorage
                 inputStorage)
                    at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.OnImport()
                    at System.Management.Automation.Runspaces.PSSnapInHelpers.ExecuteModuleInitialize
                 r(Assembly assembly, IEnumerable`1 assemblyTypes)
                    at System.Management.Automation.Runspaces.PSSnapInHelpers.AnalyzeModuleAssemblyWi
                 thReflection(Assembly assembly, String name, PSSnapInInfo psSnapInInfo,
                 PSModuleInfo moduleInfo, String helpFile, Dictionary`2& cmdlets, Dictionary`2&
                 aliases, Dictionary`2& providers)
                    at System.Management.Automation.Runspaces.PSSnapInHelpers.AnalyzePSSnapInAssembly
                 (Assembly assembly, String name, PSSnapInInfo psSnapInInfo, PSModuleInfo
                 moduleInfo, Dictionary`2& cmdlets, Dictionary`2& aliases, Dictionary`2& providers,
                 String& helpFile)
                    at System.Management.Automation.Runspaces.InitialSessionState.ImportCmdletsFromAs
                 sembly(Assembly assembly, PSModuleInfo module)
                    at Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadBinaryModule(PSModuleInfo
                 parentModule, String moduleName, String fileName, Assembly assemblyToLoad, String
                 moduleBase, SessionState ss, ImportModuleOptions options, ManifestProcessingFlags
                 manifestProcessingFlags, String prefix, Boolean& found, String shortModuleName,
                 Boolean disableFormatUpdates)
                    at Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModule(PSModuleInfo
                 parentModule, String fileName, String moduleBase, String prefix, SessionState ss,
                 Object privateData, ImportModuleOptions& options, ManifestProcessingFlags
                 manifestProcessingFlags, Boolean& found, Boolean& moduleFileFound)
                    at Microsoft.PowerShell.Commands.ImportModuleCommand.ImportModule_LocallyViaName(
                 ImportModuleOptions importModuleOptions, String name)
                    at Microsoft.PowerShell.Commands.ImportModuleCommand.ImportModule_LocallyViaName_
                 WithTelemetry(ImportModuleOptions importModuleOptions, String name)
                    at Microsoft.PowerShell.Commands.ImportModuleCommand.ProcessRecord()
                    at System.Management.Automation.CommandProcessor.ProcessRecord()
Exception      : System.InvalidOperationException
InvocationInfo : {Import-Module}
Line           : Import-Module (Join-Path -Path $PSScriptRoot -ChildPath
                 Microsoft.Azure.PowerShell.Cmdlets.Accounts.dll)

Position       : At
                 C:\Users\Jeff\Documents\PowerShell\Modules\Az.Accounts\2.10.4\Az.Accounts.psm1:117
                 char:1
                 + Import-Module (Join-Path -Path $PSScriptRoot -ChildPath Microsoft.Azu …
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 1


The Azure PowerShell team is listening, please let us know how we are doing: https://aka.ms/azpssurvey?Q_CHL=ERROR.

DEBUG: AzureQoSEvent:  Module: Az.Accounts:2.10.4; CommandName: Resolve-AzError; PSVersion: 7.3.1; IsSuccess: True; Duration: 00:00:00.1936506
DEBUG: 10:50:09 AM - [ConfigManager] Got [False] from [EnableDataCollection], Module = [], Cmdlet = [].
DEBUG: 10:50:09 AM - ResolveError end processing.

EDIT: Removed duplicated word.

[dingmeng-xue]

Thanks for importing. We will look into it.

[msJinLei]

@sharpjs How often does the issue occur? I cannot reproduce every time I run the script you provide.
I would like to know the actual frequency you meet the issue when using the real script (the script should not only contain Import-Module Az.Account)

[HaikoHertes]

It also happens to me. @msJinLei : For me on average it happens about every 3rd to 5th time I would say... But I also would say either it happens multiple times one after each other or it does not happen for a while...

[sharpjs]

@sharpjs How often does the issue occur?

@msJinLei I would estimate somewhere between 30% to 95% of invocations, depending on the number of parallel tasks to do. Higher parallelism increases the probability. I have reproduced the issue on a 16-core processor, on a 6-core processor, and on an Azure DevOps hosted agent.

I would like to know the actual frequency you meet the issue when using the real script (the script should not only contain Import-Module Az.Account)

I estimate 25% to 90% of invocations using the real script. Here is one excerpt from the real script:

$ToCreate | ForEach-Object -Parallel {
    $ErrorActionPreference = "Stop"
    Set-StrictMode -Version 3.0

    Import-Module Az.Sql

    # ... everything else ...
}

The real inner script begins by loading the modules it needs. Thus module loading is being done in parallel, regardless of the code in the "everything else" category.

EDIT: typo form → from