AzureRm.HDInsight cmdlets are giving errors;

[deepakshakya]

Cmdlet(s)

AzureRM.HDInsight

PowerShell Version

$PSVersionTable

Name Value

---

PSVersion 6.0.0-rc.2
PSEdition Core
GitCommitId v6.0.0-rc.2
OS Linux 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

Module Version

ModuleType Version Name ExportedCommands

---

Script 4.0.1 AzureRM.HDInsight {Get-AzureRmHDInsightJob, New-AzureRmHDInsightSqoopJobDefinition, Wait-AzureRmHDInsightJob, New-AzureRmHDInsightStreamingMapReduceJobDefinition...}

OS Version

Linux 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017

Description

Not able to get output using any of the HDInsights cmdlets.
Tried all possible permutations wrt powershell versions, cmdlet versions, mac, Linux etc.
Just it does not seem to be working!

PS /opt/microsoft/powershell/6.0.0-rc.2/Modules> Import-Module AzureRM.HDInsight
Import-Module : The specified module 'AzureRM.Profile' with version '4.0.0' was not loaded because no valid module file was found in any module directory.
At /opt/microsoft/powershell/6.0.0-rc.2/Modules/AzureRM.HDInsight/4.0.1/AzureRM.HDInsight.psm1:19 char:5

• Import-Module AzureRM.Profile -MinimumVersion 4.0.0 -Scope Global
  

• ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

• CategoryInfo : ResourceUnavailable: (AzureRM.Profile:String) [Import-Module], FileNotFoundException
• FullyQualifiedErrorId : Modules_ModuleWithVersionNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand

Import-Module : Could not load file or assembly 'Microsoft.Azure.Commands.ResourceManager.Common, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
At /opt/microsoft/powershell/6.0.0-rc.2/Modules/AzureRM.HDInsight/4.0.1/AzureRM.HDInsight.psm1:21 char:1

• Import-Module (Join-Path -Path $PSScriptRoot -ChildPath Microsoft.Azu ...
• CategoryInfo : NotSpecified: (:) [Import-Module], FileNotFoundException
• FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.ImportModuleCommand

PS /opt/microsoft/powershell/6.0.0-rc.2/Modules>

** It asks for AzureRM.profile
I installed it and then that module is not able to import.

PS /opt/microsoft/powershell/6.0.0-rc.2/Modules> Import-Module AzureRM.profile -RequiredVersion 4.0.0
Import-Module : The specified module 'AzureRM.profile' with version '4.0.0' was not loaded because no valid module file was found in any module directory.
At line:1 char:1

• Import-Module AzureRM.profile -RequiredVersion 4.0.0
• CategoryInfo : ResourceUnavailable: (AzureRM.profile:String) [Import-Module], FileNotFoundException
• FullyQualifiedErrorId : Modules_ModuleWithVersionNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand

PS /opt/microsoft/powershell/6.0.0-rc.2/Modules> ls -l
total 44
drwxr-xr-x 3 root root 4096 Jan 6 23:29 AzureRM.HDInsight
drwxr-xr-x 3 root root 4096 Jan 6 23:33 AzureRM.profile
drwxr-xr-x 3 root root 4096 Jan 6 21:28 Microsoft.PowerShell.Archive
drwxr-xr-x 2 root root 4096 Jan 6 21:28 Microsoft.PowerShell.Host
drwxr-xr-x 2 root root 4096 Jan 6 21:28 Microsoft.PowerShell.Management
drwxr-xr-x 2 root root 4096 Jan 6 21:28 Microsoft.PowerShell.Security
drwxr-xr-x 2 root root 4096 Jan 6 21:28 Microsoft.PowerShell.Utility
drwxr-xr-x 3 root root 4096 Jan 6 21:28 PackageManagement
drwxr-xr-x 3 root root 4096 Jan 6 21:28 PowerShellGet
drwxr-xr-x 2 root root 4096 Jan 6 21:28 PSDesiredStateConfiguration
drwxr-xr-x 2 root root 4096 Jan 6 21:28 PSReadLine
PS /opt/microsoft/powershell/6.0.0-rc.2/Modules> $env:PSModulePath
/root/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/opt/microsoft/powershell/6.0.0-rc.2/Modules
PS /opt/microsoft/powershell/6.0.0-rc.2/Modules>

Debug Output

Instructions: to get Debug Output, set $DebugPreference="Continue" and then execute the cmdlet or script causing the issue

PS /home/gyana> $DebugPreference="Continue"
PS /home/gyana> Install-Module -Name AzureRM.profile
DEBUG: 00:00:00.0000002 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:00.0000499 Name: AzureRM.profile
DEBUG: 00:00:00.0008420 INVOKING PowerShell Fn Get-DynamicOptions with args Provider that has length 1
DEBUG: 00:00:00.0014652 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:00.0034276 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:00.0079107 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:00.0079664 Name: AzureRM.profile
DEBUG: 00:00:00.0086061 INVOKING PowerShell Fn Get-DynamicOptions with args Source that has length 1
DEBUG: 00:00:00.0091912 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:00.0125073 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:00.0166854 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:00.0167613 Name: AzureRM.profile
DEBUG: 00:00:00.0174239 INVOKING PowerShell Fn Get-DynamicOptions with args Package that has length 1
DEBUG: 00:00:00.0180240 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:00.0249125 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:00.0305822 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:00.0306301 Name: AzureRM.profile
DEBUG: 00:00:00.0312601 INVOKING PowerShell Fn Get-DynamicOptions with args Install that has length 1
DEBUG: 00:00:00.0319207 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:00.0375774 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:00.0411658 Calling SearchForPackages. Name='AzureRM.profile'
DEBUG: 00:00:00.0425467 PackageProvider::FindPackage with name AzureRM.profile
DEBUG: 00:00:00.0427438 Calling SearchForPackages After Select 1
DEBUG: 00:00:00.0457408 Calling New() : MethodName = 'FindPackage'
DEBUG: 00:00:00.0462410 MessageResolver: Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:00.0467373 Type: Module
DEBUG: 00:00:00.0471679 Name: AzureRM.profile
DEBUG: 00:00:00.0475996 AllowPrereleaseVersions: False
DEBUG: 00:00:00.0480300 ProviderName: PowerShellGet
DEBUG: 00:00:00.0484782 Scope: AllUsers
DEBUG: 00:00:00.0494541 INVOKING PowerShell Fn Find-Package with args System.String[], , , that has length 4
DEBUG: 00:00:00.0505233 In PowerShellGet Provider - 'Find-Package'.
DEBUG: 00:00:00.0512509 OPTION: MessageResolver => Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:00.0517948 OPTION: Type => Module
DEBUG: 00:00:00.0522591 OPTION: Name => AzureRM.profile
DEBUG: 00:00:00.0527466 OPTION: AllowPrereleaseVersions => False
DEBUG: 00:00:00.0532404 OPTION: ProviderName => PowerShellGet
DEBUG: 00:00:00.0537091 OPTION: Scope => AllUsers
DEBUG: 00:00:00.0608043 PackageProvider::FindPackage with name AzureRM.profile
DEBUG: 00:00:00.0684524 Calling 'NuGet'::'FindPackage' - name='AzureRM.profile', requiredVersion='',minimumVersion='', maximumVersion='''.
DEBUG: 00:00:00.0692404 Iterating 'AzureRM.profile'.
DEBUG: 00:00:00.0700629 There are '0' registered sources in 'NuGet' provider.
DEBUG: 00:00:00.0707996 Source 'https://www.powershellgallery.com/api/v2/' is not one of the registered sources in 'NuGet' provider.
DEBUG: 00:00:01.1059298 Source 'https://www.powershellgallery.com/api/v2/' is validated.
DEBUG: 00:00:01.1072102 Calling 'NuGetRequest'::'GetPackageById', 'AzureRM.profile'.
DEBUG: 00:00:01.3032151 Calling 'HttpClientPackageRepository'::'FindPackagesById', 'AzureRM.profile'.
DEBUG: 00:00:01.3041317 Calling 'NuGetClient'::'FindPackage'.
DEBUG: 00:00:01.3056218 Downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.profile'&$skip=0&$top=40'.
DEBUG: 00:00:01.8259222 Completed downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.profile'&$skip=0&$top=40'.
DEBUG: 00:00:01.8299347 '39' packages received in the last request.
DEBUG: 00:00:01.8333575 Completed iterating for 'AzureRM.profile'.
DEBUG: 00:00:01.8450940 Done calling powershell «Find-Package» «PSModule»
DEBUG: 00:00:01.8546595 Calling New() : MethodName = 'GetInstalledPackages'
DEBUG: 00:00:01.8550208 MessageResolver: Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:01.8554109 Type: Module
DEBUG: 00:00:01.8557337 Name: AzureRM.profile
DEBUG: 00:00:01.8560067 AllowPrereleaseVersions: False
DEBUG: 00:00:01.8563212 ProviderName: PowerShellGet
DEBUG: 00:00:01.8566383 Scope: AllUsers
DEBUG: 00:00:01.8576183 INVOKING PowerShell Fn Get-InstalledPackage with args AzureRM.profile, 4.1.1, , that has length 4
DEBUG: 00:00:01.8590570 In PowerShellGet Provider - 'Get-InstalledPackage'.
DEBUG: 00:00:01.8596408 OPTION: MessageResolver => Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:01.8600937 OPTION: Type => Module
DEBUG: 00:00:01.8635067 OPTION: Name => AzureRM.profile
DEBUG: 00:00:01.8640233 OPTION: AllowPrereleaseVersions => False
DEBUG: 00:00:01.8646119 OPTION: ProviderName => PowerShellGet
DEBUG: 00:00:01.8650752 OPTION: Scope => AllUsers
DEBUG: 00:00:02.0286217 Done calling powershell «Get-InstalledPackage» «PSModule»
PS /home/gyana>

PS /opt/microsoft/powershell/6.0.0-rc.2/Modules> Install-Module -Name AzureRM.HDInsight
DEBUG: 00:00:00.0000002 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:00.0000569 Name: AzureRM.HDInsight
DEBUG: 00:00:00.0008489 INVOKING PowerShell Fn Get-DynamicOptions with args Provider that has length 1
DEBUG: 00:00:00.0014937 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:00.0029349 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:00.0086299 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:00.0086749 Name: AzureRM.HDInsight
DEBUG: 00:00:00.0093657 INVOKING PowerShell Fn Get-DynamicOptions with args Source that has length 1
DEBUG: 00:00:00.0099096 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:00.0127235 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:00.0166937 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:00.0167365 Name: AzureRM.HDInsight
DEBUG: 00:00:00.0173824 INVOKING PowerShell Fn Get-DynamicOptions with args Package that has length 1
DEBUG: 00:00:00.0180170 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:00.0234551 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:00.0273170 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:00.0273654 Name: AzureRM.HDInsight
DEBUG: 00:00:00.0280350 INVOKING PowerShell Fn Get-DynamicOptions with args Install that has length 1
DEBUG: 00:00:00.0295586 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:00.0341216 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:00.0376116 Calling SearchForPackages. Name='AzureRM.HDInsight'
DEBUG: 00:00:00.0388668 PackageProvider::FindPackage with name AzureRM.HDInsight
DEBUG: 00:00:00.0390754 Calling SearchForPackages After Select 1
DEBUG: 00:00:00.0418782 Calling New() : MethodName = 'FindPackage'
DEBUG: 00:00:00.0423698 MessageResolver: Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:00.0427867 Type: Module
DEBUG: 00:00:00.0432128 Name: AzureRM.HDInsight
DEBUG: 00:00:00.0435648 AllowPrereleaseVersions: False
DEBUG: 00:00:00.0439750 ProviderName: PowerShellGet
DEBUG: 00:00:00.0443978 Scope: AllUsers
DEBUG: 00:00:00.0452738 INVOKING PowerShell Fn Find-Package with args System.String[], , , that has length 4
DEBUG: 00:00:00.0464463 In PowerShellGet Provider - 'Find-Package'.
DEBUG: 00:00:00.0471289 OPTION: MessageResolver => Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:00.0474984 OPTION: Type => Module
DEBUG: 00:00:00.0479265 OPTION: Name => AzureRM.HDInsight
DEBUG: 00:00:00.0483305 OPTION: AllowPrereleaseVersions => False
DEBUG: 00:00:00.0487048 OPTION: ProviderName => PowerShellGet
DEBUG: 00:00:00.0491770 OPTION: Scope => AllUsers
DEBUG: 00:00:00.0554984 PackageProvider::FindPackage with name AzureRM.HDInsight
DEBUG: 00:00:00.0586545 Calling 'NuGet'::'FindPackage' - name='AzureRM.HDInsight', requiredVersion='',minimumVersion='', maximumVersion='''.
DEBUG: 00:00:00.0594136 Iterating 'AzureRM.HDInsight'.
DEBUG: 00:00:00.0613758 There are '0' registered sources in 'NuGet' provider.
DEBUG: 00:00:00.0627278 Source 'https://www.powershellgallery.com/api/v2/' is not one of the registered sources in 'NuGet' provider.
DEBUG: 00:00:01.1713720 Source 'https://www.powershellgallery.com/api/v2/' is validated.
DEBUG: 00:00:01.1721254 Calling 'NuGetRequest'::'GetPackageById', 'AzureRM.HDInsight'.
DEBUG: 00:00:01.3707006 Calling 'HttpClientPackageRepository'::'FindPackagesById', 'AzureRM.HDInsight'.
DEBUG: 00:00:01.3714256 Calling 'NuGetClient'::'FindPackage'.
DEBUG: 00:00:01.3728048 Downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=0&$top=40'.
DEBUG: 00:00:01.8889741 Completed downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=0&$top=40'.
DEBUG: 00:00:01.8980682 Downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=40&$top=40'.
DEBUG: 00:00:01.8989963 Downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=120&$top=40'.
DEBUG: 00:00:01.8992129 Downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=160&$top=40'.
DEBUG: 00:00:01.8995139 Downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=80&$top=40'.
DEBUG: 00:00:02.5624126 Completed downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=40&$top=40'.
DEBUG: 00:00:02.5627611 Completed downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=120&$top=40'.
DEBUG: 00:00:02.5629831 Completed downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=160&$top=40'.
DEBUG: 00:00:02.5635346 '0' packages received in the last request.
DEBUG: 00:00:02.5638656 Completed downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzureRM.HDInsight'&$skip=80&$top=40'.
DEBUG: 00:00:02.5645073 '0' packages received in the last request.
DEBUG: 00:00:02.5652262 '0' packages received in the last request.
DEBUG: 00:00:02.5659305 '0' packages received in the last request.
DEBUG: 00:00:02.5672301 Completed iterating for 'AzureRM.HDInsight'.
DEBUG: 00:00:02.5773208 Done calling powershell «Find-Package» «PSModule»
DEBUG: 00:00:02.5861849 Calling New() : MethodName = 'GetInstalledPackages'
DEBUG: 00:00:02.5866199 MessageResolver: Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:02.5870348 Type: Module
DEBUG: 00:00:02.5874475 Name: AzureRM.HDInsight
DEBUG: 00:00:02.5878586 AllowPrereleaseVersions: False
DEBUG: 00:00:02.5882884 ProviderName: PowerShellGet
DEBUG: 00:00:02.5887285 Scope: AllUsers
DEBUG: 00:00:02.5896643 INVOKING PowerShell Fn Get-InstalledPackage with args AzureRM.HDInsight, 4.0.1, , that has length 4
DEBUG: 00:00:02.5907204 In PowerShellGet Provider - 'Get-InstalledPackage'.
DEBUG: 00:00:02.5912684 OPTION: MessageResolver => Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:02.5916408 OPTION: Type => Module
DEBUG: 00:00:02.5920806 OPTION: Name => AzureRM.HDInsight
DEBUG: 00:00:02.5924796 OPTION: AllowPrereleaseVersions => False
DEBUG: 00:00:02.5929068 OPTION: ProviderName => PowerShellGet
DEBUG: 00:00:02.5933231 OPTION: Scope => AllUsers
DEBUG: 00:00:02.7166915 PowerShell Script 'PSModule' Function 'Get-InstalledPackage' returns null.
DEBUG: 00:00:02.7176152 Done calling powershell «Get-InstalledPackage» «PSModule»

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules
from 'PSGallery'?
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "N"): A
DEBUG: 00:00:05.9077745 Calling New() : MethodName = 'InstallPackage'
DEBUG: 00:00:05.9084801 MessageResolver: Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:05.9090906 Type: Module
DEBUG: 00:00:05.9096581 Name: AzureRM.HDInsight
DEBUG: 00:00:05.9102472 AllowPrereleaseVersions: False
DEBUG: 00:00:05.9108025 ProviderName: PowerShellGet
DEBUG: 00:00:05.9114629 Scope: AllUsers
DEBUG: 00:00:05.9129046 INVOKING PowerShell Fn Install-Package with args NuGet|AzureRM.HDInsight|4.0.1|https://www.powershellgallery.com/api/v2/|Module that has length 1
DEBUG: 00:00:05.9142623 In PowerShellGet Provider - 'Install-Package'.
DEBUG: 00:00:05.9155886 In PowerShellGet Provider - 'Install-PackageUtility'.
DEBUG: 00:00:05.9162846 The FastPackageReference is 'NuGet|AzureRM.HDInsight|4.0.1|https://www.powershellgallery.com/api/v2/|Module'.
DEBUG: 00:00:05.9178319 OPTION: MessageResolver => Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:05.9190417 OPTION: Type => Module
DEBUG: 00:00:05.9196808 OPTION: Name => AzureRM.HDInsight
DEBUG: 00:00:05.9202020 OPTION: AllowPrereleaseVersions => False
DEBUG: 00:00:05.9208370 OPTION: ProviderName => PowerShellGet
DEBUG: 00:00:05.9213661 OPTION: Scope => AllUsers
DEBUG: 00:00:05.9239669 ArtifactType is Module
DEBUG: 00:00:05.9459212 PowerShell Script 'PSModule' Function 'Install-Package' returns null.
DEBUG: 00:00:05.9470823 Done calling powershell «Install-Package» «PSModule»
PS /opt/microsoft/powershell/6.0.0-rc.2/Modules>

Script/Steps for Reproduction

1. Get powershell on Linux machine as per version mentioned above
2. go to powershell in su mode to "save", "install" and "import" following modules:-
   AzureRM.HDInsight
   (it says that it is installing all dependent modules but seems not).
   Hence "save", "install" and "import" AzureRM.profile if required.
3. Execute, Get-AzureRMHDInsightProperties and all other command in Azure.HDInsight

In my case, not a single command was running properly.

Any suucessful installation on Ubuntu or MAC along with cmdlets output will be helpful.

[cormacpayne]

@deepakshakya Hey Deepak, currently we have two sets of rollup modules for different platforms:

• AzureRM is the rollup module for cmdlets that can be used on Windows
• AzureRM.Netcore is the rollup module for cmdlets that can be used on Mac and Linux

Our goal is to convert all modules found in AzureRM to .NET Core in the near future, which you can track through this open issue. Please let us know if you have any questions or concerns!

CC: @twitchax