PowerShell plugins are installed inside PowerShell framework folder

[log1-c]

Hi there :)

We are currently installing the PowerShell framework, plugins and the Icinga Agent with a self-written script on some servers and via Ansible on others
Now most of the machines have the problem that the plugins will be installed inside the framework folder instead of the PowerShell modules folder, but not all of them. Strangely on a second run it works without problem on some hosts.

Following is the output of the Installer:

PS C:\temp\CQ-icinga2-agent-offline_installer> .\Install-Framework-and-agent-offline-v1.0.0.ps1 -setting settingsfile
Loading ConfigXML C:\temp\CQ-icinga2-agent-offline_installer\settingsfile
[Notice]: Removing Icinga Agent directory: "C:\ProgramData\icinga2"
[Notice]: Icinga Agent was successfully removed
True
[Notice]: Stopping service "icingapowershell"
[Notice]: Icinga PowerShell Service was successfully removed
[Error]: Your user and/or machine execution policies are configured as "AllSigned" or "RemoteSigned". This means you can only execute PowerShell scripts and modules which are signed by a trusted publisher. Please have a look at https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_signing?view=powershell-7 for further details about how to sign modules and scripts to ensure Icinga for Windows is running properly on your system.
[Notice]: Downloading Icinga PowerShell Framework into "C:\Users\username~1\AppData\Local\Temp\icinga-powershell-framework-zip"
[Notice]: Installing Icinga PowerShell Framework into "C:\Program Files\WindowsPowerShell\Modules\"
[Notice]: Installing new Icinga PowerShell Framework version
[Notice]: Unblocking Icinga PowerShell Framework files
[Notice]: Icinga PowerShell Framework seems to be successfully installed
[Notice]: To use the Icinga PowerShell Framework in the future, please initialize it by running the command "Use-Icinga" inside your PowerShell
[Notice]: Using hostname "servername" for the Icinga Agent configuration
[Notice]: Downloading "Icinga Plugins" into "C:\Users\username\AppData\Local\Temp\tmp_icinga36823945.d"
[Notice]: Installing module into "C:\Users\username\AppData\Local\Temp\tmp_icinga36823945.d"
[Notice]: Using content of folder "C:\Users\username\AppData\Local\Temp\tmp_icinga36823945.d\icinga-powershell-plugins-1.5.1" for updates
[Notice]: Plugins Module Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\icinga-powershell-plugins" is not present. Creating Directory
[Notice]: Copying files to Plugins
[Notice]: Cleaning temporary content
[Notice]: Unblocking Icinga PowerShell Files
Import-Module : The specified module 'icinga-powershell-plugins' was not loaded because no valid module file was found in any module directory.
At C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\lib\core\framework\Install-IcingaFrameworkComponent.psm1:114 char:5
+     Import-Module $RepositoryName;
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (icinga-powershell-plugins:String) [Import-Module], FileNotFoundException
    + FullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand

[Notice]: Icinga Plugins update has been completed. Please start a new PowerShell to apply it
[Notice]: The wizard is complete. These are the configured settings:
========
-AcceptConnections 0
-AddFirewallRule 0
-Ticket '12345'
-EmptyTicket 0
-InstallFrameworkPlugins 1
-PluginsUrl 'C:\temp\CQ-icinga2-agent-offline_installer\icinga-powershell-plugins-1.5.1.zip'
-InstallFrameworkService 0
-RunInstaller

========
====
Start-IcingaAgentInstallWizard -AcceptConnections 0 -AddFirewallRule 0 -Ticket '12345' -EmptyTicket 0 -InstallFrameworkPlugins 1 -PluginsUrl 'C:\temp\CQ-icinga2-agent-offline_installer\icinga-powershell-plugins-1.5.1.zip' -InstallFrameworkService 0 -RunInstaller
====
[Notice]: Downloading Icinga 2 Agent installer "Icinga2-v2.12.4-x86_64.msi" into temp directory "C:\Users\username~1\AppData\Local\Temp\Icinga2-v2.12.4-x86_64.msi"
[Notice]: Installing new Icinga Agent version into "C:\Program Files\ICINGA2"
[Notice]: Icinga Agent was successfully installed
[Notice]: Successfully backed up Icinga 2 Agent default config
[Notice]: Your hostname was successfully changed to "servername"
[Notice]: The Icinga Service User already has permission to run as service
[Passed]: Directory "C:\ProgramData\icinga2\etc" is accessible and writeable by the Icinga Service User "NT Authority\SYSTEM"
[Passed]: Directory "C:\ProgramData\icinga2\var" is accessible and writeable by the Icinga Service User "NT Authority\SYSTEM"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache" is accessible and writeable by the Icinga Service User "NT Authority\SYSTEM"
[Notice]: Service User successfully updated
[Notice]: Background daemon Cmdlet "Start-IcingaServiceCheckDaemon" has been configured
[Notice]: Feature "checker" was successfully disabled
[Notice]: Feature "notification" was successfully disabled
[Notice]: Feature "api" was successfully enabled
[Notice]: Generating host certificates for host "servername"
[Notice]: information/base: Writing private key to 'C:\ProgramData\icinga2\var\lib\icinga2\certs\servername.key'.
information/base: Writing X509 certificate to 'C:\ProgramData\icinga2\var\lib\icinga2\certs\servername.crt'.
[Warning]: Your trusted master certificate is not present. Fetching from your CA server is required
[Notice]: Fetching trusted master certificate from "satellitename"
[Notice]: information/cli: Retrieving TLS certificate for 'satellitename:5665'.

 Version:             3
 Subject:             CN = satellitename
 Issuer:              CN = Icinga CA
 Valid From:          Sep 22 13:44:20 2020 GMT
 Valid Until:         Sep 19 13:44:20 2035 GMT
 Serial:              yes

 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   satellitename
 Fingerprint:         42

***
*** You have to ensure that this certificate actually matches the parent
*** instance's certificate in order to avoid man-in-the-middle attacks.
***

information/pki: Writing certificate to file 'C:\ProgramData\icinga2\var\lib\icinga2\certs\trusted-parent.crt'.
[Warning]: Your ca.crt is not present. Manuall copy or fetching from your Icinga CA is required.
[Notice]: information/cli: Writing CA certificate to file 'C:\ProgramData\icinga2\var\lib\icinga2\certs\ca.crt'.
information/cli: !!!!!!
information/cli: !!! Certificate request for CN 'servername' is pending. Waiting for approval from the parent Icinga instance.
information/cli: !!!!!!
[Notice]: Icinga certificates successfully installed
[Notice]: Api configuration has been written successfully
[Notice]: Icinga Agent zones.conf has been written successfully
[Passed]: Icinga Agent service is installed
[Passed]: Directory "C:\ProgramData\icinga2\etc" is accessible and writeable by the Icinga Service User "NT Authority\SYSTEM"
[Passed]: Directory "C:\ProgramData\icinga2\var" is accessible and writeable by the Icinga Service User "NT Authority\SYSTEM"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache" is accessible and writeable by the Icinga Service User "NT Authority\SYSTEM"
[Passed]: Icinga Agent configuration is valid
[Passed]: Icinga Agent debug log is disabled
[Notice]: Restarting service "icinga2"
[Notice]: Updating Icinga PowerShell Service binary
[Notice]: Service User successfully updated
[Notice]: Restarting service "icingapowershell"
[Notice]: Stopping service "icingapowershell"
[Notice]: Service User successfully updated
True
Ok.

[Warning]: Please run "Write-IcingaFrameworkCodeCache" in addition to ensure your cache is updated. This should be done after each update of the Framework in case the feature was disabled during the update run.

As can bee seen there is the same notice/error as described in #237 that should be already fixed.
Only difference is that we are not using the PowerShell gallery to install the modules.
I'm not sure were this error originates from, as the Modules root is correctly set:

[Notice]: Installing Icinga PowerShell Framework into "C:\Program Files\WindowsPowerShell\Modules\"

Not sure what else to check though.
If you have any ideas what we could check, we'll do so :)

[LordHepipud]

Hello,

thank you for the issue. This is very weird and should never happen. I will have to play around with this myself. I assume it were clean machines without anything installed yet?

As far as I can tell you are removing everything in advance and use the kickstart later on, to deploy everything - correct?

[log1-c]

Yes, that is correct. I forgot to mention that.

function Remove-IcingaComponents{
    Stop-Service -Name "icingapowershell" -Force
    Stop-Service -Name "icinga2" -Force
    Use-Icinga
    Uninstall-IcingaAgent -RemoveDataFolder;
    Remove-Module -Name icinga-powershell-plugins
    Get-Module -Name icinga* | Remove-Module;
    Uninstall-IcingaFrameworkService
    if(Test-Path "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell*")
    {
        Remove-Item -Recurse -Force "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell*"
    }
    if(Test-Path "C:\Program Files\icinga-framework-service*")
    {
        Remove-Item -Recurse -Force "C:\Program Files\icinga-framework-service*"
    }
}

Ansible way

function install-icinga-agent {
    try {
        [Net.ServicePointManager]::SecurityProtocol = "tls12, tls11";
        $ProgressPreference = "SilentlyContinue";
        $global:IcingaFrameworkKickstartSource = 'https://raw.githubusercontent.com/Icinga/icinga-powershell-kickstart/master/script/icinga-powershell-kickstart.ps1';
        $Script = (Invoke-WebRequest -UseBasicParsing -Uri $global:IcingaFrameworkKickstartSource).Content;
        $Script += "`r`n`r`n Start-IcingaFrameworkWizard -RepositoryUrl 'https://github.com/Icinga/icinga-powershell-framework/archive/v$frameworkversion.zip' -ModuleDirectory 'C:\Program Files\WindowsPowerShell\Modules' -AllowUpdate 'y' -SkipWizard $TRUE;";
        Invoke-Command -ScriptBlock ([Scriptblock]::Create($Script));
        $pluginurl = "https://github.com/Icinga/icinga-powershell-plugins/archive/v" + $pluginsversion + "/v" + $pluginsversion + ".zip"
        $serviceurl = "https://github.com/Icinga/icinga-powershell-service/releases/download/v" + $serviceversion + "/icinga-service-v" + $serviceversion + ".zip"
        Use-Icinga
        Start-IcingaAgentInstallWizard -UseDirectorSelfService 0 -AutoUseFQDN 0 -AutoUseHostname 1 -LowerCase 1 -UpperCase 0 -PackageSource 'https://packages.icinga.com/windows/' -AllowVersionChanges 1 -UpdateAgent 1 -AgentVersion $agentversion -Endpoints $endpoints -CAPort $caport -AcceptConnections 0 -AddFirewallRule 0 -ConvertEndpointIPConfig 1 -EndpointConnections $endpointConnections -ParentZone $parentZone -AddDirectorGlobal 1 -AddGlobalTemplates 1 -GlobalZones $globalZones -CAEndpoint $caendpoint -EmptyTicket 0 -Ticket $ticket -ServiceUser $serviceuser -InstallFrameworkPlugins 1 -PluginsUrl $pluginurl -InstallFrameworkService 1 -FrameworkServiceUrl $serviceurl -ServiceDirectory 'C:\Program Files\icinga-framework-service\' -ServiceBin 'C:\Program Files\icinga-framework-service\icinga-service.exe' -RunInstaller
        If($frameworkversionint -ge '141') {
        Enable-IcingaFrameworkCodeCache
        Write-IcingaFrameworkCodeCache
        # Register-IcingaBackgroundDaemon -Command 'Start-IcingaServiceCheckDaemon';
        # Restart-IcingaService 'icingapowershell';
        # Register-IcingaServiceCheck -CheckCommand 'Invoke-IcingaCheckCPU' -Interval 30 -TimeIndexes 15;
        Restart-IcingaService 'icingapowershell';
            }
        }
    catch {
        "An error with Icinga Agent installation."
    }
}

"offline installer"

function Install-IcingaComponents
{
  # get ticket for agent
  $Url = "https://$($endpointConnections):$($caport)/v1/actions/generate-ticket"
  $headers = @{
  'Accept'= 'application/json'
  'Authorization'= $authorization
  'Content-Type'= 'text/plain'
  }
  $Body = @{
  cn = $hostname
  }
  $json = $Body | ConvertTo-Json
  $ticketinforamtions = Invoke-RestMethod -Method 'Post' -Uri $Url -Headers $headers -Body $json
  $ticketinforamtion = $ticketinforamtions.results.ticket
  [Net.ServicePointManager]::SecurityProtocol = "tls12, tls11";
  $ProgressPreference = "SilentlyContinue";
  $global:IcingaFrameworkKickstartSource = "$($workdir)\icinga2-agent-kickstart.ps1";
  $Script = Get-Content -Path $global:IcingaFrameworkKickstartSource -Raw;
  #$Script = [System.Text.Encoding]::UTF8.GetString($Script);
  $Script += "`r`n`r`n Start-IcingaFrameworkWizard -RepositoryUrl $RepositoryUrl -ModuleDirectory 'C:\Program Files\WindowsPowerShell\Modules\' -AllowUpdate 1 -SkipWizard $TRUE;";
  Invoke-Command -ScriptBlock ([Scriptblock]::Create($Script));
  Start-IcingaAgentInstallWizard -UseDirectorSelfService 0 -AutoUseFQDN 0 -AutoUseHostname 1 -LowerCase 1 -UpperCase 0 -PackageSource $workdir -AllowVersionChanges 1 -AgentVersion '2.12.4' -Endpoints $endpoints -CAPort $caport -AcceptConnections 0 -AddFirewallRule 0 -ConvertEndpointIPConfig 1 -EndpointConnections $endpointConnections -ParentZone $parentZone -AddDirectorGlobal 1 -AddGlobalTemplates 1 -GlobalZones $globalZones -CAEndpoint $caendpoint -EmptyTicket 0 -Ticket $ticketinforamtion -ServiceUser $serviceuser -InstallFrameworkPlugins 1 -PluginsUrl $PluginsUrl -InstallFrameworkService 0 -RunInstaller
  $ServiceData = Get-IcingaFrameworkServiceBinary -FrameworkServiceUrl $FrameworkServiceUrl -ServiceDirectory $ServiceDirectory;
  Install-IcingaFrameworkService -Path $ServiceData.ServiceBin -User $serviceuser;

  #Set Firewall to allow ICMP
  netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol="icmpv4:8,any" dir=in action=allow

  #enable icinga framework code cache
  Use-Icinga
  Enable-IcingaFrameworkCodeCache
  Write-IcingaFrameworkCodeCache
  Restart-Service icingapowershell

}

Before installing we also do a Add-MpPreference -ExclusionExtension ".conf" because some of the .conf files get blocked

[LordHepipud]

Is this behavior happening on all machines regardless of versions?
Does it only apply to online/offline installation or to both anyways?

I tried several days to reproduce the issue with your example and failed every time.

Which doesn't mean the issue isn't there, it seems to be a weird edge case occurring in some scenarios. I'm using your Ansible snippet function for example and everything works just fine as far as I can tell so far.

Do you have an idea on how I can reproduce this for troubleshooting?

Which version of the Framework did you try to install? Is it 1.5.1 as the plugins?

[log1-c]

not happening on all machines, no.
Version wise I'm not sure, as we only stumbled upon this while upgrading to 1.5.x manually on some VMs.
Online/offline way doesn't seem to matter, as it can happen with both.
What seems to be a recurring theme is that often specially hardened systems (like DCs) are affected.
But I can't tell you what is actually part of that hardening, bc I don't know^^

We are currently installing v1.5.2
We are also currently restructuring the Agent installation inside the workflow, pushing it further down the road so the VMs are complete before installting the agent. Maybe this solves this, I don't know.

Sorry that this is not much (or nothing) to go on, but I thought you maybe had some idea where to look because you saw it somewhere else ;)

I'll report more back here when we are testing the restructured workflow again

[LordHepipud]

Thanks for the additional input! I will play around a little more with different scenarios and test if I can somehow reproduce this error. Could it be that you were upgrading from 1.4.x or older to 1.5.x on the affected machines?

[log1-c]

Yes. Mostly 1.4.1. But some may even still had < 1.3

[LordHepipud]

Got it - now I can reproduce the issue. In case the module is installed into

C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\1.4.1

for example, then you will run into the issue.

[log1-c]

Hm... I didn't check that. But the 1.4.1 version at least was installed the same way as 1.5.x
I can't remember any hosts in the last days where there was a 1.4.1 folder present in C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\

Also shouldn't the complete removal of all Icinga componentes (agent, framework, plugins) make sure we start fresh?

[LordHepipud]

So, the issue is that before 1.5.x it was not properly supported for components to be installed inside version folders.
1.5.x fixed the path detection. which "resolves" that issue in some way.

Thats however the way I can entirely reproduce the above behavior. Once I manually update to 1.5.x, everything works fine.

For the removal: Have you used a new PowerShell after unintalling the components or used the one you were running with? Because this will cause the old module still being loaded inside the session. All calls made for installation are then handled by the old version and not the new one.

With #310 this entire handling will be way more robust, as with the new package manager, all components are installed properly inside version folders, as PowerShell gallery is doing it.

[log1-c]

As the whole Agent/framework/plugins setup (no matter if online or offline) is one PS script I guess there is no new PS started after the removal.
It checks if something is installed/present, removes that and then installs the new components.

Makes sense that the old stuff might still be loaded.

But I don't quite get why the Framework installer says
[Notice]: Installing Icinga PowerShell Framework into "C:\Program Files\WindowsPowerShell\Modules\"
and then the agent kickstart says
[Notice]: Plugins Module Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\icinga-powershell-plugins" is not present. Creating Directory.
I'm not sure how those two communicate (or even if they do) but wouldn't it be best if all installers used the same path?

[LordHepipud]

The problem is that for "security" reasons and to not ask every time on where modules should be installed to, the installer will lookup on where the Icinga PowerShell Framework is installed into.

Based on this location, plugins and all other components are automatically installed directly into C:\Program Files\WindowsPowerShell\Modules\.

Now for versions prior to 1.5.0, it always assumed that the installations root folder for example is C:\Program Files\WindowsPowerShell\Modules\. If you installed the Framework into C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\1.4.0, then the root folder was defined as C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\, because it simply just moved one folder upwards.

Now with 1.5.0 it will lookup the correct root folder properly and instead of returning C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\, it will return C:\Program Files\WindowsPowerShell\Modules\ to install all files directly into there.

I can't automatically assume the location, because the framework could also be installed on some custom locations like C:\icinga\PowerShell as example.

Therefor the assumption of the installation of the component is always the same diretory the Framework is installed into.

In addition, this also ensures a security feature, because only trusted locations on a system level are included and decreases the chance of side-loading modules, being executed by the frameowork.