[Flaky Test]: TestRepeatedInstallUninstallFleet – The system cannot find the file specified.

[pchila]

Failing test case

TestRepeatedInstallUninstallFleet

Error message

could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified.

Build

https://buildkite.com/elastic/elastic-agent/builds/19363#0195ec35-a3f2-4a3e-a042-bbfa1f1a81fd/272-982

OS

Windows

Stacktrace and notes

{"log.level":"error","@timestamp":"2025-03-31T13:06:20.945Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":311},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: The system cannot find the file specified.\"","ecs.version":"1.6.0"}
        Something went wrong while enrolling the Elastic Agent: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."
        Error: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
[=== ] Uninstalled  [1m7s] Error uninstalling. Printing logs
        2025-03-31T13:06:20.979Z	DEBUG	[install]	Loaded configuration from C:\Users\buildkite\AppData\Local\Temp\TestRepeatedInstallUninstallFleet4064552420\elastic-agent-9.0.0-SNAPSHOT-windows-x86_64\elastic-agent.yml
        2025-03-31T13:06:20.979Z	DEBUG	[install]	Merged configuration from C:\Users\buildkite\AppData\Local\Temp\TestRepeatedInstallUninstallFleet4064552420\elastic-agent-9.0.0-SNAPSHOT-windows-x86_64\elastic-agent.yml into result
        2025-03-31T13:06:20.979Z	DEBUG	[install]	Merged all configuration files from [C:\Users\buildkite\AppData\Local\Temp\TestRepeatedInstallUninstallFleet4064552420\elastic-agent-9.0.0-SNAPSHOT-windows-x86_64\elastic-agent.yml], no external input files
        2025-03-31T13:06:21.082Z	DEBUG	[install]	Loaded configuration from C:\Users\buildkite\AppData\Local\Temp\TestRepeatedInstallUninstallFleet4064552420\elastic-agent-9.0.0-SNAPSHOT-windows-x86_64\elastic-agent.yml
        2025-03-31T13:06:21.082Z	DEBUG	[install]	Merged configuration from C:\Users\buildkite\AppData\Local\Temp\TestRepeatedInstallUninstallFleet4064552420\elastic-agent-9.0.0-SNAPSHOT-windows-x86_64\elastic-agent.yml into result
        2025-03-31T13:06:21.082Z	DEBUG	[install]	Merged all configuration files from [C:\Users\buildkite\AppData\Local\Temp\TestRepeatedInstallUninstallFleet4064552420\elastic-agent-9.0.0-SNAPSHOT-windows-x86_64\elastic-agent.yml], no external input files
        2025-03-31T13:06:21.082Z	DEBUG	[install.composable]	Starting controller for composable inputs
        2025-03-31T13:06:21.082Z	DEBUG	[install.composable]	Started controller for composable inputs
        2025-03-31T13:06:21.082Z	DEBUG	[install.composable]	Computing new variable state for composable inputs
        2025-03-31T13:06:21.082Z	DEBUG	[install.composable]	Stopping controller for composable inputs
        2025-03-31T13:06:21.082Z	DEBUG	[install.composable]	Stopped controller for composable inputs
        Error: enroll command failed for unknown reason: exit status 1
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
    install_test.go:697:
        	Error Trace:	C:/buildkite-agent/builds/bk-agent-prod-gcp-1743425000980367647/elastic/elastic-agent/testing/integration/install_test.go:697
        	Error:      	Received unexpected error:
        	            	error running agent install command: exit status 1
        	Test:       	TestRepeatedInstallUninstallFleet/TestRepeatedInstallUninstallFleet-19
        	Messages:   	install failed: error running agent install command: exit status 1
    --- FAIL: TestRepeatedInstallUninstallFleet/TestRepeatedInstallUninstallFleet-19 (67.98s)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pkoutsovasilis]

I have an occurrence of another issue being flaky but I see the same failure between the one mentioned here. The root cause seems to be that agent never restarts. For that reason I believe that having also the logfiles from the installation directory of agent and not only the diagnostics would be helpful to debug such flaky tests 🙂

[cmacknz]

The logs from the uninstall command seem odd in the report above:

[====] Uninstalled  [1m10s] Error uninstalling. Printing logs
        2025-04-08T12:39:56.464Z	DEBUG	[install]	Loaded configuration from /tmp/TestInstallAndUnenrollWithEndpointSecurity-protected2029825172/elastic-agent-8.18.0-SNAPSHOT-linux-x86_64/elastic-agent.yml
        2025-04-08T12:39:56.464Z	DEBUG	[install]	Merged configuration from /tmp/TestInstallAndUnenrollWithEndpointSecurity-protected2029825172/elastic-agent-8.18.0-SNAPSHOT-linux-x86_64/elastic-agent.yml into result
        2025-04-08T12:39:56.464Z	DEBUG	[install]	Merged all configuration files from [/tmp/TestInstallAndUnenrollWithEndpointSecurity-protected2029825172/elastic-agent-8.18.0-SNAPSHOT-linux-x86_64/elastic-agent.yml], no external input files
        2025-04-08T12:39:56.552Z	DEBUG	[install]	Loaded configuration from /tmp/TestInstallAndUnenrollWithEndpointSecurity-protected2029825172/elastic-agent-8.18.0-SNAPSHOT-linux-x86_64/elastic-agent.yml
        2025-04-08T12:39:56.552Z	DEBUG	[install]	Merged configuration from /tmp/TestInstallAndUnenrollWithEndpointSecurity-protected2029825172/elastic-agent-8.18.0-SNAPSHOT-linux-x86_64/elastic-agent.yml into result
        2025-04-08T12:39:56.552Z	DEBUG	[install]	Merged all configuration files from [/tmp/TestInstallAndUnenrollWithEndpointSecurity-protected2029825172/elastic-agent-8.18.0-SNAPSHOT-linux-x86_64/elastic-agent.yml], no external input files
        2025-04-08T12:39:56.552Z	DEBUG	[install.composable]	Starting controller for composable inputs
        2025-04-08T12:39:56.552Z	DEBUG	[install.composable]	Started controller for composable inputs
        2025-04-08T12:39:56.552Z	DEBUG	[install.composable]	Computing new variable state for composable inputs
        2025-04-08T12:39:56.552Z	DEBUG	[install.composable]	Stopping controller for composable inputs
        2025-04-08T12:39:56.552Z	DEBUG	[install.composable]	Stopped controller for composable inputs
        Error: enroll command failed for unknown reason: exit status 1
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.18/fleet-troubleshooting.html

1. The error message is wrong, uninstall shouldn't fail with Error: enroll command failed for unknown reason: exit status 1. There must be some more descriptive error we can provide.
2. These logs are at the debug level. The endpoint security tests don't set the log level to debug.

I think we skip diagnostics because we do the enrollment at install time, which means that enrollment failures auto-uninstall the agent:

elastic-agent install --force --non-interactive --url https://216806610d6b8bffcd1568f4251bd894.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token XXXX

Collecting diagnostics is conditional on being installed and we don't hit that here:

elastic-agent/pkg/testing/fixture_install.go

Lines 312 to 320 in 5b71686

	if !f.installed {
	f.t.Logf("skipping uninstall; agent not installed (fixture.installed is false)")
	// not installed; no need to clean up or collect diagnostics
	return
	}
	// diagnostics is collected when either the environment variable
	// AGENT_COLLECT_DIAG=true or the test is marked failed
	collect := collectDiagFlag()

My suggestion would be to split install and enroll into their respective commands, so that enrollment failures leave the agent installed and let us grab logs about why that might have happened.