[WebToolsE2E][Aspire] After adding Aspire.Confluent.Kafka component and related codes, restarting messaging container resource fails with some errors "Failed to delete 'messaging-rmdxzjam' successfully before restart." in the Aspire dashboard console log

[Rita003]

INSTALL STEPS

1. Clean machine: Win11 x64 23h2 ENU

2. Install SDK 9.0.100-rc.2.24474.11

   • Apply feed for SDK 9.0 RC2

3. Install Aspire 9.0.0-preview.4.24475.9

   • Apply the dotnet8 feed for Aspire 9.0

REPRO STEPS

1. Open CMD, create an Aspire Starter App using following commands:

dotnet new aspire-starter -o AspireStarterApp
cd AspireStarterApp

2. Add the Aspire.Confluent.Kafka component to your AspireStarterApp.Web app:

cd AspireStarterApp.Web
dotnet add package Aspire.Confluent.Kafka --version 9.0.0-preview.4.24475.9

3. In the Program.cs file of the AspireStarterApp.Web project, type following codes:
   builder.AddKafkaProducer<string, string>("messaging");
4. Open appsettings.json of AspireStarterApp.AppHost project, add ConnectionStrings configuration section:

{
  "ConnectionStrings": {
    "myConnection": "broker:9092"
  }
}

5. Add the Aspire.Hosting.Kafka component to your AspireStarterApp.AppHost app:

cd ..
cd AspireStarterApp.AppHost
dotnet add package Aspire.Hosting.Kafka --version 9.0.0-preview.4.24475.9

6. Open Program.cs of AspireStarterApp.AppHost project, add following codes:

var messaging = builder.AddKafka("messaging");
	
var myService = builder.AddProject<Projects.MyService>()
                   .WithReference(messaging);

7. Run the project in CMD, and open URL in the browser.
   dotnet run
8. Click the 'Restart' option behind the messaging container resource in the Aspire dashboard

ACTUAL
Restart failed with some errors in the console

Error Log:

2024-09-27T02:55:20 Executing command 'restart'.
2024-09-27T02:55:23
 Error executing command 'restart'.
Aspire.Hosting.DistributedApplicationException: Failed to delete 'messaging-rmdxzjam' successfully before restart.
   at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass80_2`1.<<StartResourceAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 2098
--- End of stack trace from previous location ---
   at Polly.ResiliencePipeline.<>c.<<ExecuteAsync>b__3_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Outcome`1.GetResultOrRethrow()
   at Polly.ResiliencePipeline.ExecuteAsync(Func`2 callback, CancellationToken cancellationToken)
   at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass80_0.<<StartResourceAsync>g__StartExecutableOrContainerAsync|0>d`1.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 2093
--- End of stack trace from previous location ---
   at Aspire.Hosting.Dcp.ApplicationExecutor.StartResourceAsync(String resourceName, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 2041
   at Aspire.Hosting.ApplicationModel.CommandsConfigurationExtensions.<>c.<<AddLifeCycleCommands>b__3_5>d.MoveNext() in /_/src/Aspire.Hosting/ApplicationModel/CommandsConfigurationExtensions.cs:line 88
--- End of stack trace from previous location ---
   at Aspire.Hosting.Dashboard.DashboardCommandExecutor.ExecuteCommandAsync(String resourceId, ResourceCommandAnnotation annotation, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dashboard/DashboardCommandExecutor.cs:line 31
   at Aspire.Hosting.Dashboard.DashboardServiceData.ExecuteCommandAsync(String resourceId, String type, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dashboard/DashboardServiceData.cs:line 108

[davidfowl]

cc @JamesNK

[JamesNK]

Hosting tells DCP to delete a container, then gets the container a number of times until there is a 404 to ensure it is deleted. The error happens DCP successfully returns the container about 5 times.

Restart:

1. Stops the container.
   1. Hosting tells DCP to do this by setting Spec.Stop = true and sending a patch to DCP.
2. Starts the container.
   1. Hosting tells DCP to delete the container
   2. Ensures the container is deleting by getting the container and waiting for a 404 <- FAILING HERE
   3. Creates the container again.

I've noticed the messaging container is quite slow to start and stop. If a container is already stopping, does deleting the container wait for it to finish stopping? If that takes some time it could cause the timeout while waiting for it to be deleted.

@karolz-ms Do you have insight into what is happening here?

[karolz-ms]

@JamesNK the root cause of all these recent problems with restarting containers is that Docker CLI is quite unreliable, and it got worse that it used to be recently. And it is not only that commands fail sometimes, but they can also fail with a "timeout"-type of error, but actually succeed (e.g. container creation command times out, but the container actually gets created). So it is not sufficient to just retry a command, but we have to check for the result, and only retry if the world does not look like it is supposed to be.

I am working on a comprehensive change to add retries and verification of results pretty much to ALL INTERACTIONS with container orchestrator(s), so that we reduce this instability to an absolute minimum. This should land sometime next week.

@davidfowl @DamianEdwards @mitchdenny FYI

[mitchdenny]

Will this work make it in for GA @karolz-ms?

[davidfowl]

AFAIK these changes should be in.

[Rita003]

After installing the latest SDK 9.0GA + Aspire 9.0.0, there is no “Restart messaging command failed” displayed in the upper right corner of the page when click the 'Restart' option, but the messaging container state always shows stopping, and the console log also have some errors.

[davidfowl]

I see this too, I;m using podman with my 9.0 builds, @Rita003 are you using docker desktop? When I try to restart kafka, it stays in the Stopping state forever. I get the same error as above.

@karolz-ms I figured #6422 would fix it but it still fails.

cc @JamesNK

[davidfowl]

Stopping then starting works, this seems to be a timing issue:

Also saw this:

fail: Aspire.Hosting.Dcp.start-apiserver.api-server.logstorage[0]
      failed to initialize log stream   {"Kind": "usvc-dev.developer.microsoft.com/v1, Kind=Container", "Name": "kafka-eezftmjj", "UID": "317f2431-60c8-44bb-a88c-5b6c086ded5b", "Source": "startup_stderr", "Follow": true, "error": "Container is being deleted"}
fail: Aspire.Hosting.Dcp.start-apiserver.api-server.logstorage[0]
      failed to initialize log stream   {"Kind": "usvc-dev.developer.microsoft.com/v1, Kind=Container", "Name": "kafka-eezftmjj", "UID": "317f2431-60c8-44bb-a88c-5b6c086ded5b", "Source": "startup_stdout", "Follow": true, "error": "Container is being deleted"}
fail: Aspire.Hosting.Dcp.start-apiserver.api-server.logstorage[0]
      failed to initialize log stream   {"Kind": "usvc-dev.developer.microsoft.com/v1, Kind=Container", "Name": "kafka-eezftmjj", "UID": "317f2431-60c8-44bb-a88c-5b6c086ded5b", "Source": "stdout", "Follow": true, "error": "Container is being deleted"}
fail: Aspire.Hosting.Dcp.start-apiserver.api-server.logstorage[0]
      failed to initialize log stream   {"Kind": "usvc-dev.developer.microsoft.com/v1, Kind=Container", "Name": "kafka-eezftmjj", "UID": "317f2431-60c8-44bb-a88c-5b6c086ded5b", "Source": "stderr", "Follow": true, "error": "Container is being deleted"}

[Rita003]

I see this too, I;m using podman with my 9.0 builds, @Rita003 are you using docker desktop? When I try to restart kafka, it stays in the Stopping state forever. I get the same error as above.

@davidfowl Yes, I'm using docker desktop

[karolz-ms]

Roger, will take another look at this