Skip to content

[Cuda] Skip FreeDataSpace when CUDA driver is in inconsistent state #16980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 13, 2024
Merged

[Cuda] Skip FreeDataSpace when CUDA driver is in inconsistent state #16980

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions src/runtime/cuda/cuda_device_api.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,24 @@ class CUDADeviceAPI final : public DeviceAPI {
}

void FreeDataSpace(Device dev, void* ptr) final {
if (std::uncaught_exceptions() && cudaPeekAtLastError() == cudaErrorIllegalAddress) {
Copy link
Member

@masahi masahi May 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that uncaught_exceptions() is not going to add any overhead.

Copy link
Contributor Author

@Lunderberg Lunderberg May 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be minimal compared to the memory allocation/free itself. The exact implementation of std::uncaught_exceptions() is compiler-dependent, but it typically is implemented as a read of a static thread-local variable.

// For most CUDA calls, an error from an API call will be
// immediately reported, and raised as an exception. However,
// errors raised from async kernel execution leave the CUDA
// driver in an inconsistent state. These errors are "sticky",
// and are never cleared. (See [0] for more details.)
//
// If we are currently unwinding the stack due to a thrown
// exception, and the CUDA driver is in an unrecoverable error,
// do not attempt to free the CUDA allocations. Performing any
// CUDA API call while in this state will throw an additional
// exception, causing a segfault. In this case, it is better to
// allow the original error to continue propagating.
//
// [0] https://forums.developer.nvidia.com/t/cuda-errors-determine-sticky-ness/271625
return;
}

if (dev.device_type == kDLCUDAHost) {
VLOG(1) << "freeing host memory";
CUDA_CALL(cudaFreeHost(ptr));
Expand Down