bpo-39511: PyThreadState_Clear() calls on_delete (GH-18296)

PyThreadState.on_delete is a callback used to notify Python when a
thread completes. _thread._set_sentinel() function creates a lock
which is released when the thread completes. It sets on_delete
callback to the internal release_sentinel() function. This lock is
known as Threading._tstate_lock in the threading module.

The release_sentinel() function uses the Python C API. The problem is
that on_delete is called late in the Python finalization, when the C
API is no longer fully working.

The PyThreadState_Clear() function now calls the
PyThreadState.on_delete callback. Previously, that happened in
PyThreadState_Delete().

The release_sentinel() function is now called when the C API is still
fully working.

Author: vstinner

Doc/c-api/init.rst

@@ -1048,6 +1048,10 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
    Reset all information in a thread state object.  The global interpreter lock
    must be held.
 
+   .. versionchanged:: 3.9
+      This function now calls the :c:member:`PyThreadState.on_delete` callback.
+      Previously, that happened in :c:func:`PyThreadState_Delete`.
+
 
 .. c:function:: void PyThreadState_Delete(PyThreadState *tstate)
 

Misc/NEWS.d/next/C API/2020-01-31-16-35-21.bpo-39511.nv9yEn.rst

@@ -0,0 +1,3 @@
+The :c:func:`PyThreadState_Clear` function now calls the
+:c:member:`PyThreadState.on_delete` callback. Previously, that happened in
+:c:func:`PyThreadState_Delete`.

Python/pystate.c

@@ -806,6 +806,10 @@ PyThreadState_Clear(PyThreadState *tstate)
     Py_CLEAR(tstate->async_gen_finalizer);
 
     Py_CLEAR(tstate->context);
+
+    if (tstate->on_delete != NULL) {
+        tstate->on_delete(tstate->on_delete_data);
+    }
 }
 
 
@@ -830,9 +834,7 @@ tstate_delete_common(PyThreadState *tstate,
     if (tstate->next)
         tstate->next->prev = tstate->prev;
     HEAD_UNLOCK(runtime);
-    if (tstate->on_delete != NULL) {
-        tstate->on_delete(tstate->on_delete_data);
-    }
+
     PyMem_RawFree(tstate);
 
     if (gilstate->autoInterpreterState &&