PEP 689: Add docs for unstable C API (#1060)

Add docs for unstable C API (PEP 689)

Co-authored-by: C.A.M. Gerlach <[email protected]>
Co-authored-by: Ezio Melotti <[email protected]>

Author: 3 people

developer-workflow/c-api.rst

@@ -4,18 +4,20 @@
 Changing Python's C API
 =======================
 
-The C API is divided into three sections:
+The C API is divided into these tiers:
 
 1. The internal, private API, available with ``Py_BUILD_CORE`` defined.
    Ideally declared in ``Include/internal/``. Any API named with a leading
    underscore is also considered private.
-2. The public C API, available when ``Python.h`` is included normally.
+2. The Unstable C API, identified by the ``PyUnstable_`` name prefix.
+   Ideally declared in :cpy-file:`Include/cpython/` along with the general public API.
+3. The “general” public C API, available when :cpy-file:`Include/Python.h` is included normally.
    Ideally declared in ``Include/cpython/``.
-3. The Limited API, available with ``Py_LIMITED_API`` defined.
+4. The Limited C API, available with :c:macro:`Py_LIMITED_API` defined.
    Ideally declared directly under ``Include/``.
 
-Each section has higher stability & maintenance requirements, and you will
-need to think about more issues when you add or change definitions in it.
+Each tier has different stability and maintenance requirements to consider
+when you add or change definitions in it.
 
 The compatibility guarantees for public C API are explained in the
 user documentation, ``Doc/c-api/stable.rst`` (:ref:`python:stable`).
@@ -29,13 +31,19 @@ for building CPython itself, as indicated by a macro like ``Py_BUILD_CORE``.
 
 While internal API can be changed at any time, it's still good to keep it
 stable: other API or other CPython developers may depend on it.
+For users, internal API is sometimes the best workaround for a thorny problem
+--- though those use cases should be discussed on the
+`C API Discourse category <https://discuss.python.org/c/30>`_
+or an issue so we can try to find a supported way to serve them.
+
 
 With PyAPI_FUNC or PyAPI_DATA
 -----------------------------
 
 Functions or structures in ``Include/internal/`` defined with
 ``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are
 exposed only for specific use cases like debuggers and profilers.
+Ideally, these should be migrated to the :ref:`unstable-capi`.
 
 
 With the extern keyword
@@ -53,14 +61,18 @@ Private names
 --------------
 
 Any API named with a leading underscore is also considered internal.
-There are two main use cases for using such names rather than putting the
-definition in ``Include/internal/`` (or directly in a ``.c`` file):
+There is currently only one main use case for using such names rather than
+putting the definition in :cpy-file:`Include/internal/` (or directly in a ``.c`` file):
+
+* Internal helpers for other public APIs, which users should not call directly.
+
+Note that historically, underscores were used for APIs that are better served by
+the :ref:`unstable-capi`:
+
+* “provisional” APIs, included in a Python release to test real-world
+  usage of new APIs;
+* APIs for very specialized uses like JIT compilers.
 
-* Internal helpers for other public API; users should not use these directly;
-* “Provisional” API, included in a Python release to test real-world usage
-  of new API. Such names should be renamed when stabilized; preferably with
-  a macro aliasing the old name to the new one.
-  See :pep:`"Finalizing the API" in PEP 590 <590#finalizing-the-api>` for an example.
 
 Internal API Tests
 ------------------
@@ -134,6 +146,103 @@ When moving existing tests, feel free to replace ``TestError`` with
 ``PyExc_AssertionError`` unless actually testing custom exceptions.
 
 
+.. _unstable-capi:
+
+Unstable C API
+==============
+
+The unstable C API tier is meant for extensions that need tight integration
+with the interpreter, like debuggers and JIT compilers.
+Users of this tier may need to change their code with every minor release.
+
+In many ways, this tier is like the general C API:
+
+- it's available when ``Python.h`` is included normally,
+- it should be defined  in :cpy-file:`Include/cpython/`,
+- it requires tests, so we don't break it unintentionally
+- it requires docs, so both we and the users,
+  can agree on the expected behavior,
+- it is tested and documented in the same way.
+
+The differences are:
+
+- Names of functions structs, macros, etc. start with the ``PyUnstable_``
+  prefix. This defines what's in the unstable tier.
+- The unstable API can change in minor versions, without any deprecation
+  period.
+- A stability note appears in the docs.
+  This happens automatically, based on the name
+  (via :cpy-file:`Doc/tools/extensions/c_annotations.py`).
+
+Despite being “unstable”, there are rules to make sure third-party code can
+use this API reliably:
+
+* Changes and removals can be done in minor releases
+  (:samp:`3.{x}.0`, including Alphas and Betas for :samp:`3.{x}.0`).
+* Adding a new unstable API *for an existing feature* is allowed even after
+  Beta feature freeze, up until the first Release Candidate.
+  Consensus on the `Core Development Discourse <https://discuss.python.org/c/core-dev/23>`_
+  is needed in the Beta period.
+* Backwards-incompatible changes should make existing C callers fail to compile.
+  For example, arguments should be added/removed, or a function should be
+  renamed.
+* When moving an API into or out of the Unstable tier, the old name
+  should continue to be available (but deprecated) until an incompatible
+  change is made. In other words, while we're allowed to break calling code,
+  we shouldn't break it *unnecessarily*.
+
+
+Moving an API from the public tier to Unstable
+----------------------------------------------
+
+* Expose the API under its new name, with the ``PyUnstable_`` prefix.
+* Make the old name an alias (e.g. a ``static inline`` function calling the
+  new function).
+* Deprecate the old name, typically using :c:macro:`Py_DEPRECATED`.
+* Announce the change in the "What's New".
+
+The old name should continue to be available until an incompatible change is
+made. Per Python’s backwards compatibility policy (:pep:`387`),
+this deprecation needs to last at least two releases
+(modulo Steering Council exceptions).
+
+The rules are relaxed for APIs that were introduced in Python versions
+before 3.12, when the official Unstable tier was added.
+You can make an incompatible change (and remove the old name)
+as if the function was already part of the Unstable tier
+for APIs introduced before Python 3.12 that are either:
+
+* Documented to be less stable than default.
+* Named with a leading underscore.
+
+Moving an API from the private tier to unstable
+-----------------------------------------------
+
+* Expose the API under its new name, with the ``PyUnstable_`` prefix.
+* If the old name is documented, or widely used externally,
+  make it an alias and deprecate it (typically with :c:macro:`Py_DEPRECATED`).
+  It should continue to be available until an incompatible change is made,
+  as if it was previously public.
+
+  This applies even to underscored names. Python wasn't always strict with
+  the leading underscore.
+* Announce the change in What's New.
+
+Moving an API from unstable to public
+-------------------------------------
+
+* Expose the API under its new name, without the ``PyUnstable_`` prefix.
+* Make the old ``PyUnstable_*`` name be an alias (e.g. a ``static inline``
+  function calling the new function).
+* Announce the change in What's New.
+
+The old name should remain available until the
+new public name is deprecated or removed.
+There's no need to deprecate the old name (it was unstable to begin with),
+but there's also no need to break working code just because some function
+is now ready for a wider audience.
+
+
 Limited API
 ===========