MAINT: store default endianness state in a context variable

[ngoldbaum]

Towards supporting free-threaded Python, see #234 (comment)

I haven't benchmarked this but I also don't see any benchmarks in the repo. Do you think I should be worrying about any performance impact here? get_c_default_endian probably has more overhead than just reading a global.

I could also store the state in a thread-local C variable, but I thought I'd propose this approach because I think this is the most idiomatic way to do this in Python and gives you async-safety for free along with setting up a public interface with a context manager.

[ngoldbaum]

I just verified that the versions of the tests I added here in the latest version of this PR fail on 3.14.0 using the state of things in the current master branch with a C static global variable:

======================================================================
FAIL: test_default_endian_async_safe (bitarray.test_bitarray.DelSequenceIndexTests.test_default_endian_async_safe)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.14.0/lib/python3.14/site-packages/bitarray/test_bitarray.py", line 1566, in test_default_endian_async_safe
    asyncio.run(main())
    ~~~~~~~~~~~^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.14.0/lib/python3.14/asyncio/runners.py", line 204, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.14.0/lib/python3.14/asyncio/runners.py", line 127, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.14.0/lib/python3.14/asyncio/base_events.py", line 719, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.14.0/lib/python3.14/site-packages/bitarray/test_bitarray.py", line 1560, in main
    await asyncio.gather(
    ...<2 lines>...
    )
  File "/Users/goldbaum/.pyenv/versions/3.14.0/lib/python3.14/site-packages/bitarray/test_bitarray.py", line 1551, in big
    assert endian == 'big'
           ^^^^^^^^^^^^^^^
AssertionError

======================================================================
FAIL: test_default_endian_thread_safe (bitarray.test_bitarray.DelSequenceIndexTests.test_default_endian_thread_safe)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.14.0/lib/python3.14/site-packages/bitarray/test_bitarray.py", line 1591, in test_default_endian_thread_safe
    assert get_default_endian() == endian_start
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

----------------------------------------------------------------------
Ran 599 tests in 0.189s

FAILED (failures=2)

The async failure happens more often on my machine but the multithreaded case also fails about 25% of the time. I haven't seen any failures using the context variable implementation in this PR.

[ilanschnell]

Thanks @ngoldbaum for working on supporting free-threaded Python! I have started looking into the free-threaded Python myself, and came across your PyCon talk on this subject. I am still not too familiar with this topic, and find the different levels on which one may support free-threading, and how it relates to multiple module initialization a bit confusing. I would like to learn more, but feel like it's hard to grasp the big picture. Are there other resources / videos on this topic you can recommend?

[ngoldbaum]

First, I want to point out that this PR fixes a bug that you can trigger on the GIL-enabled build. While I'm working on this to support the free-threaded build, IMO this PR is unrelated to the free-threaded build and you should fix the bug I'm fixing regardless of considerations around the free-threaded build.

The GIL only protects state inside of the interpreter - it doesn't protect state in third-party extensions like the global endianness state I'm fixing here.

Now, for documentation on free-threading there's the docs we've been maintaining at https://py-free-threading.github.io. My piece of the PyCon talk covered the material on this page which covers porting C extensions. There's also the more general advice for porting any arbitrary package to support the free-threaded build that Lysandros covered. The talk was pretty constrained for time so those pages go into a lot more detail than the talk. The general advice covers the levels of support concept.

If you're already familiar with multithreaded C/C++ programming, the trickiest concept to wrap your head around IMO is what a critical section is exactly and what it protects. It offers relatively weak guarantees, but it has the advantage of preventing deadlocks and providing similar guarantees to what the GIL provided extension authors, so it's often convenient to use to make code safe that relied on the GIL. These docs do a great job of covering that.

If you're not familiar with multithreaded C/C++ programming, I've heard "C++: Concurrency in Action" is a good textbook. If you're familiar with Rust syntax, I found Rust atomics and Locks to be excellent and it has the side benefit of being free on the authors' website.

As far as videos go, maybe take a look at David Woods' EuroPython talk on parallelism in Cython? He was talking about thread safety in Cython code but most of what he's saying applies to a C extension as well and I think you're at least a little familiar with Cython already.

If you want to learn more about multithreaded programming in Python in general or specifically on the free-threaded build, I'd encourage you to read through the ft-utils tutorial. In particular, I learned a ton from the page on atomicity and consistency in multithreaded Python and the worked examples linked from that page.

I'm also happy to set up a call if you'd like a lower latency conversation about this stuff. We're also moderating a discord server if you want to pop in and chat.

[ngoldbaum]

The latest changes add, document, and test a new default_endian context manager. This has the nice side-effect of making the tests thread-safe under unittest-ft. The one failing test Rohit originally ran into in #234 is caused by modifying the default endianness state. Even using a context manager doesn't fix that, because you can still modify the global context, even if inner contexts are isolated from the global context.

[ilanschnell]

Thanks Nathan, for the detailed response and for adding the new context manager. I should let you know about the script update_doc.py. It updates README.rst, doc/reference.rst and doc/changelog.rst from the docstrings. So it would be best to only change docstrings in this PR. I always run this script as part of the release process.
Are you available for a call tomorrow (10/30) or Friday (10/31), preferable between 11am and 4pm Central Time (I'm currently in Houston) ? You can also reach me at [email protected]. Thanks!

[ngoldbaum]

Thanks for the comment, sent you an invite for Friday.

[ilanschnell]

After talking @ngoldbaum, we have decided to remove the global default endianness state completely, and that I will make an experimental thread-free enabled release of bitarray in a few days.