GH-127058: Make PySequence_Tuple safer and probably faster.

[markshannon]

Instead of allocating a tuple, then running arbitrary code with an incomplete tuple, this PR:

• Allocates a small fixed sized array, fills it then creates the tuple atomically.
• If the buffer is too small, then a list is created. The tuple is created atomically from that list.
• Adds the internal _PyList_AsTupleAndClear function which creates a tuple from a list while clearing the list. This avoids unnecessary reference count manipulation.

Since the vast majority of tuples are small, PySequence_Tuple now only does one allocation and no resizing in the common case.

• Issue: Tuples should be immutable and safe in C, as well as in Python. #127058

[iritkatriel]

I think the name _PyList_AsTupleAndClear might be confusing, because if you called Py_CLEAR on the list you wouldn't need to decref it.

[markshannon]

But if you called PyList_Clear on it, you would need to Py_DECREF it.
It is the list being cleared, not the variable holding a reference to it.

[iritkatriel]

Did you run refleak tests?

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 8badf7b 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

[iritkatriel]

If _PyTuple_FromArraySteal, do we still want to free the list items?

[markshannon]

Yes. _PyTuple_FromArraySteal steals the references, but doesn't do anything with the memory

[markshannon]

Typically it is passed a stack-allocated array, so it can't free it.

[iritkatriel]

Sorry, I meant if _PyTuple_FromArraySteal fails.

[markshannon]

Yes. _PyTuple_FromArraySteal consumes the all references regardless of whether it succeeds or fails.
It would be a pain to use otherwise.

[user202729]

note that this pull request modifies the behavior of the following code

import time
class A:
	def __len__(self):
		raise NotImplementedError("infinite sequence")
	def __iter__(self):
		while True:
			time.sleep(1)
			yield 0

tuple(A())

before this change, the above raises an error. Now it enters an infinite loop.

by itself it isn't a problem because the fact that tuple() calls __len__ is an internal implementation detail and can be changed any time, but there isn't any other way for a class to declare itself as not convertible to list/tuple because it is infinite. There is no magic method __list__/__tuple__/__sequence__.

the same thing can be said about tuple(range(10**17))—before the change, immediate MemoryError; after it, consumes an increasing amount of memory before the process gets killed (don't try to run this).