Share System.IO.Pipelines.BufferSegments (make BufferSegmentStack static) across instances of Pipe

[cobalthex]

One of the benefits of System.IO.Pipelines.Pipe is its ability to be re-used and otherwise conserve memory through usage of ArrayPools internally.

Pipes create BufferSegments internally to store the memory segments it uses, which are further stored in a ReadOnlySequence.
While the buffer segments are re-used through BufferSegmentStack, this is a unique instance on every Pipe and cannot be shared.

This creates a potential issue where if using a pool of Pipes. Each pipe can have its stack grow ("boundlessly") and maybe end up with many un/under-used segments in a long-lived pool.

I would propose that having the buffer segments be responsible for owning/renting the contained array/memory and that the Pipe is only responsible for renting segments.

[davidfowl]

Each pipe can have its stack grow ("boundlessly") and maybe end up with many un/under-used segments in a long-lived pool

How? There's a maximum amount of segments that are pooled.

Can you show how you're using the pipe and also how you're measuring the memory usage?

[cobalthex]

hmm, missed the part about restricted growth

As for an example, it would be for use in re-usable buffers for handling socket receives:

var pipe = pipePool.Rent();
while (socket is open)
{
  var memory = pipe.Writer.GetMemory();
  await socket.Receive(memory);
  pipe.Writer.Advance(...);
  if (is end of message)
  { 
    pipe.Writer.Flush();
    var reader = pipe.Reader;
    // do work here either with reader.ReadAsync() or reader.AsStream()
    pipe.Reader.AdvanceTo(...);
  }
}
pipePool.Return(pipe);

[davidfowl]

You reset the pipe during the use of the socket? Why? I'm confused on the usage pattern and savings here. What are you seeing and what are you trying to reduce?

[cobalthex]

sorry, pseudocoding, I will update the example

[cobalthex]

basically, incoming messages can be of varying sizes, so using a pipe to take advantage of its segmenting is useful. However, when re-using these pipes as sockets open/close, it would be nice if they could share internal buffer segments so that after running for a while, all of these re-used pipes don't each end up holding onto an internal stack of buffers. Especially if most messages only need 1 segment.

[davidfowl]

I still don't understand what you'd be saving. Do you have some back of the napkin math? Do you have a memory profile? I'd like to understand how much you think you'd save if the segments were shared across pipes.

Presumably if each pipe rented an array from the array pool. It wouldn't change how much memory was allocated unless your workload was constantly dropping and adding connections. If you had lots of concurrent connections, there'd be no savings...

[cobalthex]

So this would be the case of many constantly dropped/added. I cannot provide specifics unfortunately but I was thinking if there are (tens/hundreds of) thousands of connections coming and going it might end up with 75% of the segments unused most of the time.

This may be a case of me wanting to over-optimize, idk

[davidfowl]

I'd like to replicate the scenario so I can measure and play with changes. The more data you can provide the better (and would have a better likelihood of success). How many pipes are you pooling? I'm guessing you don't pool tens of thousands of pipes (that would be counter productive I think)?

I've thought about doing this in the past but haven't seen the scenario that has convinced me we should optimized for this. I could be convinced with the right evidence.

[cobalthex]

Unfortunately I cannot provide too much detail but 10k+ connections at any one time, some short-lived, some long, The size of each socket receive would vary in size from one segment to multiple (but likely usually one segment).

Maybe pooling all of those pipes is not a good idea? I am not sure, the goal is to avoid re-allocating them if they will likely be needed again (which is obv. dependent on the number of incoming sockets)

[davidfowl]

Maybe pooling all of those pipes is not a good idea? I am not sure, the goal is to avoid re-allocating them if they will likely be needed again (which is obv. dependent on the number of incoming sockets)

Pooling pipes per request is fine if you were going to pool per request. We pool pipes for HTTP/2 requests (we reuse the request object and reset the pipe on them). We don't pool pipes at the connection level because the benefits are less clear. You'd need to have a good idea of the socket churn as that's the only time you'll get reuse. The pool becomes less effective if there are lots of long lived connections so you only benefit from not allocating, if your pool size matches the churn you expect.

[cobalthex]

In my case, as these are continuous sockets, not one off (http) requests, it makes more sense (imo) to hold the pipe for the duration of the socket and return it to the pool after the socket closes. Holding onto a large array of pools I agree might not make sense, but for a ~10k object pool, overhead seems pretty minor.
These sockets are probably likely to live a few minutes on average, not sure if you consider that long lived, in which case maybe a pool doesn't make sense, though I'd prefer holding a pool of pointers rather than having to alloc on demand