read!(::IOStream, ::Array) implementation issues

[simonster]

I started trying to optimize some file IO and I noticed two potential issues.

First, we have a function readbytes_all!(::IOStream, ::Vector{UInt8}, nb) which seems to be slightly more sophisticated than read!(::IOStream, ::Vector{UInt8}). The implementation of readbytes_all! seems to suggest that ios_readall may return fewer bytes than requested even in a non-EOF condition, but read! doesn't appear to handle that.

Second, read!(::IOStream, ::Array) for isbits array element types ends up calling this code, which uses reinterpret. This is not efficient for small arrays, since reinterpret needs to allocate a new array object, although the underlying buffer is the same. The allocation overhead can be non-negligible if the array is small and read! is getting called in a loop, and it seems that we could just pass a pointer to the array to ios_readall directly.

Finally, for my purposes, these functions introduce some abstraction overhead, since what I really want to do is read some data into an arbitrary location in an existing array. At the moment, my only option is to read into a temporary array and then copy that to the existing array. It might be nice to have a way to read! into an array at a specified array offset. Or if we had zero-overhead SubArrays and an efficient read! implementation for them, that would also work.

[nalimilan]

Finally, for my purposes, these functions introduce some abstraction overhead, since what I really want to do is read some data into an arbitrary location in an existing array. At the moment, my only option is to read into a temporary array and then copy that to the existing array. It might be nice to have a way to read! into an array at a specified array offset. Or if we had zero-overhead SubArrays and an efficient read! implementation for them, that would also work.

FWIW, this is something I also needed for #14460. For now, I use sub arrays, but I haven't benchmarked it.

[rfourquet]

Do you think the small helper type at #9174 (comment) would help your use case?

[vtjnash]

the unsafe_read / unsafe_write primitives are now intended to help support this. however, i believe another pass through the read / write functions is still needed to continue cleaning it up.

[bicycle1885]

Is there any progress or other discussions related to this topic? I often want this kind of interfaces to do some buffered IO while parsing. Specifically,

1. Read data from an IO object to a buffer (array or pointer).
2. Be able to specify the upper limit of reading data.
3. Do not throw EOFError even when it fails to read the upper limit.