Skip to content
This repository was archived by the owner on Aug 2, 2021. It is now read-only.
This repository was archived by the owner on Aug 2, 2021. It is now read-only.

extend http downloader with asynchronous lookahead  #1524

@zelig

Description

@zelig

The http reader is suboptimal since it downloads in syncronous batches translating to range queries for the LazyChunkReader. Previously we optimised for buffersize in this reader, however, this still leaves longer latencies since each batch is blocking until the slowest chunk arrives.

A low hanging fruit to improve is to extend this reader with an asyncronous lookahead that brings concurrent network retrieval across batches and thereby significantly reduce download latencies.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions