Heavy performance degradation with large subchunkable L1 chunks

[trivoldus28]

I'm still running into a performance issue (has been for the past year) where the workers slow to a crawl when there are 8x L0 chunks in an L1 chunk, vs 4x L0 chunks. Using 8x on line 70 here would make the job take 1 hour vs 10m.

[trivoldus28]

I found a workaround is to increase the chunk sizes in 4x. Example:

    // reduce # of chunks, in stages of 4x to avoid catastrophic perf hits
    processing_chunk_sizes: [
        [#PROC_CHUNK_SIZE[0]*16, #PROC_CHUNK_SIZE[1]*16, #PROC_CHUNK_SIZE[2]*4],
        [#PROC_CHUNK_SIZE[0]*16, #PROC_CHUNK_SIZE[1]*8, #PROC_CHUNK_SIZE[2]*2],
        [#PROC_CHUNK_SIZE[0]*8, #PROC_CHUNK_SIZE[1]*8, #PROC_CHUNK_SIZE[2]],
        [#PROC_CHUNK_SIZE[0]*4, #PROC_CHUNK_SIZE[1]*4, #PROC_CHUNK_SIZE[2]],
        [#PROC_CHUNK_SIZE[0]*2, #PROC_CHUNK_SIZE[1]*2, #PROC_CHUNK_SIZE[2]],
        #PROC_CHUNK_SIZE]

[supersergiy]

Oh right, that's because you're using expand_bbox_processing and the overhead at the highest levels in very large if you use a huge processing chunk:

[supersergiy]

Subchunkable is working correctly, it's just using expand_bbox_processing with large processing chunk size relative to the bbox at some resolutions is always going to be inefficient. A perfect solution would be to adjust the processing chunk size for stages automatically, but it's not trivial to do it correctly while keeping the interpolate flow fully general, supporting fractional resolution changes etc.

[trivoldus28]

@supersergiy Hmm no what you described is not the problem I'm encountering. The performance hit is purely due to a lot of processes being spawned for some reason. I verified this in real time using top.