Linking against Intel ISAL for faster deflate operations

[pettyalex]

Hello,

I see that bgzf compression can optionally link against libdeflate for faster performance, but I was wondering if you've ever evaluated or considered linking against the Intel ISAL? It is BSD licensed, and offers the highest performance deflate implementation that I'm aware of, especially at lower compression ratios.

Perhaps it could be optionally linked, just like libdeflate is right now? Could it be preferred to libdeflate if both are available?

https://github.com/intel/isa-l

Benchmarks:
zlib-ng/zlib-ng#1486
powturbo/TurboBench#43

[jkbonfield]

Looking at those benchmarks it seems like Intel have improved their performance. I did evaluate this in the past and it was simply not a convincing win.

http://www.htslib.org/benchmarks/zlib.html

Although that was Intel's zlib rather than igzip specifically, but you would think it's the same technology in both? Maybe not.

Profiling a samtools view -1 -o /tmp/tmp.bam in.bam command to see where the CPU time is spent, I see this:

  51.73%  samtools  libdeflate.so.0     [.] deflate_compress_fastest
  13.63%  samtools  libdeflate.so.0     [.] deflate_decompress_default
  10.43%  samtools  libdeflate.so.0     [.] deflate_flush_block
   9.95%  samtools  libdeflate.so.0     [.] crc32_x86_pclmul_avx
   2.36%  samtools  [kernel]            [k] 0xffffffff99800190
   2.06%  samtools  libc-2.27.so        [.] __memmove_sse2_unaligned_erms
   1.62%  samtools  libdeflate.so.0     [.] deflate_make_huffman_code
   1.21%  samtools  samtools            [.] bgzf_read
   1.02%  samtools  samtools            [.] bgzf_write
   0.95%  samtools  samtools            [.] bam_read1

Libdeflate is already fast for decoding, but according to the benchmarks may be around half the speed of ISA-L for encoding. So with ISA-L outputting level 1 BAMs may be ~50% faster throughput. We can't tell though if this holds true on the small block sizes bgzf uses without trying it out.

[pettyalex]

I bring this up specifically while my group spends a lot of $ doing various bcftools tasks on cloud platforms that are spending most of their CPU time compressing output, I'm going to see if I can find some time to do this and bring you actual benchmarks on those small block sizes.

Although that was Intel's zlib rather than igzip specifically, but you would think it's the same technology in both? Maybe not.

Their newest zlib is part of Intel IPP, not ISA-L. Intel has too many competing technologies in this space, and things are ripe for confusion: https://www.intel.com/content/www/us/en/developer/articles/guide/data-compression-tuning-guide-on-xeon-systems.html

I just saw that the Intel zlib you linked isn't even the same one as the IPP maybe, but a different thing? It's kind of a nightmare:

Intel's zlib is also not open source, which can cause some pain:

[pettyalex]

Oh, it looks like you already tested igzip and the answer was "nope", so I think that this may not be worth the effort.

I may spend the time anyway just to see it with my own eyes, but you have nice documentation about all this.

[pettyalex]

Closing this, libdeflate is really fast for non-streaming chunks as htslib uses it.

[rhpvorderman]

Libdeflate is already fast for decoding, but according to the benchmarks may be around half the speed of ISA-L for encoding. So with ISA-L outputting level 1 BAMs may be ~50% faster throughput. We can't tell though if this holds true on the small block sizes bgzf uses without trying it out.

So I tried this out a few years back in this (failed) project. Even with all the overhead of the python interface, bgzf encoding was notably faster using ISA-L through python-isal, than samtools.

The thing is that in python it is very easy to use ISA-L because there is no need to link ISA-L as needs to be done in C projects. Given that ISA-L is available in conda-forge, dynamic linking is of course an option, but it will make the building of this project more difficult.

Vendoring ISA-L is also difficult as it requires a nasm assembler to be present. My personal feeling is that optionally dynamically linking ISA-L is the route of the least resistance. Then it can be present in bioconda and by extension in biocontainers and also in Debian (and derivatives) without needing to vendor ISA-L.

As I am the maintainer of python-isal and do know my way around the ISA-L library, I am willing to make the PR and do the benchmarks. But if this PR is going to be shutdown immediately because of the extra build complication it makes no sense to do the effort.

What are your opinions about this, is it worth giving it a try?

[rhpvorderman]

So I did a quick check. Almost 2x faster for a 2% increase in filesize at level 1. That is not too bad.

[jkbonfield]

I see in the zlib-ng benchmark that igzip can be double the speed for compression, at a modest cost in size.

Arguably though we'd only use level 1 for temporary files as in pipes we ought to be using level 0 (uncompressed). [I've been experimenting with just using zstd for that, but that's a totally different beast and obviously not compatible with other tools.]

Thanks for the PR in htslib so we can evaluate it.

[rhpvorderman]

If performance in pipes is lacking I don't think that using zstd will improve it. Given that linux default pipesize is 64K and bgzf blocks are close to that size, increasing the pipe size using fcntl with F_SETPIPE_SZ can lead to better performance because of less blocking pipes.

We do that in xopen to improve the throughput when piping data from pigz (see here). (That is not needed anymore as python-isal now allows threaded decompression with shared memory, but still it was an interesting finding back in the day).

[pettyalex]

I'd be surprised if any compression / decompression is faster than just sending data through a pipe, even at their slowest pipes can get several GB/s through: https://mazzo.li/posts/fast-pipes.html.

Even lz4 only decompresses around 3 or 4 GB/s and it's among the fastest tools of that type.