Add Windows support

[iqbal-lab]

Would it be possible to add (64 bit) Windows support to htslib?
I've just emailed Petr Danecek, who says he is unaware of anyone already having done this. If noone is already doing it, we will have a go in my team, but we need only a tiny fraction of htslib functionality (reading BAM, fastq, fastq.gz).

[mp15]

Depends on what you define as Windows support and how the patch was formulated. If the patch was formulated as a platform header that was minimally invasive it would probably fly. Also which compiler are you going for?

[iqbal-lab]

Hi there
I'd obviously want it to be as uninvasive as possible. Plan was to try mingw-64 first, and failing that MS Visual Studio. (I don't have an ongoing development environment in Windows which this needs to fit into, I do most of my work on Linux but need to port my application to Windows.)

[jmarshall]

There's a few different facets here. Certainly being usable on whatever reasonable platforms people want to use it on is a htslib (and samtools and bcftools) goal, and that includes Windows. However building hasn't actually been tested on Windows for a while. Potential issues:

• Much of the code uses C99-style variable declarations in the middle of blocks, which MSVC only supports very recently (from VC++2013 I understand, though I haven't yet tested it myself). If you have an older MSVC, the alternatives are to use a different compiler or to compile the code as C++ (but then you'll get a lot of warnings or worse from the code in htslib/cram/.)
• At present, the makefiles pretty much assume Unix-style compiler invocation -- which will work fine with mingw-64's GCC.
• There's no makefile rule to make a DLL, though the infrastructure is there to add it. You might like to add it, or just build it statically for the time being.

[iqbal-lab]

Thanks, very helpful.

1. One of us has VC2013 and one has an older one (I forget which) - mingw-64 might be easier, also in terms on ongoing support in a mostly linux based system.
2. Building statically would be fine for now

[tonynaggs]

mingw-64 is the preferred toolset
I have Visual Studio 2010, but hopefully this is not needed or only needed for debugging.
(gcc does have some features & extensions which are horrid for porting to other compilers, eg typeof.)

[jkbonfield]

On Wed, Apr 16, 2014 at 08:11:14AM -0700, Tony Naggs wrote:

mingw-64 is the preferred toolset

Mingw has the added benefit of being runnable from linux as a
cross-compiler, allowing it to be used in an automatic build system.
(We do this for Gap5 and it works very well.)

James Bonfield (jkb@sanger.ac.uk) | Hora aderat briligi. Nunc et Slythia Tova
| Plurima gyrabant gymbolitare vabo;
A Staden Package developer: | Et Borogovorum mimzebant undique formae,
https://sf.net/projects/staden/ | Momiferique omnes exgrabure Rathi.

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

[lh3]

Would be good to support VC in the C++ mode. VC seems popular among industrial tools. That said, supporting VC should be of low priority.

[nhayden]

Hi, I'm one of the maintainers of the Rsamtools package for Bioconductor project. I'm trying to get the new and improved htslib into an R package as a replacement for the old samtools. We support Windows with a toolset dictated by the R project, which currently means MinGW-w64. MSVC is not of interest.

Because the effort involved to get htslib ready for MinGW is non-trivial and supporting Windows is of interest to the htslib project, it seems like it'd be worth sharing upstream to see what has already been ruled out. (Before I go ahead with my proposal I should mention that I am an Autotools novice.)

As I started working through htslib code to adapt it to MinGW I stumbled across Gnulib: The GNU portability library, which looks like just what the doctor ordered. The caveat is it relies on use of Automake and Autoconf in the project. At this time all the modules I think are needed from Gnulib are licensed as LGPLv2+. IANAL but that seems to work with the current htslib licensing. Gnulib has a list of ISO C and POSIX Header File Substitutes for what's been addressed for which platforms. I think Gnulib would obviate the need for most of the platform-/feature-checking macros in the source code.

And have you considered Autotools independent of Gnulib? Elements of the build setup look like they were once part of an Autotools-based project. I think much of the complexity of the current Makefile would be addressed automatically by Autotools, while also affording all the configuration goodness that comes with the ./configure make make install incantation.

[jmarshall]

As noted in e.g. #137, we plan to have an autoconf-maintained configure script but not to use automake or the other autotools. This will be landing soon, as it is a prerequisite for merging #146.

@nhayden: What particular functions have you found to be missing in your R / MinGW-w64 environment? We would probably prefer to work around such problems in ways other than Gnulib, so evaluating what is involved really depends on exactly which library functions are problematic.

[nhayden]

I adapted HTSlib using Gnulib and Automake, and then used Libtool to enable building the shared library versions of htslib. If you’d like to try building it I suggest cloning from https://github.com/nhayden/htslib and running autoreconf -i && ./configure --disable-dependency-tracking && make && make check.

We've released an R package Rhtslib that contains htslib. The htslib unit tests (pulled prior to iRODS support) pass on Linux (Ubuntu and Fedora), Mac OS X (10.6.8 and 10.9.5), and MinGW-w64 (Windows).

I compared time required to build various targets on my Ubuntu laptop. The ./configure step (not included) takes 10 secs; bin-programs includes bgzip, tabix, and htsfile. Time is in seconds:

targets	nhayden:develop	samtools:develop
make bin-programs shared-only-libhts	12.7	23.7
" " static-only-libhts	12.8	12.1
" " static-and-shared-libhts	24.2	23.8

So you can see how competitive the Gnulib version is. The discrepancy on the shared-only-libhts is because samtools:develop also needs to build the non-PIC .o files.

Highlights of the Gnulib / Autotools approach

• Streamlined build system with near-zero maintenance cost, added configuration
  I invite you to check out the simplicity of the Makefile.am and test/Makefile.am in comparison to the current build system. Each .c or .h file appears exactly once. The setup in my develop branch handles installing headers and man pages, and automatically offers a slew of configuration options that are currently missing or hand-maintained in the htslib Makefile.
• Near-automatic portability
  With Gnulib I was able to make the codebase work on MinGW-w64 with very few changes to the codebase--mostly adding #include “config.h” to source files and adding the O_BINARY flag where appropriate. With Gnulib integrated the number of supported platforms has increased dramatically, and it’s easy to add modules if new portability issues come up.
• POSIX-style programming, cleanup of source
  I removed some preprocessor platform / feature tests, but could have removed many more, because this is now handled by Gnulib. I think the only preprocessor platform checks that will remain will test for when O_BINARY is necessary, and where the obsolete valloc function is used since it’s not available for Windows. Another advantage is going forward you’ll be able to continue writing htslib in a POSIX-y way and Gnulib will make it “just work” (assuming relevant modules are available).

Integration

I would have preferred to submit a corresponding pull request, but I have not pulled from htslib/develop since iRODS support was added due to the complexity of integrating the configury. I've tried merging from develop branch and most everything merges automatically. I have faked version.h-related stuff and have not updated the pkg-config file.

If you are interested in the Gnulib direction I would be happy to work with you to integrate iRODS support and other configury. I welcome contributions and feedback.

What’s missing from MinGW-w64 / R environment

@jmarshall: It's hard to get an accurate count, as Gnulib operates in terms of "modules" of portability code, and it automatically pulls in module dependencies. Some of the portability issues are trivial, e.g., defining the missing ETIMEDOUT symbol, while others require significant expertise; all Unix-style file control / file system and socket features are missing, for example.

The list of modules--which roughly correspond to either missing / incomplete headers or missing components of unistd.h--is as follows:

• headers: alloca, arpa_inet, fcntl, inttypes, netdb, stat-size (size-related components of sys/stat.h), stddef, stdint, stdlib, string, sys_select, sys_socket (and modules recv, send), sys_stat, sys_time, unistd
• unistd.h: fdatasync, getpagesize,

Licensing

Almost all the modules are LGPLv2+ (or unlimited or public domain); stat-size is GPL. I imported stat-size because the st_blksize member is missing from struct stat.

Is the GPL a problem? It seems like it would be manageable to find a workaround for st_blksize's missingness, or even just fake it with reasonable values for relevant platforms. If I don't import the stat-size module all imported modules and deps are LGPLv2+, unlimited, or public domain.

Again, I welcome contributions and feedback.

[wookietreiber]

👍 @nhayden

[mp15]

For htslib code LGPL is not a problem, but GPL is. The reason for this is that we are pitching htslib as a reference implementation that anyone (including commercial companies) can take and integrate into their code instead of writing their own implementation.

[jkbonfield]

On Fri, Mar 13, 2015 at 03:24:40AM -0700, Martin Pollard wrote:

For htslib code LGPL is not a problem, but GPL is. The reason for
this is that we are pitching htslib as a reference implementation that
anyone (including commercial companies) can take and integrate into
their code instead of writing their own implementation.

I am not sure it matters greatly if it is simply for the Windows
version. Essentially we'd be dual licensing it and all windows code
inherently then becomes GPLed, but the primary code is still also
released under MIT.

However I'm no expert and it's a valid concern that needs consideration.

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

[jkbonfield]

On Thu, Mar 12, 2015 at 04:15:44PM -0700, Nate Hayden wrote:

Almost all the modules are LGPLv2+ (or unlimited or public domain);
stat-size is GPL. I imported stat-size because the st_blksize member
is missing from struct stat.

One question I have on Windows is, does st_blksize actually work? Ie
does it change per file system type? Or alternatively is it just
botched in there as a constant figure (eg 4096) in order to make code
compile correctly by exposing this field in the structure?

If it is the latter then for sure the best route is simply to have a
#ifdef in the one place in htslib that accesses this variable. Make
it return 0 instead and then hfile_init will fall back on picking its
own default for "capacity".

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

[nhayden]

stat-size module

• @jkbonfield It looks like the stat-size module (see stat-size.h, rather readable) relies on DEV_BSIZE, BSIZE, or BBSIZE to be defined or it falls through to 4096 as you guessed. (param.h exists in the MinGW-w64 / R environment but is missing those constants.) From there it does some extra probing on platforms for cases where relevant struct stat members can be misreported. I've updated hfile.c to return 0 if struct stat lacks st_blksize member, per your suggestion.

Licensing

• @mp15 @jkbonfield I updated so all included Gnulib modules licensed under LGPLv2+, public domain, or unlimited: 600962a

With licensing concerns out of the way, what are your thoughts on the Gnulib direction?