BUG: 8.3.0+ builds break compatibility with Debian Buster and Ubuntu <= 18.04

[rathboma]

From 8.3.0 onward the build image is now Ubuntu 20.04 as GitHub Actions has removed Ubuntu 18.04 support.

Unfortunately this brings back an old set of issues -- any app using the prebuilds of better-sqlite3 will experience crashes when distributed through Snap, or when running on a system that contains a lower version of glibc (which is a LOT of systems).

This will specifically be a problem for those running prebuilds on servers with Ubuntu 16.04, 18.04, 14.04, or Debian Buster, all of which are still supported and used in many businesses. It is also a problem for Electron apps (mine included)

Context: Glibc 2.29 is a breaking change, but is only present on newer OSes.

SQLite3 is perfectly happy being built with older versions of glibc, so maybe there's a way to get those older versions on the build machines, or to build inside a docker container using a Ubuntu-18.04 or Debian buster image?

Related, & previously resolved by downgrading to Ubuntu 18.04 build images:

• glibc_2.29 not found beekeeper-studio/beekeeper-studio#1566
• Use of GLIBC 2.29 prevents pre-builds from being usable #586
• Installation error: version `GLIBC_2.29' not found #403

[mceachen]

Thanks for taking the time to do this research--apologies for the regression.

Can you submit a PR that reverts #982 ? @JoshuaWise has granted me a small subset of permissions for this repo--I can't self-approve any changes, for example.

[rathboma]

@mceachen I don't think it can be reverted. GitHub no longer supports the 18.04 build image

[mceachen]

SADNESS actions/runner-images#6002

[rathboma]

Yeah, that's why I wanted to start the discussion again, because I'm not sure this is going to be super easy to solve without some bigger build changes :-/

[mceachen]

I was hoping that glibc had some sort of autoconf/environment variable magicks for GLIBC_V2.21_MODE, but it seems that newer glibc removed older kernel support, so that's not going to happen.

All my googling seems to point to using a VM/docker instance to restore ubuntu 18 builds.

[rathboma]

Yeah same. Seems like it is either that or a CHROOT with a specific version of glibc installed.

I think 2.29 introduced a bunch of new stuff that isn't backwards compatible, because usually they're very good about backwards compatibility

[rathboma]

Or I guess installing a local version of an older glibc somewhere and using that one?

[mceachen]

I looked at this a bit more.

I would suggest we drop Ubuntu 18 support. EOL is at the end of this month. This will be @JoshuaWise 's call.

If we do want to try to kick the can down the road a couple months, we can push this out until september by adding this GHA job:

  prebuild-glibc:
    name: Prebuild on ancient glibc
    runs-on: ubuntu-latest
    container: node:16-stretch
    needs: publish
    steps:
      - uses: actions/checkout@v3
      - run: apt-get update && apt-get install build-essential git python3
      - run: npm install --ignore-scripts
      - run: ${{ NODE_BUILD_CMD }} -u ${{ secrets.GITHUB_TOKEN }}

The stretch build has 2.24:

$ docker run -it node:16-stretch sh
# head -1 /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
# ldd --version
ldd (Debian GLIBC 2.24-11+deb9u4) 2.24

[rathboma]

Great! I think pushing it out is a great idea.

Standard support for 18.04 ends soon, but ESM support runs for a few more years, many individuals get this for free, and most enterprises use it also. 14.04 is still supported with ESM until 2024!

[rathboma]

Never seen that container directive before. I'm going to try that myself

[mceachen]

I didn't test the new draft job yet, but see c99fca6 for what I was thinking.

[rathboma]

Looks great. It's very frustrating to have to micro manage builds like this right? I have the same frustrations, lol

[m4heshd]

@mceachen This is why I explicitly mentioned this HUGE breaking change in the PR note. (#982)

This change is necessary since GitHub has dropped support for the ubuntu-18.04 image. (actions/runner-images#6002)

This will also mean that Linux distributions with only glibc version 2.29 and above will be supported by the prebuilt binaries.

I've shopped around for many workarounds for this before the change. Unfortunately, we have to go along with all these deprecations and there's no way around it. I've even looked at using a custom runner image (I usually waste a ton of time looking for workarounds even when they're quite unnecessary). But that's practically useless since Node 18 itself requires glibc >= 2.28 and Node 16 is soon EOL. That timeline unfortunately didn't line up with GH's support drop for the Ubuntu image. Polluting the build workflow for mere short-term backward compatibility just felt wrong. But that's just my way of doing things. I would love some input from @JoshuaWise on this.

@rathboma If this is a necessity for your application, I think a solution on the application's scope would fit much better since you have full control of the building, packaging, and distribution of your app. This is just my own opinion. I never use prebuilt binaries on my distributions. I always rebuild them while packaging. You can use a self-hosted runner while doing that to please your requirements. Please correct me if I'm missing anything here because I have no experience in distributing in packmans like Snap.