Update to cmake 3.23.3 #253

mayeut · 2022-06-19T17:38:29Z

Update to cmake 3.23.3
Drop manylinux1 wheels
cmake >= 3.23 requires a version of libuv that can't be built on manylinux1.
This drops manylinux1 wheels in favor of manylinux2010 wheels.

mayeut · 2022-06-19T17:40:09Z

CMakeLists.txt

          string(STRIP "${GLIBC_MINOR_}" GLIBC_MINOR)
          if("${GLIBC_MAJOR}.${GLIBC_MINOR}" VERSION_LESS "2.12")
-            set(UseCustomLibUV ON)
+            # set(UseCustomLibUV ON)


The UseCustomLibUV code path is still there. It could probably be cleaned up ?

now cleaned up

mayeut · 2022-06-19T17:42:00Z

setup.py

+    # Note: This is a bare exception that re-raises so that we don't interfere
+    # with anything the installation machinery might want to do. Because we
+    # print this for any exception this msg can appear (e.g. in verbose logs)
+    # even if there's no failure. For example, SetupRequirementsError is raised
+    # during PEP517 building and prints this text. setuptools raises SystemExit
+    # when compilation fails right now, but it's possible this isn't stable
+    # or a public API commitment so we'll remain ultra conservative.
+    import platform
+    try:
+        import pkg_resources
+    except ImportError:
+        pass
+
+    print(
+        """
+    =============================DEBUG ASSISTANCE=============================
+    If you are seeing a compilation error please try the following steps to
+    successfully install cryptography:
+    1) Upgrade to the latest pip and try again. This will fix errors for most
+       users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
+    2) If on Linux, with glibc < 2.12, you can set PIP_ONLY_BINARY=cmake in
+       order to retrieve the last manylinux1 compatible wheel.
+    3) If on Linux, with glibc < 2.12, you can cap "cmake<3.23" in your
+       requirements in order to retrieve the last manylinux1 compatible wheel.
+    4) Open an issue with the debug information that follows at
+       https://github.com/scikit-build/cmake-python-distributions/issues
+    """
+    )
+    print("    Python: %s" % '.'.join(str(v) for v in sys.version_info[:3]))
+    print("    platform: %s" % platform.platform())
+    if sys.platform.startswith("linux"):
+        try:
+            print("    glibc: %s" % os.confstr("CS_GNU_LIBC_VERSION"))
+        except:
+            try:
+                import ctypes
+                process_namespace = ctypes.CDLL(None)
+                gnu_get_libc_version = process_namespace.gnu_get_libc_version
+                gnu_get_libc_version.restype = ctypes.c_char_p
+                glibc_version = gnu_get_libc_version()
+                if not isinstance(glibc_version, str):
+                    glibc_version = glibc_version.decode("ascii")
+                    print("    glibc: %s" % glibc_version)
+            except:
+                pass
+    if sys.platform.startswith("darwin"):
+        try:
+            macos_ver = subprocess.check_output(
+                [
+                    sys.executable,
+                    "-sS",
+                    "-c",
+                    "import platform; print(platform.mac_ver()[0])",
+                ],
+                universal_newlines=True,
+                env={"SYSTEM_VERSION_COMPAT": "0"},
+            ).strip()
+            print("    macos: %s" % macos_ver)
+        except:
+            try:
+                print("    macos: %s" % platform.mac_ver()[0])
+            except:
+                pass
+    print("    machine: %s" % platform.machine())
+    print("    bits: %d" % (64 if sys.maxsize > 2**32 else 32))
+    for dist in ["pip", "setuptools", "scikit-build"]:
+        try:
+            version = pkg_resources.get_distribution(dist).version
+        except pkg_resources.DistributionNotFound:
+            version = "n/a"
+        print("    {}: {}".format(dist, version))
+    for key in ["PEP517_BUILD_BACKEND"]:
+        if key in os.environ:
+            print("    {}={}".format(key, os.environ[key]))
+    print(
+        """\
+    =============================DEBUG ASSISTANCE=============================
+    """
+    )
+    raise


This mostly comes from cryptography with a bit of rework on the information being dumped.

scikit-build-core can provide a similar dump I think, with a configurable message. I'll keep it in mind when designing it.

mayeut · 2022-06-19T17:59:55Z

The actual message that gets printed on manylinux1 can be seen in this action run.

Here are the stats for last week cmake==3.22.* regarding manylinux1/manylinux2010 switch compatibility:

glibc focused only:

libc_version	percent	download_count	cumulative
2.2	0.01%	22	0.01%
2.3	0.02%	26	0.03%
2.5	0.06%	88	0.09%
2.7	0.02%	35	0.11%
2.9	0.02%	34	0.13%
2.12	0.26%	413	0.39%
2.15	0.00%	2	0.39%
2.17	6.53%	10387	6.92%
2.18	0.01%	13	6.93%
2.19	0.11%	167	7.04%
2.22	0.02%	28	7.06%
2.23	12.32%	19586	19.38%
2.24	0.67%	1070	20.05%
2.26	11.27%	17907	31.31%
2.27	18.63%	29621	49.95%
2.28	13.52%	21489	63.47%
2.29	0.03%	46	63.50%
2.31	33.75%	53646	97.25%
2.32	0.03%	43	97.27%
2.33	0.41%	658	97.69%
2.34	0.27%	426	97.95%
2.35	2.05%	3252	100.00%

manylinux focused:

manylinux	percent	download_count	cumulative
1	11.2%	17470	11.2%
2010	5.6%	8700	16.8%
2014	20.5%	31951	37.3%
2_18	0.0%	13	37.3%
2_19	0.0%	54	37.4%
2_22	0.0%	20	37.4%
2_23	1.8%	2842	39.2%
2_24	0.4%	685	39.6%
2_26	10.0%	15544	49.6%
2_27	14.9%	23223	64.5%
2_28	5.9%	9249	70.5%
2_29	0.0%	35	70.5%
2_30	0.0%	13	70.5%
2_31	26.9%	41924	97.4%
2_32	0.0%	37	97.4%
2_33	0.4%	657	97.9%
2_34	0.2%	361	98.1%
2_35	1.9%	2981	100.0%

henryiii · 2022-06-26T11:58:32Z

The problem is that every manylinux1 usage will break¹ when this gets merged. That's 17k users, or 11% of our users according to the above numbers. Only 0.26% actually are stuck on old systems (and I'd rather bet they are mostly building manuylinux1 wheels too), the others are due to using the old pip version packaged on their system; but still, that's a lot of breakage. One idea could be to drop Python 2 support. That would keep 2.7 users happy. But a lot of them are on 3.6, maybe drop that too? It's irritating because we trivially can support those versions, but not the old Pip's they tend to have. I guess I'd be biased to a breakage a user can fix with a limit / pin over making valid usage break (with an artificial lower limit from us).

I guess there may not be much we can do about it, save to add a note to the readme to cap <3.23 if you need to support manylinux1 (usually due to pip version). ~~I think we still need to make one more 3.22 release~~² (maybe we should try adding the ability to target external CMake when building before we do that?), then put this in.

Break meaning start building from scratch, which of course might succeed. ↩
Hadn't realized you already released. Okay, guess it's not that important that someone be able to target an existing binary before dropping manylinux1. ↩

henryiii · 2022-06-26T12:00:50Z

I like the error message, quite helpful!

setup.py

nortismo · 2022-07-21T06:30:45Z

Hi there!
First of all thank you for your work! Are there any plans to continue on this and merge it soon? The update would be super useful to me!

cmake >= 3.23 requires a version of libuv that can't be built on manylinux1. This drops manylinux1 wheels in favor of manylinux2010 wheels.

No need to build a custom libuv anymore, remove dead code.

mayeut · 2022-07-31T08:19:59Z

I guess I'd be biased to a breakage a user can fix with a limit / pin over making valid usage break (with an artificial lower limit from us).

same here, I don't want to introduce an artificial limit on cmake when 95% of breakage can be solved by a pip update. For the other 5% of breakage, users can pin.

maybe we should try adding the ability to target external CMake when building before we do that?

I don't think we need to wait for that (and I'm not clear on how to implement that either).

The only thing that might be worth adding is to repeat points 1, 2 & 3 of the debug message in the README file

henryiii · 2022-07-31T18:23:17Z

I think it's time to do this, and see how much trouble it causes. We can still go back and release 2.22.<whatever version they just released> if we want to get any new features (like the ability to target external CMake versions. I was waiting to release to add that, but we don't have to do them in order.

I do want to get this out before 2.24 lands, which looks like it could happen any day, and before manylinux2010 is dropped, which is something like tomorrow. ;)

mayeut commented Jun 19, 2022

View reviewed changes

mayeut force-pushed the update-to-cmake-3.23.2 branch from 54db4a6 to 9e742fe Compare June 19, 2022 17:47

mayeut force-pushed the update-to-cmake-3.23.2 branch 4 times, most recently from 9940f0d to 02510da Compare June 19, 2022 20:43

mayeut marked this pull request as ready for review June 20, 2022 06:18

henryiii mentioned this pull request Jun 26, 2022

Update to CMake 3.23.0 #232

Closed

henryiii reviewed Jun 26, 2022

View reviewed changes

setup.py Show resolved Hide resolved

mayeut force-pushed the update-to-cmake-3.23.2 branch from 02510da to 1955c08 Compare June 26, 2022 14:54

henryiii approved these changes Jun 29, 2022

View reviewed changes

mayeut force-pushed the update-to-cmake-3.23.2 branch from e480970 to caf051d Compare July 30, 2022 13:13

mayeut changed the title ~~Update to cmake 3.23.2~~ Update to cmake 3.23.3 Jul 30, 2022

mayeut and others added 4 commits July 31, 2022 10:03

Update to CMake 3.23.3

02a0328

Drop manylinux1 wheels

6718532

cmake >= 3.23 requires a version of libuv that can't be built on manylinux1. This drops manylinux1 wheels in favor of manylinux2010 wheels.

docs: mention final manylinux1 version

5c67764

chore: clean-up libuv related code

5c3dc76

No need to build a custom libuv anymore, remove dead code.

mayeut force-pushed the update-to-cmake-3.23.2 branch from a568eba to 5c3dc76 Compare July 31, 2022 08:13

mayeut requested a review from henryiii July 31, 2022 08:13

henryiii approved these changes Jul 31, 2022

View reviewed changes

henryiii merged commit e4dd1d5 into scikit-build:master Jul 31, 2022

mayeut deleted the update-to-cmake-3.23.2 branch August 1, 2022 06:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update to cmake 3.23.3 #253

Update to cmake 3.23.3 #253

Uh oh!

mayeut commented Jun 19, 2022 •

edited

Loading

Uh oh!

mayeut Jun 19, 2022

Uh oh!

mayeut Jul 30, 2022

Uh oh!

mayeut Jun 19, 2022

Uh oh!

henryiii Jun 27, 2022

Uh oh!

mayeut commented Jun 19, 2022 •

edited

Loading

Uh oh!

henryiii commented Jun 26, 2022 •

edited

Loading

Uh oh!

henryiii commented Jun 26, 2022

Uh oh!

Uh oh!

nortismo commented Jul 21, 2022

Uh oh!

mayeut commented Jul 31, 2022

Uh oh!

henryiii commented Jul 31, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Update to cmake 3.23.3 #253

Update to cmake 3.23.3 #253

Uh oh!

Conversation

mayeut commented Jun 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mayeut Jun 19, 2022

Choose a reason for hiding this comment

Uh oh!

mayeut Jul 30, 2022

Choose a reason for hiding this comment

Uh oh!

mayeut Jun 19, 2022

Choose a reason for hiding this comment

Uh oh!

henryiii Jun 27, 2022

Choose a reason for hiding this comment

Uh oh!

mayeut commented Jun 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Jun 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

henryiii commented Jun 26, 2022

Uh oh!

Uh oh!

nortismo commented Jul 21, 2022

Uh oh!

mayeut commented Jul 31, 2022

Uh oh!

henryiii commented Jul 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mayeut commented Jun 19, 2022 •

edited

Loading

mayeut commented Jun 19, 2022 •

edited

Loading

henryiii commented Jun 26, 2022 •

edited

Loading

henryiii commented Jul 31, 2022 •

edited

Loading