Use ptrdiff_t sized offsets for gvars_offsets to allow large sysimages #57366
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gbaraldi
added
backport 1.11
vtjnash
approved these changes
Feb 12, 2025
|
Can you test that this is working before merging? |
giordano
reviewed
Feb 12, 2025
test/compileall.jl
Outdated
| @test success(`$(Base.julia_cmd()) -J $(dir)/sys.so -e 'Base.scrub_repl_backtrace(nothing); exit()'`) | ||
|
|
||
|
|
||
| if !Sys.iswindows() #Windows doesn't support large images |
There was a problem hiding this comment.
Does this work on 32-bit systems? Edit: ah, you skip the test below? Isn't it simpler to just exclude 32-bit systems together with Windows here?
giordano
force-pushed
the
gb/size-t-offset
branch
from
8fa9985 to
4170a4a
Compare
|
x86_64 Darwin tests are systematically failing on this PR despite multiple retries. |
|
This keeps failing: https://buildkite.com/julialang/julia-master/builds/45061#01952ab6-a051-466e-aa0a-49ff9c521962/758-1238 |
vtjnash
reviewed
Mar 14, 2025
There was a problem hiding this comment.
I think you are relying on the .ldata section existing to be allowed to use large images like this, but that is an ELF-only feature. Since we use the Small or Medium CodeModels, individual object file relocations will only be able to reference a shared object smaller than about 2GB
|
I guess we need to detect large images and switch to the large code model. (Maybe printing an info that the image is very large and will have lower performance. I think aarch64 is fine here because there the code model allows for 4GB relocations |
JeffBezanson
removed
the
merge me
This was referenced Mar 24, 2025
51 tasks
This was referenced Apr 25, 2025
58 tasks
|
anything i can do to help get this merged for 1.12-rc2? |
|
The issue here is that this doesn't fix the underlying issue that OSs don't support shared libraries over 2GB/4GB. |
This was referenced Aug 19, 2025
|
If this doesn't fix things right now, should we remove the 1.12 label? |
|
Yeah, I'm in favor of trying a different approach. I think we probably need to pursue the split file (code in Maybe close and file an issue for the changes? |
|
The compression also gives us quite a bit more room. |
gbaraldi
removed
backport 1.11
DilumAluthge
added a commit
that referenced
this pull request
Sep 5, 2025
Backported PRs: - [x] #54840 <!-- Add boundscheck in speccache_eq to avoid OOB access due to data race --> - [x] #42080 <!-- recommend explicit `using Foo: Foo, ...` in package code (was: "using considered harmful") --> - [x] #58127 <!-- [DOC] Update installation docs: /downloads/ => /install/ --> - [x] #58202 <!-- [release-1.11] malloc: use jl_get_current_task to fix null check --> - [x] #58584 <!-- Make `Ptr` values static-show w/ type-information --> - [x] #58637 <!-- Make late gc lower handle insertelement of alloca use. --> - [x] #58837 <!-- fix null comparisons for non-standard address spaces --> - [x] #57826 <!-- Add a `similar` method for `Type{<:CodeUnits}` --> - [x] #58293 <!-- fix trailing indices stackoverflow in reinterpreted array --> - [x] #58887 <!-- Pkg: Allow configuring can_fancyprint(io::IO) using IOContext --> - [x] #58937 <!-- Fix nthreadpools size in JLOptions --> - [x] #58978 <!-- Fix precompilepkgs warn loaded setting --> - [x] #58998 <!-- Bugfix: Use Base.aligned_sizeof instead of sizeof in Mmap.mmap --> - [x] #59120 <!-- Fix memory order typo in "src/julia_atomics.h" --> - [x] #59170 <!-- Clarify and enhance confusing precompile test --> Need manual backport: - [ ] #56329 <!-- loading: clean up more concurrency issues --> - [ ] #56956 <!-- Add "mea culpa" to foreign module assignment error. --> - [ ] #57035 <!-- linux: workaround to avoid deadlock inside dl_iterate_phdr in glibc --> - [ ] #57089 <!-- Block thread from receiving profile signal with stackwalk lock --> - [ ] #57249 <!-- restore non-freebsd-unix fix for profiling --> - [ ] #58011 <!-- Remove try-finally scope from `@time_imports` `@trace_compile` `@trace_dispatch` --> - [ ] #58062 <!-- remove unnecessary edge from `exp_impl` to `pow` --> - [ ] #58157 <!-- add showing a string to REPL precompile workload --> - [ ] #58209 <!-- Specialize `one` for the `SizedArray` test helper --> - [ ] #58108 <!-- Base.get_extension & Dates.format made public --> - [ ] #58356 <!-- codegen: remove readonly from abstract type calling convention --> - [ ] #58415 <!-- [REPL] more reliable extension loading --> - [ ] #58510 <!-- Don't filter `Core` methods from newly-inferred list --> - [ ] #58110 <!-- relax dispatch for the `IteratorSize` method for `Generator` --> - [ ] #58965 <!-- Fix `hygienic-scope`s in inner macro expansions --> - [ ] #58971 <!-- Fix alignment of failed precompile jobs on CI --> - [ ] #59066 <!-- build: Also pass -fno-strict-aliasing for C++ --> Contains multiple commits, manual intervention needed: - [ ] #55877 <!-- fix FileWatching designs and add workaround for a stat bug on Apple --> - [ ] #56755 <!-- docs: fix scope type of a `struct` to hard --> - [ ] #57809 <!-- Fix fptrunc Float64 -> Float16 rounding through Float32 --> - [ ] #57398 <!-- Make remaining float intrinsics require float arguments --> - [ ] #56351 <!-- Fix `--project=@script` when outside script directory --> - [ ] #57129 <!-- clarify that time_ns is monotonic --> - [ ] #58134 <!-- Note annotated string API is experimental in Julia 1.11 in HISTORY.md --> - [ ] #58401 <!-- check that hashing of types does not foreigncall (`jl_type_hash` is concrete evaluated) --> - [ ] #58435 <!-- Fix layout flags for types that have oddly sized primitive type fields --> - [ ] #58483 <!-- Fix tbaa usage when storing into heap allocated immutable structs --> - [ ] #58512 <!-- Make more types jl_static_show readably --> - [ ] #58012 <!-- Re-enable tab completion of kwargs for large method tables --> - [ ] #58683 <!-- Add 0 predecessor to entry basic block and handle it in inlining --> - [ ] #59112 <!-- Add builtin function name to add methods error --> Non-merged PRs with backport label: - [ ] #59329 <!-- aotcompile: destroy LLVM context after serializing combined module --> - [ ] #58848 <!-- Set array size only when safe to do so --> - [ ] #58535 <!-- gf.c: include const-return methods in `--trace-compile` --> - [ ] #58038 <!-- strings/cstring: `transcode`: prevent Windows sysimage invalidation --> - [ ] #57604 <!-- `@nospecialize` for `string_index_err` --> - [ ] #57366 <!-- Use ptrdiff_t sized offsets for gvars_offsets to allow large sysimages --> - [ ] #56890 <!-- Enable getting non-boxed LLVM type from Julia Type --> - [ ] #56823 <!-- Make version of opaque closure constructor in world --> - [ ] #55958 <!-- also redirect JL_STDERR etc. when redirecting to devnull --> - [ ] #55956 <!-- Make threadcall gc safe --> - [ ] #55534 <!-- Set stdlib sources as read-only during installation --> - [ ] #55499 <!-- propagate the terminal's `displaysize` to the `IOContext` used by the REPL --> - [ ] #55458 <!-- Allow for generically extracting unannotated string --> - [ ] #55457 <!-- Make AnnotateChar equality consider annotations --> - [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows --> - [ ] #53957 <!-- tweak how filtering is done for what packages should be precompiled --> - [ ] #51479 <!-- prevent code loading from lookin in the versioned environment when building Julia --> - [ ] #50813 <!-- More doctests for Sockets and capitalization fix --> - [ ] #50157 <!-- improve docs for `@inbounds` and `Base.@propagate_inbounds` --> --------- Co-authored-by: Kiran Pamnany <[email protected]> Co-authored-by: adienes <[email protected]> Co-authored-by: Gabriel Baraldi <[email protected]> Co-authored-by: Keno Fischer <[email protected]> Co-authored-by: Simeon David Schaub <[email protected]> Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Alex Arslan <[email protected]> Co-authored-by: Fons van der Plas <[email protected]> Co-authored-by: Ian Butterworth <[email protected]> Co-authored-by: JonasIsensee <[email protected]> Co-authored-by: Curtis Vogt <[email protected]> Co-authored-by: Dilum Aluthge <[email protected]> Co-authored-by: DilumAluthgeBot <[email protected]> Co-authored-by: DilumAluthge <[email protected]>
|
I like that you're doing a fix or some sort of workaround. Though it excludes Windows, and we really want Windows to have a level playing field?! 2GB sysimage seems way excessive, also 4GB, and could we have the gvar only 32-bit but unsigned for now, giving a bit of breathing room, even though not a full solution? How many gvar are there? Should we be worried about them expanding (four extra zeros for each for normal sized sysimages)? Compression is not an alternate to this you mean, you mean a mitigation to expanded pointers? For languages we compare ourself to are 2-4+ GB .exe (or zip or .jar etc) files normal or too be expected? The sysimage is is our sort of .exe subsitute, and we rather want arbitrary large code, or at least reasonably large max. It just seems there should be ways to trim more and/ore move around to other files, ande larger sysimage a last resort. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hopefully this fixes JuliaLang/PackageCompiler.jl#1019. This does mean a very slight size increase (each offset is twice as large but shouldn't make too much difference (it's an extra 32bit per gvar))