Simplify and document building offline/with vendored dependencies

[blenderfreaky]

When building in an offline environment, for example in a Nix sandbox or an airgapped environment, the build will try and fail to find several dependencies. Without internet access, FetchContent will always fail unless FETCHCONTENT_SOURCE_DIR_<name> is set.

@sarnex as discussed in #19347, here is a list of the dependencies I ran into and how they're currently handled:

opencl/CMakeLists.txt

ocl-icd: always uses FetchContent. There's a TODO comment for that already

ocl-headers: you can set OpenCL_HEADERS, but when I try that, it fails. (Cannot specify compile definitions for target "Headers" which is not built by this project.)

sycl/cmake/modules/FetchEmhash.cmake / xptifw/src/CMakeLists.txt

emhash: Should be able to find system install, though I haven't tested that as it isn't pre-packaged for nix

llvm/lib/SYCLLowerIR/CMakeLists.txt

vc-intrinsics: find_package is defined, but when not using that, there's seems to be two ways and I'm unsure which is preferred: adding the repo to llvm/projects/LLVM_EXTERNAL_PROJECTS, or setting LLVMGenXIntrinsics_SOURCE_DIR.

llvm/lib/SYCLNativeCPUUtils/CMakeLists.txt

oneapi-ck: you have to set NATIVECPU_OCK_USE_FETCHCONTENT=Off and OCK_SOURCE_DIR both. Sidenote, this also gives Calling FetchContent_Populate(oneapi-ck) is deprecated

llvm-spirv/CMakeLists.txt

spirv-headers: you have to set LLVM_EXTERNAL_SPIRV_HEADERS_SOURCE_DIR.

unified-runtime/source/common/CMakeLists.txt

unified-memory-framework: will use find_package but only if UR_USE_EXTERNAL_UMF=On is set

hdr_histogram: only used if UR_ENABLE_LATENCY_HISTOGRAM=On is set. Will always use FetchContent

unified-runtime/test/CMakeLists.txt

googletest: will always use FetchContent

This also applies to UMF (which uses FetchContent for googlebenchmark as well)

cmake/FetchLevelZero.cmake

level-zero: needs(?) to be specified via UR_LEVEL_ZERO_LOADER_LIBRARY, UR_LEVEL_ZERO_INCLUDE_DIR
compute-runtime: find_package works and is the default.

UMF

jemalloc only uses find_package on Windows, and always uses FetchContent otherwise.

---

I think it would be nice to have a documentation entry on these dependencies (maybe a brief section on vendored/offline builds).
Also, there's a good bit of inconsistency between these. Some packages default to find_package, some to FetchContent, some toggle based off a boolean flag (UR_USE_EXTERNAL_UMF).

Tangentially, what's the most stable way to get the git tags used (for use in an update script)? Is grepping for set(<DEP>_TAG ([a-z0-9]+)) or similar stable-ish?

[blenderfreaky]

This may be out-of-scope, but unified-runtime/cmake/FetchLevelZero.cmake, sycl/CMakeLists.txt and sycl/cmake/modules/FetchEmhash.cmake contain file(COPY ...) operations that copy from some of the dependency paths.
CMake copies permissions by default, so if the user-provided path points to the nix store or similar, those permissions will be readonly.

With OpenCL headers this causes a compilation failure. It copies a folder from the nix store and the copy ends up read-only. A bit after, it tries to copy more files into that folder, which makes it fail with permission denied.

Patching in NO_SOURCE_PERMISSIONS fixes this edge-case. As there's no executables or anything that may lose executable permission in the relevant folders, I don't think this should cause any issues to add upstream.

[sarnex]

Thanks, first I'll work on making sure we only use FetchContent as a last resort for the missed deps above.

Tangentially, what's the most stable way to get the git tags used (for use in an update script)? Is grepping for set(_TAG ([a-z0-9]+)) or similar stable-ish?

The official releases in this repo are tags of the format v[0-9]+\.[0-9]+\.[0-9]+, the current latest release tag is v6.1.0.

[blenderfreaky]

Thanks, first I'll work on making sure we only use FetchContent as a last resort for the missed deps above.

Thanks a lot!

The official releases in this repo are tags of the format v[0-9]+.[0-9]+.[0-9]+, the current latest release tag is v6.1.0.

What I meant was the tags/revs of the dependencies that FetchContent pulls in. For example with vc-intrinsics, there seems to not have been a proper release since 2022, but a much more recent revision is being pulled in.

[sarnex]

Ah sorry I see, TBH there's no good way and if you have to grep the source we already have a build system problem :)