`include!(concat!(env!("OUT_DIR"), "/bindings.rs"))` should work

[matklad]

To support code-gen from build.rs, we should fully support all elements of include!(concat!(env!("OUT_DIR"), "/bindings.rs")) incantation.

This needs a few moving parts to be in place:

• we should extract the OUT_DIR from cargo. It looks like this info might be missing from cargo metadata at present, but we can always start with heuristic, while working on proper solution upstream in parallel
• we need to support env! macro. To do this, we need to add an env: FxHashMap<String, String> to the CrateGraph, similarly to how we added cfg recently. When expanding env!, we should look in the map first, and fall back to real std::env::var after that.
• include! and concat! seem to use "eager expansion" of some sorts. We need to figure what this means, how it is implemented in rustc, and add it to rust-analyzer. Might be tricky.
• handling the actual include! shouldn't be hard: we only need to plug the included file into the raw_items machinery. Work to support include!("some-path.rs") can be done in parallel to other bullet points here

[metajack]

The use case for this in Libra is that we generate protobufs using prost in build.rs. Our developers would very much like access to these in their IDEs, Soecifically they want to see type signatures when they are using some protobuf.

[davidbarsky]

+1 to @metajack's comment. This is really common when generating RPC bindings of any kind. Alternatives would include having the build.rs modify the cargo src directory, but those crates are rejected from crates.io and cross-compilation tools such as cross assume a read-only file system.

---

While this is an unstable options, cargo build --build-plan -Z unstable-options does contain a seemingly non-optional OUT_DIR field in the env object. For instance, running cargo build --build-plan -Z unstable-options on rust-analyzer yield one of many JSON objects like:

    {
      "package_name": "memchr",
      "package_version": "2.2.1",
      "target_kind": [
        "lib"
      ],
      "kind": "Host",
      "compile_mode": "build",
      "deps": [
        167
      ],
      "outputs": [
        "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps/libmemchr-c3055839db19f1ef.rlib",
        "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps/libmemchr-c3055839db19f1ef.rmeta"
      ],
      "links": {},
      "program": "rustc",
      "args": [
        "--crate-name",
        "memchr",
        "/Users/dbarsky/.cargo/registry/src/gitproxy.zycloud.tk-1ecc6299db9ec823/memchr-2.2.1/src/lib.rs",
        "--error-format=json",
        "--json=diagnostic-rendered-ansi,artifacts",
        "--crate-type",
        "lib",
        "--emit=dep-info,metadata,link",
        "-C",
        "debuginfo=2",
        "--cfg",
        "feature=\"default\"",
        "--cfg",
        "feature=\"use_std\"",
        "-C",
        "metadata=c3055839db19f1ef",
        "-C",
        "extra-filename=-c3055839db19f1ef",
        "--out-dir",
        "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps",
        "-L",
        "dependency=/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps",
        "--cap-lints",
        "allow"
      ],
      "env": {
        "CARGO": "/Users/dbarsky/.rustup/toolchains/nightly-2019-10-04-x86_64-apple-darwin/bin/cargo",
        "CARGO_MANIFEST_DIR": "/Users/dbarsky/.cargo/registry/src/gitproxy.zycloud.tk-1ecc6299db9ec823/memchr-2.2.1",
        "CARGO_PKG_AUTHORS": "Andrew Gallant <jamslam@gmail.com>:bluss",
        "CARGO_PKG_DESCRIPTION": "Safe interface to memchr.",
        "CARGO_PKG_HOMEPAGE": "https://github.com/BurntSushi/rust-memchr",
        "CARGO_PKG_NAME": "memchr",
        "CARGO_PKG_REPOSITORY": "https://github.com/BurntSushi/rust-memchr",
        "CARGO_PKG_VERSION": "2.2.1",
        "CARGO_PKG_VERSION_MAJOR": "2",
        "CARGO_PKG_VERSION_MINOR": "2",
        "CARGO_PKG_VERSION_PATCH": "1",
        "CARGO_PKG_VERSION_PRE": "",
        "DYLD_FALLBACK_LIBRARY_PATH": "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps:/Users/dbarsky/.rustup/toolchains/nightly-2019-10-04-x86_64-apple-darwin/lib:/Users/dbarsky/.rustup/toolchains/nightly-2019-10-04-x86_64-apple-darwin/lib:/Users/dbarsky/lib:/usr/local/lib:/usr/lib",
        "OUT_DIR": "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/build/memchr-24c08495884abe2b/out"
      },
      "cwd": "/Users/dbarsky/.cargo/registry/src/gitproxy.zycloud.tk-1ecc6299db9ec823/memchr-2.2.1"
    },

However, I suspect that cargo build --build-plan -Z unstable-options might not be a viable option to rust-analyzer due to its instability.

[matklad]

Another option for code-gen is to generate files to src *outside* of build.rs, store generated files in the repo, and use a test to verify that generated files are fresh. This has the benefit that downstream crates do not need to build code generator and its deps. This is what rust-analyzer does for the ast generation. But we really should just fix this :)

…

On Monday, 7 October 2019, David Barsky ***@***.***> wrote: +1 to @metajack <https://github.com/metajack>'s comment. This is *really* common when generating RPC bindings of *any* kind. Alternatives would include having the build.rs modify the cargo src directory, but those crates are rejected from crates.io *and* cross-compilation tools such as cross assume a read-only file system. ------------------------------ While this is an unstable options, cargo build --build-plan -Z unstable-options *does* contain a seemingly non-optional OUT_DIR field in the env object. For instance, running cargo build --build-plan -Z unstable-options on rust-analyzer yield one of *many* JSON objects like: { "package_name": "memchr", "package_version": "2.2.1", "target_kind": [ "lib" ], "kind": "Host", "compile_mode": "build", "deps": [ 167 ], "outputs": [ "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps/libmemchr-c3055839db19f1ef.rlib", "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps/libmemchr-c3055839db19f1ef.rmeta" ], "links": {}, "program": "rustc", "args": [ "--crate-name", "memchr", "/Users/dbarsky/.cargo/registry/src/gitproxy.zycloud.tk-1ecc6299db9ec823/memchr-2.2.1/src/lib.rs", "--error-format=json", "--json=diagnostic-rendered-ansi,artifacts", "--crate-type", "lib", "--emit=dep-info,metadata,link", "-C", "debuginfo=2", "--cfg", "feature=\"default\"", "--cfg", "feature=\"use_std\"", "-C", "metadata=c3055839db19f1ef", "-C", "extra-filename=-c3055839db19f1ef", "--out-dir", "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps", "-L", "dependency=/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps", "--cap-lints", "allow" ], "env": { "CARGO": "/Users/dbarsky/.rustup/toolchains/nightly-2019-10-04-x86_64-apple-darwin/bin/cargo", "CARGO_MANIFEST_DIR": "/Users/dbarsky/.cargo/registry/src/gitproxy.zycloud.tk-1ecc6299db9ec823/memchr-2.2.1", "CARGO_PKG_AUTHORS": "Andrew Gallant ***@***.***>:bluss", "CARGO_PKG_DESCRIPTION": "Safe interface to memchr.", "CARGO_PKG_HOMEPAGE": "https://github.com/BurntSushi/rust-memchr", "CARGO_PKG_NAME": "memchr", "CARGO_PKG_REPOSITORY": "https://github.com/BurntSushi/rust-memchr", "CARGO_PKG_VERSION": "2.2.1", "CARGO_PKG_VERSION_MAJOR": "2", "CARGO_PKG_VERSION_MINOR": "2", "CARGO_PKG_VERSION_PATCH": "1", "CARGO_PKG_VERSION_PRE": "", "DYLD_FALLBACK_LIBRARY_PATH": "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/deps:/Users/dbarsky/.rustup/toolchains/nightly-2019-10-04-x86_64-apple-darwin/lib:/Users/dbarsky/.rustup/toolchains/nightly-2019-10-04-x86_64-apple-darwin/lib:/Users/dbarsky/lib:/usr/local/lib:/usr/lib", "OUT_DIR": "/Users/dbarsky/Developer/Rust/rust-analyzer/target/debug/build/memchr-24c08495884abe2b/out" }, "cwd": "/Users/dbarsky/.cargo/registry/src/gitproxy.zycloud.tk-1ecc6299db9ec823/memchr-2.2.1" }, However, I suspect that cargo build --build-plan -Z unstable-options might not be a viable option to rust-analyzer due to its instability. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1964>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANB3M6NQF2JFE7WJFSD33DQNOEPPANCNFSM4I6IHPSQ> .

[davidbarsky]

If I'm not mistaken, Cargo will ship with OUT_DIR support in build-script-executed JSON messages (thanks, @matklad!) on January 30th 2020, but I think there might still be some additional work needed in rust-analyzer to expand and understand macros like concat!, include!, and env!. I hope I'm wrong!

Until then, could a rust-project.json (ref. #2434 (comment)) be written by an end user to point rust-analyzer at generated rust code until this feature is properly supported? It is by no means the ideal state, but it would make working with generated bindings (such as RPC-based bindings) a bit easier.

[matklad]

Until then, could a rust-project.json (ref. #2434 (comment)) be written by an end user to point rust-analyzer at generated rust code until this feature is properly supported?

I don't think so, first, an infrastructure to support include! should be written.

[davidbarsky]

I think I might’ve not been precise in my wording: would it be possible to supply arbitrary paths to rust-analyzer so that it’d treat the supplied paths as part of the project?

No worries if the answer is “no” for whatever reason! My proposal is a hack.

[matklad]

If the path is inside the directory with Cargo.toml, that it should already be consider a pat of the project, I think. If the path is outside of Cargo.toml, then project.json can help, yes!