Skip to content

test: with_stderr_(data|contains) has no effect when chained with run_expect_error() #14076

@choznerol

Description

@choznerol

Problem

While working on #14060, I noticed a test case with both with_stderr_contains and run_expect_error would always pass no matter what's passed to with_stderr_contains(...).

  cargo_process("help empty-alias")
        .env("PATH", Path::new(""))
        .with_stderr_contains("THIS RANDOM SENTENCE SHOULD FAIL THIS TEST BUT DIDN'T [..] The subcommand 'empty-alias' wasn't recognized [..]",
        )
        .run_expect_error();

The all 3 usages of run_expect_error in the codebase have the same false positive issue: choznerol@18f4c3f

Steps

  1. Run cargo test --test testsuite help_alias; cargo test --test testsuite config_invalid_empty; cargo test --test testsuite use_mtime_cache_in_cargo_home;. All 3 tests should pass.
  2. Check out this reproduce commit choznerol@18f4c3f on my fork. It just puts random stuff into with_stderr_contains. Rerun step 1 and notice all 3 tests still passed.
  3. (Optional) Same issue even if replacing with_stderr_contains with with_stderr_data. To validate this, you may checkout test: migrate help to snapbox #14060 and run cargo test --test testsuite help_alias.

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.79.0-nightly (0637083df 2024-04-02)
release: 1.79.0-nightly
commit-hash: 0637083df5bbdcc951845f0d2eff6999cdb6d30a
commit-date: 2024-04-02
host: aarch64-apple-darwin
libgit2: 1.7.2 (sys:0.18.3 vendored)
libcurl: 8.4.0 (sys:0.4.72+curl-8.6.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 3.2.1 30 Jan 2024
os: Mac OS 14.4.1 [64-bit]

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-testing-cargo-itselfArea: cargo's testsC-bugCategory: bugE-easyExperience: EasyS-acceptedStatus: Issue or feature is accepted, and has a team member available to help mentor or review

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions