Fix `torch.nn.functional.hardswish` gradients corner case #148049

zeshengzong · 2025-02-27T03:04:18Z

Changes

Change hardswish gradient compute condition as torch.nn.functional.hardswish
Enable cuda for test test_hardswish_grad_corner
Add test case for value=-3

Test Result

pytest test/test_nn.py -k test_hardswish
pytest test/test_unary_ufuncs.py -k test_hardswish
pytest test/inductor/test_torchinductor.py -k test_hardswish

cc @ezyang @albanD @gqchen @pearu @nikitaved @soulitzer @Varal7 @xmfan @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

pytorch-bot · 2025-02-27T03:04:22Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/148049

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit 98ee89a with merge base 2bcc3ac ():

NEW FAILURE - The following job has failed:

linux-binary-manywheel / manywheel-py3_9-cuda12_8-test / test (gh)
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 8, 0) but found runtime version (9, 7, 1). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN. one possibility is that there is a conflicting cuDNN in LD_LIBRARY_PATH.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / linux-focal-py3_9-clang9-xla / test (xla, 1, 1, lf.linux.12xlarge) (gh) (trunk failure)
ModuleNotFoundError: No module named 'torch.version'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

test/test_nn.py

aten/src/ATen/native/cpu/Activation.cpp

soulitzer

Thanks!

soulitzer · 2025-02-27T21:57:25Z

Failures looks legit:

FAILED [0.7651s] test_jit_fuser_te.py::TestTEFuserStatic::test_hardswish_fwd_bwd - AssertionError: Tensor-likes are not close!

Mismatched elements: 1 / 20 (5.0%)
Greatest absolute difference: 0.06496521830558777 at index (7,) (up to 1e-05 allowed)
Greatest relative difference: 1.0 at index (7,) (up to 1.3e-06 allowed)

To execute this test, run the following from the base repo dir:
    PYTORCH_TEST_WITH_DYNAMO=1 python test/test_jit_fuser_te.py TestTEFuserStatic.test_hardswish_fwd_bwd

CI

zeshengzong · 2025-03-03T11:21:36Z

@soulitzer please check changes when available, thanks!

soulitzer · 2025-03-03T17:27:24Z

torch/csrc/jit/runtime/symbolic_script.cpp

            def backward(grad_output):
                m = (self > 3.).type_as(result)
-                m = torch.where((self >= -3.) & (self <= 3.),  self / 3. + .5, m)
+                m = torch.where((self > -3.) & (self < 3.),  self / 3. + .5, m)


I think you also need to change line 939 to self >= 3.

Changed, thanks!

soulitzer

LGTM, thanks!

soulitzer · 2025-03-04T15:38:08Z

@pytorchbot merge

pytorchmergebot · 2025-03-04T15:41:48Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-03-04T15:42:05Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 1, 3, lf.linux.2xlarge)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

soulitzer · 2025-03-04T15:58:09Z

@pytorchbot merge -i

pytorchmergebot · 2025-03-04T15:59:47Z

Merge started

Your change will be merged while ignoring the following 1 checks: pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 1, 3, lf.linux.2xlarge)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-03-04T16:10:41Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-13)

Details for Dev Infra team

Raised by workflow job

soulitzer · 2025-03-04T16:45:00Z

test/test_nn.py

        inputs.requires_grad = True
        self.assertTrue(gradcheck(F.hardswish, (inputs,)))

-    @onlyCPU


looks like we're failing on mps on some dtypes

Refactor test case make it works on cuda and cpu, is there should have a @onlyCUDAAndCPU annotation?

I don't see one, but I wonder if onlyNativeDeviceTypes works

Changed to @onlyNativeDeviceTypes

zeshengzong · 2025-03-11T11:29:53Z

torch/_decomp/decompositions.py

        0.0,
-        torch.where(self <= 3, grad_output * ((self / 3) + 0.5), grad_output),
+        torch.where(self < 3, grad_output * ((self / 3) + 0.5), grad_output),
    )


Hello @soulitzer, I change here, does the failing test can run locally? Thanks!

hmm there's some amount of setup which I cannot recall on the top of my head, but maybe not too hard to test a small example instead (I've also added some labels which would hopefully test on CI)

pytorch-bot · 2025-03-11T15:50:10Z

To add the ciflow label ciflow/inductor please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-03-11T15:50:10Z

To add the ciflow label ciflow/inductor-perf-compare please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-03-11T15:50:10Z

To add the ciflow label ciflow/inductor-periodic please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

zeshengzong · 2025-03-14T03:18:05Z

Hi @soulitzer, shall we try to merge again, thanks!

soulitzer · 2025-03-14T15:32:58Z

@pytorchbot merge

pytorch-bot · 2025-03-14T15:33:03Z

This PR needs to be approved by an authorized maintainer before merge.

soulitzer · 2025-03-14T15:34:18Z

Hi @soulitzer, shall we try to merge again, thanks!

thanks for the quick fix

soulitzer · 2025-03-14T15:34:24Z

@pytorchbot merge

pytorchmergebot · 2025-03-14T15:36:12Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-03-14T17:55:00Z

Merge failed

Reason: 1 jobs have failed, first few of them are: linux-binary-manywheel / manywheel-py3_9-cuda12_8-test / test

Details for Dev Infra team

Raised by workflow job

soulitzer · 2025-03-14T18:12:32Z

@pytorchbot merge -i

pytorchmergebot · 2025-03-14T18:14:26Z

Merge started

Your change will be merged while ignoring the following 2 checks: pull / linux-focal-py3_9-clang9-xla / test (xla, 1, 1, lf.linux.12xlarge), linux-binary-manywheel / manywheel-py3_9-cuda12_8-test / test

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added module: cpu CPU specific problem (e.g., perf, algorithm) release notes: nn release notes category labels Feb 27, 2025

zeshengzong marked this pull request as ready for review February 27, 2025 03:11

zeshengzong requested review from eqy and syed-ahmed as code owners February 27, 2025 03:12

pytorchbot added the open source label Feb 27, 2025

nikitaved reviewed Feb 27, 2025

View reviewed changes

test/test_nn.py Outdated Show resolved Hide resolved

nikitaved reviewed Feb 27, 2025

View reviewed changes

aten/src/ATen/native/cpu/Activation.cpp Show resolved Hide resolved

nikitaved added the module: autograd Related to torch.autograd, and the autograd engine in general label Feb 27, 2025

zou3519 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 27, 2025

zou3519 requested a review from soulitzer February 27, 2025 16:17

soulitzer previously approved these changes Feb 27, 2025

View reviewed changes

soulitzer reviewed Mar 3, 2025

View reviewed changes

soulitzer previously approved these changes Mar 4, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 4, 2025

pytorchmergebot added the merging label Mar 4, 2025

pytorchmergebot removed the merging label Mar 4, 2025

pytorchmergebot added the merging label Mar 4, 2025

pytorchmergebot removed the merging label Mar 4, 2025

soulitzer reviewed Mar 4, 2025

View reviewed changes

zeshengzong force-pushed the fix/aten/hardswish_backward branch from 89c2cb0 to 98ee89a Compare March 11, 2025 11:26

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Mar 11, 2025

zeshengzong commented Mar 11, 2025

View reviewed changes

soulitzer added ciflow/inductor ciflow/inductor-perf-compare ciflow/inductor-periodic labels Mar 11, 2025

pytorch-bot bot removed ciflow/inductor ciflow/inductor-perf-compare ciflow/inductor-periodic labels Mar 11, 2025

soulitzer approved these changes Mar 14, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 14, 2025

pytorchmergebot added the merging label Mar 14, 2025

pytorchmergebot removed the merging label Mar 14, 2025

pytorchmergebot added the merging label Mar 14, 2025

pytorchmergebot closed this in 97272e4 Mar 14, 2025

pytorchmergebot removed the merging label Mar 14, 2025

daisyden mentioned this pull request Sep 4, 2025

nn.hardswish grad is incorrect at corner case intel/torch-xpu-ops#2014

Closed

Uh oh!

Fix torch.nn.functional.hardswish gradients corner case #148049

Fix torch.nn.functional.hardswish gradients corner case #148049

Uh oh!

Conversation

zeshengzong commented Feb 27, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Test Result

Uh oh!

pytorch-bot bot commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/148049

❌ 1 New Failure, 1 Unrelated Failure

Uh oh!

Uh oh!

Uh oh!

soulitzer left a comment

Choose a reason for hiding this comment

Uh oh!

soulitzer commented Feb 27, 2025

Uh oh!

zeshengzong commented Mar 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soulitzer left a comment

Choose a reason for hiding this comment

Uh oh!

soulitzer commented Mar 4, 2025

Uh oh!

pytorchmergebot commented Mar 4, 2025

Merge started

Uh oh!

pytorchmergebot commented Mar 4, 2025

Merge failed

Uh oh!

soulitzer commented Mar 4, 2025

Uh oh!

pytorchmergebot commented Mar 4, 2025

Merge started

Uh oh!

pytorchmergebot commented Mar 4, 2025

Merge failed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pytorch-bot bot commented Mar 11, 2025

Uh oh!

pytorch-bot bot commented Mar 11, 2025

Uh oh!

pytorch-bot bot commented Mar 11, 2025

Uh oh!

zeshengzong commented Mar 14, 2025

Uh oh!

soulitzer commented Mar 14, 2025

Uh oh!

pytorch-bot bot commented Mar 14, 2025

Uh oh!

soulitzer commented Mar 14, 2025

Uh oh!

soulitzer commented Mar 14, 2025

Uh oh!

pytorchmergebot commented Mar 14, 2025

Merge started

Uh oh!

pytorchmergebot commented Mar 14, 2025

Fix `torch.nn.functional.hardswish` gradients corner case #148049

Fix `torch.nn.functional.hardswish` gradients corner case #148049

zeshengzong commented Feb 27, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Feb 27, 2025 •

edited

Loading