[FEATURE] Add feature of attach_grad to nonleaf variables in HybridizedBlock.

[KexinFeng]

Description

This is the same as this #20559. The PR adds the support for fetching the gradients of intermediate variables in a gluon HybridizedBlock. This applies uniformly to both when block.hybridize() is on and off. This generates the attach_grad implemented in implemented in PR#20500.

The motivation of this feature comes from this issue#11865.

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• block.py where mark_vars and get_mark_vars are added along with MXNDArrayMarkDCVariables.
• cached_op.invoke in cpp backend and CachedOp.__call__ have been editted to include the pass of marked nonleaf ndarrays.
• set_nleafs method is added into CachedOp class to store the marked nonleaf ndarrays.
• Inside void RunGraph, marked nonleaf ndarrays are linked to the marked computational node for autograd computation.

Comments

• This feature is built on top of PR#20500. The modification here is mainly in the invoke of CachedOp computation.

[mxnet-bot]

Hey @KexinFeng , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [website, clang, miscellaneous, sanity, windows-cpu, edge, windows-gpu, centos-cpu, unix-cpu, unix-gpu, centos-gpu]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[KexinFeng]

@mxnet-bot run ci [unix-cpu, unix-gpu, windows-gpu]

[mxnet-bot]

Jenkins CI successfully triggered : [windows-gpu, unix-cpu, unix-gpu]

[KexinFeng]

@mxnet-bot run ci [edge, unix-gpu]

[mxnet-bot]

Jenkins CI successfully triggered : [unix-gpu, edge]

[KexinFeng]

@mxnet-bot run ci [unix-gpu]

[mxnet-bot]

Jenkins CI successfully triggered : [unix-gpu]

[KexinFeng]

@mxnet-bot run ci [unix-gpu]

[mxnet-bot]

Jenkins CI successfully triggered : [unix-gpu]