[Adreno] Enable static texture planning for constant data

[elvin-n]

Previous PR11161 added topi compute and schedules which manages textures in dynamic mode through explicit call of cache_read in adreno primitive schedules. This PR adds top-down approach of memory annotation plus adding support in dependent parts.

1. Passes of analyzing memory scope for expressions in the relay graph and modifies relay graph by adding/modifying VirutalDevice with required memory scope
2. Add support of memory scope into graph memory planner
3. Support of static planned textures in json and opencl runtime
4. Modification of Relay->TIR settling taking into account relay expr virtual devices for variables

[elvin-n]

@csullivan @mbs-octoml PR is ready for review

[mbs-octoml]

Thanks, this is quite the heroic piece of work.

Can I suggest splitting this into smaller chunks?

• Plumbing mem scope through TE (and perhaps handle the mem-scope-per-output issue)
• Mem scope in graph executor mem planner, codegen & runtime.
• 1D & 2D mem pools in graph executor runtime
• Schedules
• New annotator pass, which perhaps should also be made target-specific since the rules for deriving scope from shape seem pretty target specific. Ie it should be moved to a target specific dir, and the pass can be the identity if the current target does not match. Unfortunately there is no generic mechanism for registering a target-specific pass that will work for this.

I'm not sure how to reconcile the fairly generic multi-dimensional buffer support we now have with the hard 1d vs 2d distinction you're adding in graph executor. Is there a discussion about that someplace I've missed?

I can't quite tell if your StorageInfo pass supports tuples & tuple projection. That whole pass needs commenting. But in any case worth mentioning you've eschewed the collect-and-solve constraint approach used by plan devices in favor of an eager back propagation of scopes from argument to constants. Fair enough, but please document the subset of relay this supports.

[mbs-octoml]

See tvm::relay::FlattenTupleType

[elvin-n]

this is done by intend so far - if we use FlattenTupleType, we would analyze each tensor individually that is wrong for this case. Currently I have an impression that if we have tuple been planned in static memory planner, the memory scope should be unique for all tensor contained in tuple

[elvin-n]

I am closing this PR because I split it to several ones:

• PR11874 for handling of properly lowering the relay graph having memory scopes
• PR11875 for changes in JSON file and support of new entites in graph executor
• PR11876 for changes in static memory planner
• PR11878 for Adreno specific markup pass annotating relay expr for memory scope to be handled in above PR

[elvin-n]

@mbs-octoml - several more answers on initial comment

New annotator pass, which perhaps should also be made target-specific since the rules for deriving scope from shape seem pretty target specific.

The annotation pass consist of two parts - generic one and target specific. The generic one goes by graph and detects the targets and then construct name of the function including all found targets. It is not well robust for several targets, but works quite deterministic for one any target. We are introducing Adreno specific transformation, but other can be easily added. If we need to move adreno specific part into target specific directory, I will appreciate if you can suggest the proper place for target specific relay transformations.

I'm not sure how to reconcile the fairly generic multi-dimensional buffer support we now have with the hard 1d vs 2d distinction you're adding in graph executor. Is there a discussion about that someplace I've missed?

There is no fairly generic multi-dimensional buffer in the memory planner - memory manager operates for now by continuous 1d memory blocks or other words flatten 1d buffers. Later on in tir this flatten memory can be managed as multidimensional array using tvm arithmetic. But on the memory management stage it is represented always as 1d memory. Until this PR.

I can't quite tell if your StorageInfo pass supports tuples & tuple projection. That whole pass needs commenting. But in any case worth mentioning you've eschewed the collect-and-solve constraint approach used by plan devices in favor of an eager back propagation of scopes from argument to constants. Fair enough, but please document the subset of relay this supports.

It is still open question how to deal with tuple, I need some examples of ops producing tuples and not be merged into prim function. Could you please share which ops generate tuple of tensors as its output?