[RELAY][OP] Relay Operator Sprint

[tqchen]

Now that the Relay RFC is being merged and we are stabilizing the type inference interface, we should sprint to add new operators to relay to make it on parity with NNVM.

#1798 shows an example on how to do so for conv2d operator.

General Steps of Porting

• Implement the TypeRelation function, when necessary
  • The shapes represented by IndexExpr(symbolic integer)
    • When possible, support symbolic shape inference
    • You can, however, get the integer out from symbolic shape if it is a must, that will require the inference to work on concrete shapes.
  • User reporter->Assign to set the inferred result
  • Use reporter->AssertEQ to assert symbolic integer equivalence
    • It will return false if there is an unsatisfied constraint
• Use tvm::Attrs to replace dmlc::Parameter
• We switch to directly create python wrappers by calling into positional functions so that the operator signature is explicit in python

General Principles

• Numpy consistency, always consistent with numpy
  • All binary operators broadcast
  • This means we will use add, subtract instead of broadcast_add, broadcast_sub ...
  • elemwise_add version will not be supported for now as we can just use the broadcast version
• Consistent with nnvm when possible
• Fields in Attrs
  • Use concrete types when possible(int, string, bool)
  • If you need None, you can use IndexExpr, which gives you that

List of Operators to be covered

Generally, we need to cover everything we have so far https://docs.tvm.ai/nnvm_top.html
Please use this issue to coordinate what you will be working on. As we expect things to move quickly, try to do "fine grained locking" and only claim things that you are working on right now and aim to get things in a few days.

The List

Level 1: Common Basic Ops

Enough to get MLP

• nn.dense
• nn.relu
• tanh
• sigmoid
• exp
• log
• sqrt
• add
• subtract
• multiply
• divide
• mod
• nn.batch_flatten
• concatenate
• nn.softmax
• nn.log_softmax
• nn.batch_norm
• nn.dropout
• expand_dims

Level 2: Convolutions

Enough to get convnet

• nn.conv2d
• nn.conv2d_transpose
• nn.max_pool2d
• nn.avg_pool2d
• nn.global_max_pool2d
• nn.global_avg_pool2d
• nn.pad
• nn.lrn

Level 3: Additional Math And Transform Operators

• reshape
• copy
• negative
• floor
• ceil
• round
• trunc
• clip
• abs
• leaky_relu
• tranpose
• split
• squeeze
• take
• full
• zeros
• ones
• transpose
• zeros_like
• ones_like

Level 4: All broadcast and reduction functions that are not in previous level

• pow
• less
• greater
• less_than
• greater_than
• right_shift
• left_shift
• maximum
• minimum
• sum
• max
• prod
• argmax, argmin
• strided_slice
• broadcast_to
• where

Level 5: Vision Operators

• image.resize
• vision.multibox_prior
• vision.nms

Level 10: Backend Operators

Operators necessary as intermediate stage of optimizations, or gradient, can be influx

[MarisaKirisame]

I had 'implemented' some elem wise function which I need for ad (negative, multiplication, division)
('implemented' because it isnt possible to lower code right now)
I can take some operators, and I dont have any particular preference.

[srkreddy1238]

I could get through resize operator (will PR once #1798 is merged) to start with and proceeding with transforms.

[junrushao]

I strongly agree with the numpy consistency, e.g. nnvm.symbol.flatten should be renamed.

A good example could be that TensorFlow's API uses batch_xxx for batched operators.

[junrushao]

I can work on some shape-related APIs.

[srkreddy1238]

@junrushao1994
Can you share the ops you are working on to avoid the duplication?

[srkreddy1238]

@tqchen
How about relaying an assert on a condition which involve two variables ?
Ref. Transposed convolution where channels should divide groups and both are variables.

[junrushao]

@srkreddy1238
I am starting with expand_dims, the easiest one
#1819

[tqchen]

One note about int32 vs int64 when constructing constant was raised by @junrushao1994 @srkreddy1238 . This is an issue we should think about now. int32 will likely cause some regression on large arrays which need to be fixed. I think we should prefer int64 when possible for constants, and let the compiler to automatically detect and downgrade to int32.

A temporary workaround is always to keep the inferred shape type consistent with the input shape type, and we can make the switch more easily in one place later

[junrushao]

Another thing I am concerning is user friendliness.

First, examples provided by Python API docs should be at least runnable by copy-pasting, like PyTorch (https://pytorch.org/docs/stable/tensors.html) or NumPy (https://docs.scipy.org/doc/numpy/reference/generated/numpy.expand_dims.html).

Second, Python API docs should be self-contained, at least those designed for DL practitioners who may not take a good look at the C++ code.

It does not seem to be a big deal for now, but we should put more effort into it in the future.

[tqchen]

+1 for API docs friendless, I would recommend we do it now than later. Maybe I am having a bad lead example in the conv2d docs as it was pretty minimum, I will send an updated PR to update that, and let us make sure the new ops are being well documented with examples, especially non-trival ones

[srkreddy1238]

Expr like below is not getting simplified !!

TensorTypeNode(float32, [n, c, int32((float32(100)*2.000000f)), int32((float32(200)*2.000000f))])

any idea ?

[tqchen]

The eager CSE is done among integer expression only so far. For floating points, we still need to call explicitly simplification, or use as_const_int to get out and explicit simplify

[junrushao]

I am going to grab transpose (update: no, it does not exist in the list)

I am going to grab less, greater, less_equal, greater_equal...