compatability between mxnet, tvm and nnvm

[kaishijeng]

I have mxnet 0.11.0 installed and mxnet is running OK by itself
However, I got the following error of running python2 deploy_model_on_rasp.py from nnvm tutorial:

[10:06:53] /home/sky/2TB/src/tvm/dmlc-core/include/dmlc/logging.h:308: [10:06:53] /home/sky/2TB/src/mxnet/dmlc-core/include/dmlc/././any.h:289: Check failed: type_->ptype_info == &typeid(T) The stored type mismatch stored=N4nnvm5OpMapISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS6_EES9_EEEE requested=N4nnvm5OpMapISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS6_EES9_EEEE

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/libtvm.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f3d0eb8bd3c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(ZZN4nnvm2Op8set_attrISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS7_EESA_EEEERS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKT_iENKUlPN4dmlc3anyEE_clESR+0x136) [0x7f3d213177d6]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm2Op13UpdateAttrMapERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt8functionIFvPN4dmlc3anyEEE+0xcc) [0x7f3d2243013c]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm2Op8set_attrISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS7_EESA_EEEERS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKT_i+0x190) [0x7f3d213075b0]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(+0x1f77c3) [0x7f3d212727c3]
[bt] (5) /lib64/ld-linux-x86-64.so.2(+0x106ba) [0x7f3d26c206ba]
[bt] (6) /lib64/ld-linux-x86-64.so.2(+0x107cb) [0x7f3d26c207cb]
[bt] (7) /lib64/ld-linux-x86-64.so.2(+0x158e2) [0x7f3d26c258e2]
[bt] (8) /lib64/ld-linux-x86-64.so.2(+0x10564) [0x7f3d26c20564]
[bt] (9) /lib64/ld-linux-x86-64.so.2(+0x14da9) [0x7f3d26c24da9]

terminate called after throwing an instance of 'dmlc::Error'
what(): [10:06:53] /home/fc/2TB/src/mxnet/dmlc-core/include/dmlc/././any.h:289: Check failed: type_->ptype_info == &typeid(T) The stored type mismatch stored=N4nnvm5OpMapISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS6_EES9_EEEE requested=N4nnvm5OpMapISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS6_EES9_EEEE

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/libtvm.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f3d0eb8bd3c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(ZZN4nnvm2Op8set_attrISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS7_EESA_EEEERS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKT_iENKUlPN4dmlc3anyEE_clESR+0x136) [0x7f3d213177d6]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm2Op13UpdateAttrMapERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt8functionIFvPN4dmlc3anyEEE+0xcc) [0x7f3d2243013c]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm2Op8set_attrISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS7_EESA_EEEERS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKT_i+0x190) [0x7f3d213075b0]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(+0x1f77c3) [0x7f3d212727c3]
[bt] (5) /lib64/ld-linux-x86-64.so.2(+0x106ba) [0x7f3d26c206ba]
[bt] (6) /lib64/ld-linux-x86-64.so.2(+0x107cb) [0x7f3d26c207cb]
[bt] (7) /lib64/ld-linux-x86-64.so.2(+0x158e2) [0x7f3d26c258e2]
[bt] (8) /lib64/ld-linux-x86-64.so.2(+0x10564) [0x7f3d26c20564]
[bt] (9) /lib64/ld-linux-x86-64.so.2(+0x14da9) [0x7f3d26c24da9]

Aborted (core dumped)

Any idea why this happen?

[tqchen]

can you check if you can run unittest cases of nnvm compiler alone?

[tqchen]

if you have older version of tvm, it is likely that you will need to rebuild and install tvm with the same version as in nnvm repo.

[kaishijeng]

How do I run unittest cases of nnvm compiler?
My tvm is the latest one from github

[tqchen]

under nnvm, do

nosetests -v tests/python/unittest 
nosetests -v tests/python/compiler

[kaishijeng]

Below is what I got:

test_graph.test_json_pass ... ok
test_graph.test_json_pass_with_attr ... ok
test_graph.test_graph_json_attr ... ok
test_graph.test_list_args ... ok
test_graph.test_infer_shape ... ok
test_graph.test_infer_shape_known_partial ... ok
test_graph.test_infer_type ... ok
test_graph.test_plan_memory ... ok
test_graph.test_print_graph_ir ... ok
test_infer_shape.test_dense ... ok
test_infer_shape.test_concatenate ... ok
test_infer_shape.test_expand_dims ... ok
test_infer_shape.test_split ... ok
test_infer_shape.test_batchnorm ... ok
test_infer_shape.test_flatten ... ok
test_infer_shape.test_conv2d ... ok
test_infer_shape.test_conv2d_transpose ... ok
test_infer_shape.test_max_pool2d ... ok
test_infer_shape.test_global_pool2d ... ok
test_infer_shape.test_reshape ... ok
test_infer_shape.test_transpose ... ok
test_infer_shape.test_broadcast_to ... ok
test_infer_shape.test_broadcast_binary ... ok
test_infer_shape.test_reduce ... ok
test_symbol.test_dense ... ok
test_symbol.test_compose ... ok
test_symbol.test_default_input ... [10:43:26] /home/sky/2TB/src/tvm/dmlc-core/include/dmlc/logging.h:308: [10:43:26] src/core/symbolic.cc:309: Not enough argument to call operator elemwise_add

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/libtvm.so(_ZN4dmlc15LogMessageF atalD1Ev+0x3c) [0x7fab29378d3c]
[bt] (1) /home/sky/2TB/src/nnvm/python/nnvm/../../lib/libnnvm_compiler.so(ZN4nnvm6Symbol7ComposeERKN4dmlc10array_vie wIPKS0_EERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4hashISE_ESt8equal_toISE_ESaIS t4pairIKSE_S4_EEERSK+0x10ef) [0x7fab2873eccf]
[bt] (2) /home/sky/2TB/src/nnvm/python/nnvm/../../lib/libnnvm_compiler.so(NNSymbolCompose+0x2e0) [0x7fab28729470]
[bt] (3) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fab3f0a1e40]
[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7fab3f0a18ab]
[bt] (5) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7fab3f2b13df]
[bt] (6) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7fab3f2b5d82]
[bt] (7) /usr/bin/python(PyObject_Call+0x43) [0x4b0cb3]
[bt] (8) /usr/bin/python(PyEval_EvalFrameEx+0x5faf) [0x4c9faf]
[bt] (9) /usr/bin/python(PyEval_EvalCodeEx+0x255) [0x4c2765]

ok
test_symbol.test_copy ... ok
test_symbol.test_op_name ... ok
test_top_level1.test_dense ... ok
test_top_level1.test_concatenate_split ... ok
test_top_level1.test_expand_dims ... ok
test_top_level1.test_unary ... ok
test_top_level1.test_batchnorm ... ok
test_top_level2.test_conv2d ... ok
test_top_level2.test_max_pool2d ... ok
test_top_level3.test_reshape ... ok
test_top_level3.test_scalar_op ... ok
test_top_level3.test_leaky_relu ... ok
test_top_level4.test_binary_broadcast ... ok
test_top_level4.test_broadcast_to ... ok

---

Ran 41 tests in 0.098s

OK

[tqchen]

I do not have a quite clear clue on what is happening in mxnet side. You can try to upgrade mxnet using the latest pip version, that is how we build docker image to test mxnet frontend https://github.com/dmlc/nnvm/blob/master/tests/ci_build/Dockerfile.gpu

[kaishijeng]

If I import mxnet before nnvm, the issue seems to disappear.

However, I got a different error below which says llvm is not enabled:

TVM: Initializing cython mode...
[11:16:58] /home/sky/2TB/src/tvm/dmlc-core/include/dmlc/logging.h:308: [11:16:58] /home/sky/2TB/src/tvm/src/codegen/codegen.cc:27: Check failed: bf != nullptr Tar get llvm is not enabled

Stack trace returned 10 entries:

[tqchen]

this is mainly because tvm need to build with llvm support, modify config.mk to do that

[kaishijeng]

I installed llvm 5.0/clang 5.0 and is able to pass this error now.
However, another error occurs:

Intrinsic name not mangled correctly for type arguments! Should be: llvm.fmuladd.f32
float (float, float, float)* @llvm.fmuladd.f32.f32.f32
LLVM ERROR: Broken function found, compilation aborted!

[tqchen]

This seems to be an error that was fixed in later version of LLVM5, try llvm from here http://apt.llvm.org/

[tqchen]

The problem of old llvm should also be fixed by #519

[kaishijeng]

It works OK with PR#519
By the way, how do I measure execution time on Raspberry PI and how to compare with a regular mxnet mobilenet on PI?

Thanks,

[tqchen]

You can use the RPC module which comes with remote executor, see https://github.com/dmlc/nnvm/blob/master/examples/benchmark/rasp_imagenet_bench.py and
http://nnvm.tvmlang.org/tutorials/deploy_model_on_rasp.html#sphx-glr-tutorials-deploy-model-on-rasp-py

For MXNet reference, we do need to do it manually by scripting a python script

[tqchen]

I am going to close this issue as the original problem has been resolved. Feel free to open another one if you have other questions