Skip to content

Commit 120cb5a

Browse files
authored
[Doc] Add I/O binding example using onnx data type in python API summary (microsoft#22695)
### Description Add I/O binding example using onnx data type in python API summary. The API is available since 1.20 release. ### Motivation and Context Follow up of microsoft#22306 to add some documentation.
1 parent 4ffc1ff commit 120cb5a

File tree

1 file changed

+28
-1
lines changed

1 file changed

+28
-1
lines changed

docs/python/api_summary.rst

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -244,9 +244,36 @@ You can also bind inputs and outputs directly to a PyTorch tensor.
244244
)
245245
246246
session.run_with_iobinding(binding)
247-
247+
248248
You can also see code examples of this API in in the `ONNX Runtime inferences examples <https://github.com/microsoft/onnxruntime-inference-examples/blob/main/python/api/onnxruntime-python-api.py>`_.
249249

250+
Some onnx data type (like TensorProto.BFLOAT16, TensorProto.FLOAT8E4M3FN and TensorProto.FLOAT8E5M2) are not supported by Numpy. You can directly bind input or output with Torch tensor of corresponding data type
251+
(like torch.bfloat16, torch.float8_e4m3fn and torch.float8_e5m2) in GPU memory.
252+
253+
.. code-block:: python
254+
255+
x = torch.ones([3], dtype=torch.float8_e5m2, device='cuda:0')
256+
y = torch.empty([3], dtype=torch.bfloat16, device='cuda:0')
257+
258+
binding = session.io_binding()
259+
binding.bind_input(
260+
name='X',
261+
device_type='cuda',
262+
device_id=0,
263+
element_type=TensorProto.FLOAT8E5M2,
264+
shape=tuple(x.shape),
265+
buffer_ptr=x.data_ptr(),
266+
)
267+
binding.bind_output(
268+
name='Y',
269+
device_type='cuda',
270+
device_id=0,
271+
element_type=TensorProto.BFLOAT16,
272+
shape=tuple(y.shape),
273+
buffer_ptr=y.data_ptr(),
274+
)
275+
session.run_with_iobinding(binding)
276+
250277
251278
API Details
252279
===========

0 commit comments

Comments
 (0)