onnxruntime Support¶

Introduction of ONNX Runtime¶

ONNX Runtime is a cross-platform inference and training accelerator compatible with many popular ML/DNN frameworks. Check its github for more information.

Installation¶

Please note that only onnxruntime>=1.8.1 of on Linux platform is supported by now.

Install ONNX Runtime python package¶

CPU Version

pip install onnxruntime==1.8.1 # if you want to use cpu version

GPU Version

pip install onnxruntime-gpu==1.8.1 # if you want to use gpu version

Install float16 conversion tool (optional)¶

If you want to use float16 precision, install the tool by running the following script:

pip install onnx onnxconverter-common

Build custom ops¶

Download ONNXRuntime Library¶

Download onnxruntime-linux-*.tgz library from ONNX Runtime releases, extract it, expose ONNXRUNTIME_DIR and finally add the lib path to LD_LIBRARY_PATH as below:

CPU Version

wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz

tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
cd onnxruntime-linux-x64-1.8.1
export ONNXRUNTIME_DIR=$(pwd)
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH

GPU Version

In X64 GPU:

wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz

tar -zxvf onnxruntime-linux-x64-gpu-1.8.1.tgz
cd onnxruntime-linux-x64-gpu-1.8.1
export ONNXRUNTIME_DIR=$(pwd)
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH

In Arm GPU:

# Arm not have 1.8.1 version package
wget https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/onnxruntime-linux-aarch64-1.10.0.tgz

tar -zxvf onnxruntime-linux-aarch64-1.10.0.tgz
cd onnxruntime-linux-aarch64-1.10.0
export ONNXRUNTIME_DIR=$(pwd)
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH

You can also go to ONNX Runtime Release to find corresponding release version package.

Build on Linux¶

CPU Version

cd ${MMDEPLOY_DIR} # To MMDeploy root directory
mkdir -p build && cd build
cmake -DMMDEPLOY_TARGET_DEVICES='cpu' -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
make -j$(nproc) && make install

GPU Version

cd ${MMDEPLOY_DIR} # To MMDeploy root directory
mkdir -p build && cd build
cmake -DMMDEPLOY_TARGET_DEVICES='cuda' -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
make -j$(nproc) && make install

How to convert a model¶

You could follow the instructions of tutorial How to convert model

How to add a new custom op¶

Reminder¶

The custom operator is not included in supported operator list in ONNX Runtime.
The custom operator should be able to be exported to ONNX.

Main procedures¶

Take custom operator roi_align for example.

Create a roi_align directory in ONNX Runtime source directory ${MMDEPLOY_DIR}/csrc/backend_ops/onnxruntime/
Add header and source file into roi_align directory ${MMDEPLOY_DIR}/csrc/backend_ops/onnxruntime/roi_align/
Add unit test into tests/test_ops/test_ops.py Check here for examples.

Finally, welcome to send us PR of adding custom operators for ONNX Runtime in MMDeploy. :nerd_face: