This tutorial/article is a detailed version of the steps in '.Add new ML models to VVASThis section details the different possibilities of running ML models using VVAS.
Requirements:
- VVAS 2.0
- Vitis AI 2.5
- Petalinux 2022.2 KV260 Starter Kit bsp: xilinx-kv260-starterkit-v2022.2-10141622.bsp
If you are using VVAS 3.0, please read this tutorial carefully and focus on the points covered in How to Add a Model in VVAS 3.0.
overview
To add new models to VVAS, you need to modify/customize the VVAS sources and then build/create the VVAS sources through a cross compiler (SDK). In this tutorial, we will start with a Petalinux project, create the Petalinux SDK (for cross-compilation on the host PC/machine), modify the VVAS sources (from github.com/xilinx/vvas/tree/2.0), and then create the SDK Compile/create the source with . environment.
Creating a Petalinux SDK using the Vitis AI package- Create a Petalinux project from the base BSP you downloaded for the KV260 Starter Kit
$ petalinux-create -t project -s xilinx-kv260-starterkit-v2022.2-10141622.bsp -n kv260_2022_2_vvas_bringup- Vitis AI recipes come from the following sources: https://github.com/Xilinx/Vitis-AI/tree/2.5/setup/petalinux/recipes-vitis-ai and copy
recipes-vitis-aiin<petalinux_project>/project-spec/meta-userfolder - Add the following line to your Petalinux project to enable the Vitis AI automation package group.
<petalinux_project>/project-spec/meta-user/confFile:
IMAGE_INSTALL:append = " vitis-ai-library "
IMAGE_INSTALL:append = " vitis-ai-library-dev "- Build the petalinux project and SDK using the following command:
petalinux-build -sThis results in sdk.sh file in <petalinux_project>/images/linux folder
- run the above
sdk.shInstall the SDK on the host machine for cross-compiling VVAS source code on the host machine.
./sdk.sh- Get the VVAS source code from the following github repository.https://github.com/Xilinx/VVAS
- Modify your git branch as follows:
vvas-rel-v2.0
git checkout vvas-rel-v2.0This section details the different possibilities of running ML models using VVAS.
If your model corresponds to one of the classes already supported by the vvas_xinfer plugin, no changes are required other than specifying the correct class name in the infer json file used by vvas_xinfer.
If the class corresponding to your new model is not supported by vvas_xinfer but is supported by the Vitis AI library, follow these steps to add the new model class to vvas_xinfer.
Add the model class type to the file vvas_xdpumodels.hpp: vvas-accel-sw-libs/vvas_xdpuinfer/src/vvas_xdpumodels.hpp
If I only change the above files, the build will fail with the following error:
The following files need to be updated along with the above files:VVAS/vvas-gst-plugins/gst-libs/gst/vvas/gstvvasinpinfer.h
update vvas_dpuinfer.cpp Use the model class header file. You can refer to other class implementations to see what changes are required.
It also updates model variables based on the new model.
update meson.build New class and corresponding Vitis AI library name.
Changing only the above will throw an option not found error. So you need to add the following option:VVAS/vvas-accel-sw-libs/meson_options.txt
Add a new file corresponding to your new class implementation. VVAS/vvas-accel-sw-libs/vvas_xdpuinfer/src. – Can reference implementations of other classes.
Update meson.build
If the new model results are not supported by existing fields in the VVAS inference metadata structure, this may also need to be changed to support the new metadata. If you need to make changes, https://github.com/Xilinx/VVAS/blob/vvas-rel-v2.0/vvas-gst-plugins/gst-libs/gst/vvas/gstinferenceprediction.h
If your model is not supported by the classes supported by the Vitis AI library, or if your model is not in a DPU (Deep Learning Processing Unit) deployable format, you must first convert it to a DPU deployable state. For more information about this, see Deploying NN Models with Vitis AI. For user guides, see the Vitis AI 3.0 documentation.
Once xmodel is available for user models, see the Rawtensor example for more information on how to use this model with VVAS.
Then you can check if the added model is loaded in the shared library (libvvas_xdpuinfer.so) Check if this is not the case by decompiling and running the following commands on your edge device (here the KV260 board):
Here we take an example of YoloV3 implementationThe vvas_xdpuinfer library has the following files for each ML model:
vvas_xyolov3.hpp
- declaration of
vvas_xyolov3
#pragma once
#include "vvas_xdpupriv.hpp"#include <vitis/ai/yolov3.hpp>
#include <vitis/ai/nnpp/yolov3.hpp>
using namespace std;
using namespace cv;
class vvas_xyolov3:public vvas_xdpumodel
{
int log_level = 0;
std::unique_ptr < vitis::ai::YOLOv3 > model; //calling vitis ai YoloV3 model class
public:
vvas_xyolov3 (vvas_xkpriv * kpriv, const std::string & model_name,
bool need_preprocess); // constructor
virtual int run (vvas_xkpriv * kpriv, std::vector<cv::Mat>& images,
GstInferencePrediction **predictions); //model run method
virtual int requiredwidth (void);
virtual int requiredheight (void);
virtual int supportedbatchsz (void);
virtual int close (void);
virtual ~vvas_xyolov3 ();
};
vvas_xyolov3.cpp
Constructor definition:
#include "vvas_xyolov3.hpp"
#include <algorithm>vvas_xyolov3::vvas_xyolov3 (vvas_xkpriv * kpriv, const std::string & model_name,
bool need_preprocess)
{
log_level = kpriv->log_level;
kpriv->labelflags = VVAS_XLABEL_REQUIRED;
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "enter");
if (kpriv->labelptr == NULL) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level, "label not found");
kpriv->labelflags |= VVAS_XLABEL_NOT_FOUND;
} else
kpriv->labelflags |= VVAS_XLABEL_FOUND;
model = vitis::ai::YOLOv3::create (model_name, need_preprocess);
}
Here we will create a model using the Vitis AI Yolov3 create method.
Implementing the vvas_xyolov3::run methodThe model is now run and the results are saved. GstInferencePredidction **prediction
int
vvas_xyolov3::run (vvas_xkpriv * kpriv, std::vector<cv::Mat>& images,
GstInferencePrediction **predictions)
{
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "enter batch");
auto results = model->run (images); labels *lptr;
char *pstr; /* prediction string */
if (kpriv->labelptr == NULL) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level, "label not found");
return false;
}
if (kpriv->objs_detection_max > 0) {
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "sort detected objects based on bbox area");
/* sort objects based on dimension to pick objects with bigger bbox */
for (unsigned int i = 0u; i < results.size(); i++) {
std::sort(results[i].bboxes.begin(), results[i].bboxes.end(), compare_by_area);
}
} else {
LOG_MESSAGE (LOG_LEVEL_WARNING, kpriv->log_level, "max-objects count is zero. So, not doing any metadata processing");
return true;
}
for (auto i = 0u; i < results.size(); i++) {
GstInferencePrediction *parent_predict = NULL;
unsigned int cur_objs = 0;
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level, "objects detected %lu",
results[i].bboxes.size());
if (results[i].bboxes.size()) {
BoundingBox parent_bbox;
int cols = images[i].cols;
int rows = images[i].rows;
parent_predict = predictions[i];
for (auto & box:results[i].bboxes) {
lptr = kpriv->labelptr + box.label;
if (kpriv->filter_labels.size()) {
bool found_label = false;
for (unsigned int n = 0; n < kpriv->filter_labels.size(); n++) {
const char *filter_label = kpriv->filter_labels[n].c_str();
const char *current_label = lptr->display_name.c_str();
if (!strncmp (current_label, filter_label, strlen (filter_label))) {
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "current label %s is in filter_label list", current_label);
found_label = true;
}
}
if (!found_label)
continue;
}
if (!parent_predict) {
parent_bbox.x = parent_bbox.y = 0;
parent_bbox.width = cols;
parent_bbox.height = rows;
parent_predict = gst_inference_prediction_new_full (&parent_bbox);
}
int label = box.label;
float xmin = box.x * cols + 1;
float ymin = box.y * rows + 1;
float xmax = xmin + box.width * cols;
float ymax = ymin + box.height * rows;
if (xmin < 0.)
xmin = 1.;
if (ymin < 0.)
ymin = 1.;
if (xmax > cols)
xmax = cols;
if (ymax > rows)
ymax = rows;
float confidence = box.score;
BoundingBox bbox;
GstInferencePrediction *predict;
GstInferenceClassification *c = NULL;
bbox.x = xmin;
bbox.y = ymin;
bbox.width = xmax - xmin;
bbox.height = ymax - ymin;
predict = gst_inference_prediction_new_full (&bbox);
c = gst_inference_classification_new_full (label, confidence,
lptr->display_name.c_str (), 0, NULL, NULL, NULL);
gst_inference_prediction_append_classification (predict, c);
if (parent_predict->predictions == NULL)
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level, "parent_predict->predictions is NULL");
gst_inference_prediction_append (parent_predict, predict);
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"RESULT: %s(%d) %f %f %f %f (%f)", lptr->display_name.c_str (), label,
xmin, ymin, xmax, ymax, confidence);
cur_objs++;
if (cur_objs == kpriv->objs_detection_max) {
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "reached max limit of objects to add to metadata");
break;
}
}
if (parent_predict) {
pstr = gst_inference_prediction_to_string (parent_predict);
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "prediction tree : \n%s",
pstr);
free(pstr);
}
}
predictions[i] = parent_predict;
}
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level, " ");
return true;
}
For more information about data structures for storing prediction results, see the following link:
- https://developer.ridgerun.com/wiki/index.php/GstInference/Metadatas/GstInferenceMeta
- https://xilinx.github.io/VVAS/1.1/build/html/docs/common/A-VVAS-Inference-Metadata.html
- Step 1: Source sysroot path (if not already done)
source <sysroot path>/environment-setup-aarch64-xilinx-linux./build_install_vvas.sh Edgevvas_installer.tar.gz /installer
- Step 3: Copy the VVAS installer to the embedded board
scp install/vvas_installer.tar.gz <board ip>:/- Step 4: Install VVAS on the embedded board
cd /
tar -xvf vvas_installer.tar.gzUpdated model json.
Test pipeline:
sudo gst-launch-1.0 filesrc location="walking_nv12.yuv" ! rawvideoparse format=nv12 width=1920 height=1080 framerate=30/1 \
! "video/x-raw, width=1920, height=1080, format=NV12" \
! vvas_xmultisrc kconfig="/opt/xilinx/kv260-smartcam/share/vvas/vvas-test1/preprocess.json" ! queue ! vvas_xfilter kernels-config="/opt/xilinx/kv260-smartcam/share/vvas/vvas-test1/aiinference.json" ! perf ! fakesinkBelow is the log of the pipeline where the Yolov4 Tiny model class was updated.
What's next?In the above tutorial, the predictions are displayed in the console as a result of a post-processing step.
References:So next, consider adding post-processing to the newly added model that saves the model output to a prediction tree data structure.
- The above tutorial is based on the Xilinx tutorial
Adding new model in VVAShttps://xilinx.github.io/VVAS/main/build/html/docs/common/adding_new_model.html This is for VVAS 3.0, but with some modifications it can be used to add models in VVAS 2.0.
- Compared to VVAS 2.0, VVAS 3.0 moves the core libraries into one repository.
vvas-coreThe repository is available at: https://github.com/Xilinx/vvas-core/tree/61935434021f7a7ee16e26d22ffa1b96a0e4895c - update
vvas-core/common/vvas_core /vvas_dpucommon.hUse the new VVAS model class typedef instead.VVAS/vvas-gst-plugins/gst-libs/gst/vvas/gstvvasinpinfer.h - Add the VVAS new model source files to the following location:
***
See you in the next tutorial!
Kudos to Sanam@LogicTronix for creating this detailed tutorial.
If you have any questions, please contact us at info@logictronix.com.
