Echo/fcb_photo_review
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

更换文档检测模型

Browse Source
This commit is contained in:
liuyebo
2024-08-27 14:42:45 +08:00
parent aea6f19951
commit 1514e09c40
2072 changed files with 254336 additions and 4967 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
paddle_detection/deploy/pipeline/docs/images/ppvehicleplate.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 578 KiB

243
paddle_detection/deploy/pipeline/docs/tutorials/PPHuman_QUICK_STARTED.md Normal file
View File

@@ -0,0 +1,243 @@
[English](PPHuman_QUICK_STARTED_en.md) | 简体中文
# PP-Human快速开始
## 目录
- [环境准备](#环境准备)
- [模型下载](#模型下载)
- [配置文件说明](#配置文件说明)
- [预测部署](#预测部署)
- [在线视频流](#在线视频流)
- [Jetson部署说明](#Jetson部署说明)
- [参数说明](#参数说明)
- [方案介绍](#方案介绍)
- [行人检测](#行人检测)
- [行人跟踪](#行人跟踪)
- [跨镜行人跟踪](#跨镜行人跟踪)
- [属性识别](#属性识别)
- [行为识别](#行为识别)
## 环境准备
环境要求： PaddleDetection版本 >= release/2.4 或 develop版本
PaddlePaddle和PaddleDetection安装
```
# PaddlePaddle CUDA10.1
python -m pip install paddlepaddle-gpu==2.2.2.post101 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html
# PaddlePaddle CPU
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
# 克隆PaddleDetection仓库
cd <path/to/clone/PaddleDetection>
git clone https://github.com/PaddlePaddle/PaddleDetection.git
# 安装其他依赖
cd PaddleDetection
pip install -r requirements.txt
```
1. 详细安装文档参考[文档](../../../../docs/tutorials/INSTALL_cn.md)
2. 如果需要TensorRT推理加速（测速方式），请安装带`TensorRT版本Paddle`。您可以从[Paddle安装包](https://paddleinference.paddlepaddle.org.cn/v2.2/user_guides/download_lib.html#python)下载安装，或者按照[指导文档](https://www.paddlepaddle.org.cn/inference/master/optimize/paddle_trt.html)使用docker或自编译方式准备Paddle环境。
## 模型下载
PP-Human提供了目标检测、属性识别、行为识别、ReID预训练模型，以实现不同使用场景，用户可以直接下载使用
| 任务 | 端到端速度（ms）| 模型方案 | 模型体积 |
| :---------: | :-------: | :------: |:------: |
| 行人检测（高精度） | 25.1ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) | 182M |
| 行人检测（轻量级） | 16.2ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_pipeline.zip) | 27M |
| 行人检测（超轻量级） | 10ms(Jetson AGX) | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/pphuman/ppyoloe_plus_crn_t_auxhead_320_60e_pphuman.tar.gz) | 17M |
| 行人跟踪（高精度） | 31.8ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) | 182M |
| 行人跟踪（轻量级） | 21.0ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_pipeline.zip) | 27M |
| 行人跟踪（超轻量级） | 13.2ms(Jetson AGX) | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/pphuman/ppyoloe_plus_crn_t_auxhead_320_60e_pphuman.tar.gz) | 17M |
| 跨镜跟踪(REID) | 单人1.5ms | [REID](https://bj.bcebos.com/v1/paddledet/models/pipeline/reid_model.zip) | REID：92M |
| 属性识别（高精度） | 单人8.5ms | [目标检测](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)<br> [属性识别](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_small_person_attribute_954_infer.zip) | 目标检测：182M<br>属性识别：86M |
| 属性识别（轻量级） | 单人7.1ms | [目标检测](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)<br> [属性识别](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPLCNet_x1_0_person_attribute_945_infer.zip) | 目标检测：182M<br>属性识别：86M |
| 摔倒识别 | 单人10ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) <br> [关键点检测](https://bj.bcebos.com/v1/paddledet/models/pipeline/dark_hrnet_w32_256x192.zip) <br> [基于关键点行为识别](https://bj.bcebos.com/v1/paddledet/models/pipeline/STGCN.zip) | 多目标跟踪：182M<br>关键点检测：101M<br>基于关键点行为识别：21.8M |
| 闯入识别 | 31.8ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) | 多目标跟踪：182M |
| 打架识别 | 19.7ms | [视频分类](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) | 90M |
| 抽烟识别 | 单人15.1ms | [目标检测](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)<br>[基于人体id的目标检测](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppyoloe_crn_s_80e_smoking_visdrone.zip) | 目标检测：182M<br>基于人体id的目标检测：27M |
| 打电话识别 | 单人6.0ms | [目标检测](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)<br>[基于人体id的图像分类](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_tiny_calling_halfbody.zip) | 目标检测：182M<br>基于人体id的图像分类：45M |
下载模型后，解压至`./output_inference`文件夹。
在配置文件中，模型路径默认为模型的下载路径，如果用户不修改，则在推理时会自动下载对应的模型。
**注意：**
- 模型精度为融合数据集结果，数据集包含开源数据集和企业数据集
- ReID模型精度为Market1501数据集测试结果
- 预测速度为T4下，开启TensorRT FP16的效果, 模型预测速度包含数据预处理、模型预测、后处理全流程
## 配置文件说明
PP-Human相关配置位于```deploy/pipeline/config/infer_cfg_pphuman.yml```中，存放模型路径，该配置文件中包含了目前PP-Human支持的所有功能。如果想要查看某个单一功能的配置，请参见```deploy/pipeline/config/examples/```中相关配置。此外，配置文件中的内容可以通过```-o```命令行参数修改，如修改属性的模型目录，则可通过```-o ATTR.model_dir="DIR_PATH"```进行设置。
功能及任务类型对应表单如下：
| 输入类型 | 功能 | 任务类型 | 配置项 |
|-------|-------|----------|-----|
| 图片 | 属性识别 | 目标检测 属性识别 | DET ATTR |
| 单镜头视频 | 属性识别 | 多目标跟踪 属性识别 | MOT ATTR |
| 单镜头视频 | 行为识别 | 多目标跟踪 关键点检测 摔倒识别 | MOT KPT SKELETON_ACTION |
例如基于视频输入的属性识别，任务类型包含多目标跟踪和属性识别，具体配置如下：
```
crop_thresh: 0.5
attr_thresh: 0.5
visual: True
MOT:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1
enable: True
ATTR:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/PPLCNet_x1_0_person_attribute_945_infer.zip
batch_size: 8
enable: True
```
**注意：**
- 如果用户需要实现不同任务，可以在配置文件对应enable选项设置为True。
## 预测部署
1. 直接使用默认配置或者examples中配置文件，或者直接在`infer_cfg_pphuman.yml`中修改配置：
```
# 例：行人检测，指定配置文件路径和测试图片，图片输入默认打开检测模型
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml --image_file=test_image.jpg --device=gpu
# 例：行人属性识别，直接使用examples中配置
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml --video_file=test_video.mp4 --device=gpu
```
2. 使用命令行进行功能开启，或者模型路径修改：
```
# 例：行人跟踪，指定配置文件路径，模型路径和测试视频, 命令行中指定的模型路径优先级高于配置文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml -o MOT.enable=True MOT.model_dir=ppyoloe_infer/ --video_file=test_video.mp4 --device=gpu
# 例：行为识别，以摔倒识别为例，命令行中开启SKELETON_ACTION模型
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml -o SKELETON_ACTION.enbale=True --video_file=test_video.mp4 --device=gpu
```
### 在线视频流
在线视频流解码功能基于opencv的capture函数，支持rtsp、rtmp格式。
- rtsp拉流预测
对rtsp拉流的支持，使用--rtsp RTSP [RTSP ...]参数指定一路或者多路rtsp视频流，如果是多路地址中间用空格隔开。(或者video_file后面的视频地址直接更换为rtsp流地址)，示例如下：
```
# 例：行人属性识别，单路视频流
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE] --device=gpu
# 例：行人属性识别，多路视频流
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE1] rtsp://[YOUR_RTSP_SITE2] --device=gpu
```
- 视频结果推流rtsp
预测结果进行rtsp推流，使用--pushurl rtsp:[IP] 推流到IP地址端，PC端可以使用[VLC播放器](https://vlc.onl/)打开网络流进行播放，播放地址为 `rtsp:[IP]/videoname`。其中`videoname`是预测的视频文件名，如果视频来源是本地摄像头则`videoname`默认为`output`.
```
# 例：行人属性识别，单路视频流，该示例播放地址为 rtsp://[YOUR_SERVER_IP]:8554/test_video
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml --video_file=test_video.mp4 --device=gpu --pushurl rtsp://[YOUR_SERVER_IP]:8554
```
注：
1. rtsp推流服务基于 [rtsp-simple-server](https://github.com/aler9/rtsp-simple-server), 如使用推流功能请先开启该服务.
使用方法很简单，以linux平台为例：1）下载对应平台release包；2）解压后在命令行执行命令 `./rtsp-simple-server`即可，成功后进入服务开启状态就可以接收视频流了。
2. rtsp推流如果模型处理速度跟不上会出现很明显的卡顿现象，建议跟踪模型使用ppyoloe_s或ppyoloe-plus-tiny版本，方式为修改配置中跟踪模型mot_ppyoloe_l_36e_pipeline.zip替换为mot_ppyoloe_s_36e_pipeline.zip。
### Jetson部署说明
由于Jetson平台算力相比服务器有较大差距，有如下使用建议：
1. 模型选择轻量级版本，我们最新提供了轻量级[PP-YOLOE-Plus Tiny模型](../../../../configs/pphuman/README.md)，该模型在Jetson AGX上可以实现4路视频流20fps实时跟踪。
2. 如果需进一步提升速度，建议开启跟踪跳帧功能，推荐使用2或者3: `skip_frame_num: 3`，该功能当前默认关闭。
上述修改可以直接修改配置文件（推荐），也可以在命令行中修改（字段较长，不推荐）。
PP-YOLOE-Plus Tiny模型在AGX平台不同功能开启时的速度如下：（跟踪人数为3人情况下，以属性为例，总耗时为跟踪13.3+5.2*3≈29ms）
| 功能 | 平均每帧耗时(ms) | 运行帧率(fps) |
|:----------|:----------|:----------|
| 跟踪 | 13 | 77 |
| 属性识别 | 29 | 34 |
| 摔倒识别 | 64.5 | 15.5 |
| 抽烟识别 | 68.8 | 14.5 |
| 打电话识别 | 22.5 | 44.5 |
| 打架识别 | 3.98 | 251 |
### 参数说明
| 参数 | 是否必须|含义 |
|-------|-------|----------|
| --config | Yes | 配置文件路径 |
| -o | Option | 覆盖配置文件中对应的配置 |
| --image_file | Option | 需要预测的图片 |
| --image_dir | Option | 要预测的图片文件夹路径 |
| --video_file | Option | 需要预测的视频，或者rtsp流地址（推荐使用rtsp参数） |
| --rtsp | Option | rtsp视频流地址，支持一路或者多路同时输入 |
| --camera_id | Option | 用来预测的摄像头ID，默认为-1(表示不使用摄像头预测，可设置为：0 - (摄像头数目-1) )，预测过程中在可视化界面按`q`退出输出预测结果到：output/output.mp4|
| --device | Option | 运行时的设备，可选择`CPU/GPU/XPU`，默认为`CPU`|
| --pushurl | Option| 对预测结果视频进行推流的地址，以rtsp://开头，该选项优先级高于视频结果本地存储，打开时不再另外存储本地预测结果视频|
| --output_dir | Option|可视化结果保存的根目录，默认为output/|
| --run_mode | Option |使用GPU时，默认为paddle, 可选（paddle/trt_fp32/trt_fp16/trt_int8）|
| --enable_mkldnn | Option | CPU预测中是否开启MKLDNN加速，默认为False |
| --cpu_threads | Option| 设置cpu线程数，默认为1 |
| --trt_calib_mode | Option| TensorRT是否使用校准功能，默认为False。使用TensorRT的int8功能时，需设置为True，使用PaddleSlim量化后的模型时需要设置为False |
| --do_entrance_counting | Option | 是否统计出入口流量，默认为False |
| --draw_center_traj | Option | 是否绘制跟踪轨迹，默认为False |
| --region_type | Option | 'horizontal'（默认值）、'vertical'：表示流量统计方向选择；'custom'：表示设置闯入区域 |
| --region_polygon | Option | 设置闯入区域多边形多点的坐标，无默认值 |
| --do_break_in_counting | Option | 此项表示做区域闯入检查 |
## 方案介绍
PP-Human v2整体方案如下图所示:
<div width="1000" align="center">
<img src="../../../../docs/images/pphumanv2.png"/>
</div>
### 行人检测
- 采用PP-YOLOE L 作为目标检测模型
- 详细文档参考[PP-YOLOE](../../../../configs/ppyoloe/)和[检测跟踪文档](pphuman_mot.md)
### 行人跟踪
- 采用SDE方案完成行人跟踪
- 检测模型使用PP-YOLOE L(高精度)和S(轻量级)
- 跟踪模块采用OC-SORT方案
- 详细文档参考[OC-SORT](../../../../configs/mot/ocsort)和[检测跟踪文档](pphuman_mot.md)
### 跨镜行人跟踪
- 使用PP-YOLOE + OC-SORT得到单镜头多目标跟踪轨迹
- 使用ReID（StrongBaseline网络）对每一帧的检测结果提取特征
- 多镜头轨迹特征进行匹配，得到跨镜头跟踪结果
- 详细文档参考[跨镜跟踪](pphuman_mtmct.md)
### 属性识别
- 使用PP-YOLOE + OC-SORT跟踪人体
- 使用PP-HGNet、PP-LCNet（多分类模型）完成识别属性，主要属性包括年龄、性别、帽子、眼睛、上衣下衣款式、背包等
- 详细文档参考[属性识别](pphuman_attribute.md)
### 行为识别：
- 提供四种行为识别方案
- 1. 基于骨骼点的行为识别，例如摔倒识别
- 2. 基于图像分类的行为识别，例如打电话识别
- 3. 基于检测的行为识别，例如吸烟识别
- 4. 基于视频分类的行为识别，例如打架识别
- 详细文档参考[行为识别](pphuman_action.md)

247
paddle_detection/deploy/pipeline/docs/tutorials/PPHuman_QUICK_STARTED_en.md Normal file
View File

@@ -0,0 +1,247 @@
English | [简体中文](PPHuman_QUICK_STARTED.md)
# Quick Start for PP-Human
## Contents
- [Environment Preparation](#Environment-Preparation)
- [Model Download](#Model-Download)
- [Configuration](#Configuration)
- [Inference Deployment](#Inference-Deployment)
- [rtsp_stream](#rtsp_stream)
- [Nvidia_Jetson](#Nvidia_Jetson)
- [Parameters](#Parameters)
- [Solutions](#Solutions)
- [Pedestrian Detection](#edestrian-Detection)
- [Pedestrian Tracking](#Pedestrian-Tracking)
- [Multi-camera & multi-pedestrain tracking](#Multi-camera-&-multi-pedestrain-tracking)
- [Attribute Recognition](#Attribute-Recognition)
- [Behavior Recognition](#Behavior-Recognition)
## Environment Preparation
Environment requirements： PaddleDetection>= release/2.4 or develop version
Installation of PaddlePaddle and PaddleDetection
```
# PaddlePaddle CUDA10.1
python -m pip install paddlepaddle-gpu==2.2.2.post101 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html
# PaddlePaddle CPU
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
#Clone PaddleDetection repositories
cd <path/to/clone/PaddleDetection>
git clone https://github.com/PaddlePaddle/PaddleDetection.git
# Install dependencies
cd PaddleDetection
pip install -r requirements.txt
```
1. For installation details, please refer to [Installation Tutorials](../../../../docs/tutorials/INSTALL.md)
2. If you need TensorRT inference acceleration (speed measurement), please install PaddlePaddle with `TensorRT version`. You can download and install it from the [PaddlePaddle Installation Package](https://paddleinference.paddlepaddle.org.cn/v2.2/user_guides/download_lib.html#python) or follow the [Instructions](https://www. paddlepaddle.org.cn/inference/master/optimize/paddle_trt.html) or use docker, or self-compiling to prepare the environment.
## Model Download
PP-Human provides object detection, attribute recognition, behaviour recognition and ReID pre-trained models for different applications. Developers can download them directly.
| Task | End-to（ms） | Model Solution | Model Size |
|:--------------------------------------:|:--------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------:|
| Pedestrian Detection (high precision) | 25.1ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) | 182M |
| Pedestrian Detection (Lightweight) | 16.2ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_pipeline.zip) | 27M |
| Pedestrian detection (super lightweight) | 10ms(Jetson AGX) | [Multi-object tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/pphuman/ppyoloe_plus_crn_t_auxhead_320_60e_pphuman.tar.gz) | 17M |
| Pedestrian Tracking (high precision) | 31.8ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) | 182M |
| Pedestrian Tracking (Lightweight) | 21.0ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_pipeline.zip) | 27M |
| Pedestrian tracking（super lightweight） | 13.2ms(Jetson AGX) | [Multi-object tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/pphuman/ppyoloe_plus_crn_t_auxhead_320_60e_pphuman.tar.gz) | 17M |
| MTMCT(REID) | Single Person 1.5ms | [REID](https://bj.bcebos.com/v1/paddledet/models/pipeline/reid_model.zip) | REID：92M |
| Attribute Recognition (high precision) | Single Person 8.5ms | [Object Detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)<br> [Attribute Recognition](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_small_person_attribute_954_infer.zip) | Object Detection：182M<br>Attribute Recogniton：86M |
| Attribute Recognition (Lightweight) | Single Person 7.1ms | [Object Detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)<br> [Attribute Recogniton](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPLCNet_x1_0_person_attribute_945_infer.zip) | Object Detection：182M<br>Attribute Recogniton：86M |
| Falling Detection | Single Person 10ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) <br> [Keypoint Detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/dark_hrnet_w32_256x192.zip) <br> [Skeleton Based Action Recognition](https://bj.bcebos.com/v1/paddledet/models/pipeline/STGCN.zip) | Multi-Object Tracking：182M<br>Keypoint Detection：101M<br>Skeleton Based Action Recognition：21.8M |
| Breaking-In Detection | 31.8ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) | Multi-Object Tracking：182M |
| Fighting Detection | 19.7ms | [Video Classification](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) | 90M |
| Smoking Detection | Single Person 15.1ms | [Object Detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)<br>[Object Detection Based On Body ID](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppyoloe_crn_s_80e_smoking_visdrone.zip) | Object Detection：182M<br>Object Detection Based On Body ID：27M |
| Phone-calling Detection | Single Person 6.0ms | [Object Detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)<br>[Image Classification Based On Body ID](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_tiny_calling_halfbody.zip) | Object Detection：182M<br>Image Classification Based On Body ID：45M |
Download the model and unzip it into the `. /output_inference` folder.
In the configuration file, the model path defaults to the download path of the model. If the user does not change it, the corresponding model will be downloaded automatically upon inference.
**Note:**
- Model accuracy is tested on fused datasets, which contain both open source and enterprise datasets.
- ReID model accuracy is tested on the Market1501 dataset
- Prediction speed is obtained at T4 with TensorRT FP16 enabled, which includes data pre-processing, model inference and post-processing.
## Configuration
The PP-Human-related configuration is located in ``deploy/pipeline/config/infer_cfg_pphuman.yml``, and this configuration file contains all the features currently supported by PP-Human. If you want to see the configuration for a specific feature, please refer to the relevant configuration in ``deploy/pipeline/config/examples/``. In addition, the contents of the configuration file can be modified with the `-o`command line parameter. E.g. to modify the model directory of an attribute, developers can run ```-o ATTR.model_dir="DIR_PATH"``.
The features and corresponding task types are as follows.
| Input | Feature | Task | Config |
| ------------------- | --------------------- | ---------------------------------------------------------- | ----------------------- |
| Image | Attribute Recognition | Object Detection Attribute Recognition | DET ATTR |
| Single-camera video | Attribute Recognition | Multi-Object Tracking Attribute Recognition | MOT ATTR |
| Single-camera video | Behaviour Recognition | Multi-Object Tracking Keypoint Detection Falling detection | MOT KPT SKELETON_ACTION |
Take attribute recognition based on video input as an example: Its task type includes multi-object tracking and attributes recognition. The specific configuration is as follows.
```
crop_thresh: 0.5
attr_thresh: 0.5
visual: True
MOT:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1
enable: True
ATTR:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/PPLCNet_x1_0_person_attribute_945_infer.zip
batch_size: 8
enable: True
```
**Note:**
- If developer needs to carry out different tasks, set the corresponding enables option to be True in the configuration file.
## Inference Deployment
1. Use the default configuration directly or the configuration file in examples, or modify the configuration in `infer_cfg_pphuman.yml`
```
# Example: In pedestrian detection model, specify configuration file path and test image, and image input opens detection model by default
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml --image_file=test_image.jpg --device=gpu
# Example: In pedestrian attribute recognition, directly configure the examples
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml --video_file=test_video.mp4 --device=gpu
```
2. Use the command line to enable functions or change the model path.
```
# Example: Pedestrian tracking, specify config file path, model path and test video. The specified model path on the command line has a higher priority than the config file.
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml -o MOT.enable=True MOT.model_dir=ppyoloe_infer/ --video_file=test_video.mp4 --device=gpu
# Example: In behaviour recognition, with fall recognition as an example, enable the SKELETON_ACTION model on the command line
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml -o SKELETON_ACTION.enbale=True --video_file=test_video.mp4 --device=gpu
```
### rtsp_stream
The online stream decode based on opencv Capture function, normally support rtsp and rtmp.
- rtsp pull stream
For rtsp pull stream, use `--rtsp RTSP [RTSP ...]` parameter to specify one or more rtsp streams. Separate the multiple addresses with a space, or replace the video address directly after the video_file with the rtsp stream address), examples as follows
```
# Example: Single video stream for pedestrian attribute recognition
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE] --device=gpu
# Example: Multiple-video stream for pedestrian attribute recognition
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE1] rtsp://[YOUR_RTSP_SITE2] --device=gpu |
```
- rtsp push stream
For rtsp push stream, use `--pushurl rtsp:[IP]` parameter to push stream to a IP set, and you can visualize the output video by [VLC Player](https://vlc.onl/) with the `open network` funciton. the whole url path is `rtsp:[IP]/videoname`, the videoname here is the basename of the video file to infer, and the default of videoname is `output` when the video come from local camera and there is no video name.
```
# Example：Pedestrian attribute recognition，in this example the whole url path is: rtsp://[YOUR_SERVER_IP]:8554/test_video
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml --video_file=test_video.mp4 --device=gpu --pushurl rtsp://[YOUR_SERVER_IP]:8554
```
Note:
1. rtsp push stream is based on [rtsp-simple-server](https://github.com/aler9/rtsp-simple-server), please enable this serving first.
It's very easy to use: 1) download the [release package](https://github.com/aler9/rtsp-simple-server/releases) which is compatible with your workspace. 2) run command './rtsp-simple-server', which works as a rtsp server.
2. the output visualize will be frozen frequently if the model cost too much time, we suggest to use faster model like ppyoloe_s or ppyoloe_plus_tiny in tracking, this is simply replace mot_ppyoloe_l_36e_pipeline.zip with mot_ppyoloe_s_36e_pipeline.zip in model config yaml file.
### Nvidia_Jetson
Due to the large gap in computing power of the Jetson platform compared to the server, we suggest:
1. choose a lightweight model, we provide a new model named [PP-YOLOE-Plus Tiny](../../../../configs/pphuman/README.md)，which achieve 20fps with four rtsp streams work togather on Jetson AGX.
2. For further speedup, you can set frame skipping of tracking; we recommend 2 or 3: `skip_frame_num: 3`
PP-YOLOE-Plus Tiny module speed test data on AGX：（three people in video, for example of attribute，the whole time cost per frame is 13.3+5.2*3≈29ms）
| module | time cost per frame(ms) | speed(fps) |
|:----------|:----------|:----------|
| tracking | 13 | 77 |
| Attribute | 29 | 34 |
| falldown | 64.5 | 15.5 |
| smoking | 68.8 | 14.5 |
| calling | 22.5 | 44.5 |
| fighting | 3.98 | 251 |
### Parameters
| Parameters | Necessity | Implications |
| ---------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| --config | Yes | Path to configuration file |
| -o | Option | Overwrite the corresponding configuration in the configuration file |
| --image_file | Option | Images to be predicted |
| --image_dir | Option | Path to the images folder to be predicted |
| --video_file | Option | Video to be predicted, or rtsp stream address (rtsp parameter recommended) |
| --rtsp | Option | rtsp video stream address, supports one or more simultaneous streams input |
| --camera_id | Option | The camera ID for prediction, default is -1 ( for no camera prediction, can be set to 0 - (number of cameras - 1) ), press `q` in the visualization interface during the prediction process to output the prediction result to: output/output.mp4 |
| --device | Option | Running device, options include `CPU/GPU/XPU`, and the default is `CPU`. |
| --pushurl | Option | push the output video to rtsp stream, normaly start with `rtsp://`; this has higher priority than local video save, while this is set, pipeline will not save local visualize video, the default is "", means this will not work now.|
| --output_dir | Option | The root directory for the visualization results, and the default is output/ |
| --run_mode | Option | For GPU, the default is paddle, with (paddle/trt_fp32/trt_fp16/trt_int8) as optional |
| --enable_mkldnn | Option | Whether to enable MKLDNN acceleration in CPU prediction, the default is False |
| --cpu_threads | Option | Set the number of cpu threads, and the default is 1 |
| --trt_calib_mode | Option | Whether TensorRT uses the calibration function, and the default is False; set to True when using TensorRT's int8 function and False when using the PaddleSlim quantized model |
| --do_entrance_counting | Option | Whether to count entrance/exit traffic flows, the default is False |
| --draw_center_traj | Option | Whether to map the trace, the default is False |
| --region_type | Option | 'horizontal' (default), 'vertical': traffic count direction; 'custom': set break-in area |
| --region_polygon | Option | Set the coordinates of the polygon multipoint in the break-in area. No default. |
| --do_break_in_counting | Option | Area break-in checks |
## Solutions
The overall solution for PP-Human v2 is shown in the graph below:
### Pedestrian detection
- Take PP-YOLOE L as the object detection model
- For detailed documentation, please refer to [PP-YOLOE](../../../../configs/ppyoloe/) and [Multiple-Object-Tracking](pphuman_mot_en.md)
### Pedestrian tracking
- Vehicle tracking by SDE solution
- Adopt PP-YOLOE L (high precision) and S (lightweight) for detection models
- Adopt the OC-SORT solution for racking module
- Refer to [OC-SORT](../../../../configs/mot/ocsort) and [Multi-Object Tracking](pphuman_mot_en.md) for details
### Multi-camera & multi-pedestrain tracking
- Use PP-YOLOE & OC-SORT to acquire single-camera multi-object tracking trajectory
- Extract features for each frame using ReID (StrongBaseline network).
- Match multi-camera trajectory features to obtain multi-camera tracking results.
- Refer to [Multi-camera & multi-pedestrain tracking](pphuman_mtmct_en.md) for details.
### Attribute Recognition
- Use PP-YOLOE + OC-SORT to track the human body.
- Use PP-HGNet, PP-LCNet (multi-classification model) to complete the attribute recognition. Main attributes include age, gender, hat, eyes, top and bottom dressing style, backpack.
- Refer to [attribute recognition](pphuman_attribute_en.md) for details.
### Behaviour Recognition:
- Four behaviour recognition solutions are provided:
- 1. Behaviour recognition based on skeletal points, e.g. falling recognition
- 2. Behaviour recognition based on image classification, e.g. phone call recognition
- 3. Behaviour recognition based on detection, e.g. smoking recognition
- 4. Behaviour recognition based on Video classification, e.g. fighting recognition
- For details, please refer to [Behaviour Recognition](pphuman_action_en.md)

244
paddle_detection/deploy/pipeline/docs/tutorials/PPVehicle_QUICK_STARTED.md Normal file
View File

@@ -0,0 +1,244 @@
[English](PPVehicle_QUICK_STARTED_en.md) | 简体中文
# PP-Vehicle快速开始
## 目录
- [环境准备](#环境准备)
- [模型下载](#模型下载)
- [配置文件说明](#配置文件说明)
- [预测部署](#预测部署)
- [在线视频流](#在线视频流)
- [Jetson部署说明](#Jetson部署说明)
- [参数说明](#参数说明)
- [方案介绍](#方案介绍)
- [车辆检测](#车辆检测)
- [车辆跟踪](#车辆跟踪)
- [车牌识别](#车牌识别)
- [属性识别](#属性识别)
- [违章停车识别](#违章停车识别)
## 环境准备
环境要求： PaddleDetection版本 >= release/2.5 或 develop版本
PaddlePaddle和PaddleDetection安装
```
# PaddlePaddle CUDA10.1
python -m pip install paddlepaddle-gpu==2.2.2.post101 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html
# PaddlePaddle CPU
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
# 克隆PaddleDetection仓库
cd <path/to/clone/PaddleDetection>
git clone https://github.com/PaddlePaddle/PaddleDetection.git
# 安装其他依赖
cd PaddleDetection
pip install -r requirements.txt
```
1. 详细安装文档参考[文档](../../../../docs/tutorials/INSTALL_cn.md)
2. 如果需要TensorRT推理加速（测速方式），请安装带`TensorRT版本Paddle`。您可以从[Paddle安装包](https://paddleinference.paddlepaddle.org.cn/v2.2/user_guides/download_lib.html#python)下载安装，或者按照[指导文档](https://www.paddlepaddle.org.cn/inference/master/optimize/paddle_trt.html)使用docker或自编译方式准备Paddle环境。
## 模型下载
PP-Vehicle提供了目标检测、属性识别、行为识别、ReID预训练模型，以实现不同使用场景，用户可以直接下载使用
| 任务 | 端到端速度（ms）| 模型方案 | 模型体积 |
| :---------: | :-------: | :------: |:------: |
| 车辆检测（高精度） | 25.7ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) | 182M |
| 车辆检测（轻量级） | 13.2ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_ppvehicle.zip) | 27M |
| 车辆检测（超轻量级） | 10ms（Jetson AGX） | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppvehicle/ppyoloe_plus_crn_t_auxhead_320_60e_ppvehicle.tar.gz) | 17M |
| 车辆跟踪（高精度） | 40ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) | 182M |
| 车辆跟踪（轻量级） | 25ms | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_ppvehicle.zip) | 27M |
| 车辆跟踪（超轻量级） | 13.2ms（Jetson AGX） | [多目标跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppvehicle/ppyoloe_plus_crn_t_auxhead_320_60e_ppvehicle.tar.gz) | 17M |
| 车牌识别 | 4.68ms | [车牌检测](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz) <br> [车牌字符识别](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz) | 车牌检测：3.9M <br> 车牌字符识别： 12M |
| 车辆属性 | 7.31ms | [车辆属性](https://bj.bcebos.com/v1/paddledet/models/pipeline/vehicle_attribute_model.zip) | 7.2M |
| 车道线检测 | 47ms | [车道线模型](https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip) | 47M |
下载模型后，解压至`./output_inference`文件夹。
在配置文件中，模型路径默认为模型的下载路径，如果用户不修改，则在推理时会自动下载对应的模型。
**注意：**
- 检测跟踪模型精度为公开数据集BDD100K-MOT和UA-DETRAC整合后的联合数据集PPVehicle的结果，具体参照[ppvehicle](../../../../configs/ppvehicle)
- 预测速度为T4下，开启TensorRT FP16的效果, 模型预测速度包含数据预处理、模型预测、后处理全流程
## 配置文件说明
PP-Vehicle相关配置位于```deploy/pipeline/config/infer_cfg_ppvehicle.yml```中，存放模型路径，完成不同功能需要设置不同的任务类型
功能及任务类型对应表单如下：
| 输入类型 | 功能 | 任务类型 | 配置项 |
|-------|-------|----------|-----|
| 图片 | 属性识别 | 目标检测 属性识别 | DET ATTR |
| 单镜头视频 | 属性识别 | 多目标跟踪 属性识别 | MOT ATTR |
| 单镜头视频 | 车牌识别 | 多目标跟踪 车牌识别 | MOT VEHICLEPLATE |
例如基于视频输入的属性识别，任务类型包含多目标跟踪和属性识别，具体配置如下：
```
crop_thresh: 0.5
visual: True
warmup_frame: 50
MOT:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1
enable: True
VEHICLE_ATTR:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/vehicle_attribute_model.zip
batch_size: 8
color_threshold: 0.5
type_threshold: 0.5
enable: True
```
**注意：**
- 如果用户需要实现不同任务，可以在配置文件对应enable选项设置为True。
- 如果用户仅需要修改模型文件路径，可以在命令行中--config后面紧跟着 `-o MOT.model_dir=ppyoloe/` 进行修改即可，也可以手动修改配置文件中的相应模型路径，详细说明参考下方参数说明文档。
## 预测部署
1. 直接使用默认配置或者examples中配置文件，或者直接在`infer_cfg_ppvehicle.yml`中修改配置：
```
# 例：车辆检测，指定配置文件路径和测试图片
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml --image_file=test_image.jpg --device=gpu
# 例：车辆车牌识别，指定配置文件路径和测试视频
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_plate.yml --video_file=test_video.mp4 --device=gpu
```
2. 使用命令行进行功能开启，或者模型路径修改：
```
# 例：车辆跟踪，指定配置文件路径和测试视频，命令行中开启MOT模型并修改模型路径，命令行中指定的模型路径优先级高于配置文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml -o MOT.enable=True MOT.model_dir=ppyoloe_infer/ --video_file=test_video.mp4 --device=gpu
# 例：车辆违章分析，指定配置文件和测试视频，命令行中指定违停区域设置、违停时间判断。
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml \
--video_file=../car_test.mov \
--device=gpu \
--draw_center_traj \
--illegal_parking_time=3 \
--region_type=custom \
--region_polygon 600 300 1300 300 1300 800 600 800
```
### 在线视频流
在线视频流解码功能基于opencv的capture函数，支持rtsp、rtmp格式。
- rtsp拉流预测
对rtsp拉流的支持，使用--rtsp RTSP [RTSP ...]参数指定一路或者多路rtsp视频流，如果是多路地址中间用空格隔开。(或者video_file后面的视频地址直接更换为rtsp流地址)，示例如下：
```
# 例：车辆属性识别，单路视频流
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE] --device=gpu
# 例：车辆属性识别，多路视频流
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE1] rtsp://[YOUR_RTSP_SITE2] --device=gpu
```
- 视频结果推流rtsp
预测结果进行rtsp推流，使用--pushurl rtsp:[IP] 推流到IP地址端，PC端可以使用[VLC播放器](https://vlc.onl/)打开网络流进行播放，播放地址为 `rtsp:[IP]/videoname`。其中`videoname`是预测的视频文件名，如果视频来源是本地摄像头则`videoname`默认为`output`.
```
# 例：车辆属性识别，单路视频流，该示例播放地址为 rtsp://[YOUR_SERVER_IP]:8554/test_video
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_attr.yml -o visual=False --video_file=test_video.mp4 --device=gpu --pushurl rtsp://[YOUR_SERVER_IP]:8554
```
注：
1. rtsp推流服务基于 [rtsp-simple-server](https://github.com/aler9/rtsp-simple-server), 如使用推流功能请先开启该服务.
使用方法很简单，以linux平台为例：1）下载对应平台release包；2）解压后在命令行执行命令 `./rtsp-simple-server`即可，成功后进入服务开启状态就可以接收视频流了。
2. rtsp推流如果模型处理速度跟不上会出现很明显的卡顿现象，建议跟踪模型使用ppyoloe_s版本，即修改配置中跟踪模型mot_ppyoloe_l_36e_pipeline.zip替换为mot_ppyoloe_s_36e_pipeline.zip。
### Jetson部署说明
由于Jetson平台算力相比服务器有较大差距，有如下使用建议：
1. 模型选择轻量级版本，我们最新提供了轻量级[PP-YOLOE-Plus Tiny模型](../../../../configs/ppvehicle/README.md)，该模型在Jetson AGX上可以实现4路视频流20fps实时跟踪。
2. 如果需进一步提升速度，建议开启跟踪跳帧功能，推荐使用2或者3: `skip_frame_num: 3`，该功能当前默认关闭。
上述修改可以直接修改配置文件（推荐），也可以在命令行中修改（字段较长，不推荐）。
PP-YOLOE-Plus Tiny模型在AGX平台不同功能开启时的速度如下：（测试视频跟踪车辆为1个）
| 功能 | 平均每帧耗时(ms) | 运行帧率(fps) |
|:----------|:----------|:----------|
| 跟踪 | 13 | 77 |
| 属性识别 | 20.2 | 49.4 |
| 车牌识别 | - | - |
### 参数说明
| 参数 | 是否必须|含义 |
|-------|-------|----------|
| --config | Yes | 配置文件路径 |
| -o | Option | 覆盖配置文件中对应的配置 |
| --image_file | Option | 需要预测的图片 |
| --image_dir | Option | 要预测的图片文件夹路径 |
| --video_file | Option | 需要预测的视频，或者rtsp流地址 |
| --rtsp | Option | rtsp视频流地址，支持一路或者多路同时输入 |
| --camera_id | Option | 用来预测的摄像头ID，默认为-1(表示不使用摄像头预测，可设置为：0 - (摄像头数目-1) )，预测过程中在可视化界面按`q`退出输出预测结果到：output/output.mp4|
| --device | Option | 运行时的设备，可选择`CPU/GPU/XPU`，默认为`CPU`|
| --pushurl | Option| 对预测结果视频进行推流的地址，以rtsp://开头，该选项优先级高于视频结果本地存储，打开时不再另外存储本地预测结果视频, 默认为空，表示没有开启|
| --output_dir | Option|可视化结果保存的根目录，默认为output/|
| --run_mode | Option |使用GPU时，默认为paddle, 可选（paddle/trt_fp32/trt_fp16/trt_int8）|
| --enable_mkldnn | Option | CPU预测中是否开启MKLDNN加速，默认为False |
| --cpu_threads | Option| 设置cpu线程数，默认为1 |
| --trt_calib_mode | Option| TensorRT是否使用校准功能，默认为False。使用TensorRT的int8功能时，需设置为True，使用PaddleSlim量化后的模型时需要设置为False |
| --do_entrance_counting | Option | 是否统计出入口流量，默认为False |
| --draw_center_traj | Option | 是否绘制跟踪轨迹，默认为False |
| --region_type | Option | 'horizontal'（默认值）、'vertical'：表示流量统计方向选择；'custom'：表示设置车辆禁停区域 |
| --region_polygon | Option | 设置禁停区域多边形多点的坐标，无默认值 |
| --illegal_parking_time | Option | 设置禁停时间阈值，单位秒（s），-1（默认值）表示不做检查 |
## 方案介绍
PP-Vehicle 整体方案如下图所示:
<div width="1000" align="center">
<img src="https://user-images.githubusercontent.com/31800336/218659932-31f4298c-042d-436d-9845-18879f5d31e3.png"/>
</div>
### 车辆检测
- 采用PP-YOLOE L 作为目标检测模型
- 详细文档参考[PP-YOLOE](../../../../configs/ppyoloe/)和[检测跟踪文档](ppvehicle_mot.md)
### 车辆跟踪
- 采用SDE方案完成车辆跟踪
- 检测模型使用PP-YOLOE L(高精度)和S(轻量级)
- 跟踪模块采用OC-SORT方案
- 详细文档参考[OC-SORT](../../../../configs/mot/ocsort)和[检测跟踪文档](ppvehicle_mot.md)
### 属性识别
- 使用PaddleClas提供的特色模型PP-LCNet，实现对车辆颜色及车型属性的识别。
- 详细文档参考[属性识别](ppvehicle_attribute.md)
### 车牌识别
- 使用PaddleOCR特色模型ch_PP-OCRv3_det+ch_PP-OCRv3_rec模型，识别车牌号码
- 详细文档参考[车牌识别](ppvehicle_plate.md)
### 违章停车识别
- 车辆跟踪模型使用高精度模型PP-YOLOE L，根据车辆的跟踪轨迹以及指定的违停区域判断是否违章停车，如果存在则展示违章停车车牌号。
- 详细文档参考[违章停车识别](ppvehicle_illegal_parking.md)
### 违法分析-逆行
- 违法分析-逆行，通过使用高精度分割模型PP-Seg，对车道线进行分割拟合，然后与车辆轨迹组合判断车辆行驶方向是否与道路方向一致。
- 详细文档参考[违法分析-逆行](ppvehicle_retrograde.md)
### 违法分析-压线
- 违法分析-逆行，通过使用高精度分割模型PP-Seg，对车道线进行分割拟合，然后与车辆区域是否覆盖实线区域，进行压线判断。
- 详细文档参考[违法分析-压线](ppvehicle_press.md)

254
paddle_detection/deploy/pipeline/docs/tutorials/PPVehicle_QUICK_STARTED_en.md Normal file
View File

@@ -0,0 +1,254 @@
English | [简体中文](PPVehicle_QUICK_STARTED.md)
# Quick Start for PP-Vehicle
## Content
- [Environment Preparation](#Environment-Preparation)
- [Model Download](#Model-Download)
- [Configuration](#Configuration)
- [Inference Deployment](#Inference-Deployment)
- [rtsp_stream](#rtsp_stream)
- [Nvidia_Jetson](#Nvidia_Jetson)
- [Parameters](#Parameters)
- [Solutions](#Solutions)
- [Vehicle Detection](#Vehicle-Detection)
- [Vehicle Tracking](#Vehicle-Tracking)
- [License Plate Recognition](#License-Plate-Recognition)
- [Attribute Recognition](#Attribute-Recognition)
- [Illegal Parking Detection](#Illegal-Parking-Detection)
## Environment Preparation
Environment Preparation: PaddleDetection version >= release/2.5 or develop
Installation of PaddlePaddle and PaddleDetection
```
# PaddlePaddle CUDA10.1
python -m pip install paddlepaddle-gpu==2.2.2.post101 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html
# PaddlePaddle CPU
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
# Clone PaddleDetectionrepositories
cd <path/to/clone/PaddleDetection>
git clone https://github.com/PaddlePaddle/PaddleDetection.git
# Installing dependencies
cd PaddleDetection
pip install -r requirements.txt
```
1. For installation details, please refer to [Installation Tutorials](../../../../docs/tutorials/INSTALL.md)
2. If you need TensorRT inference acceleration (speed measurement), please install PaddlePaddle with `TensorRT version`. You can download and install it from the [PaddlePaddle Installation Package](https://paddleinference.paddlepaddle.org.cn/v2.2/user_guides/download_lib.html#python) or follow the [Instructions]([https://www](https://www). paddlepaddle.org.cn/inference/master/optimize/paddle_trt.html) or use docker, or self-compiling to prepare the environment.
## Model Download
PP-Vehicle provides object detection, attribute recognition, behaviour recognition and ReID pre-trained models for different applications. Developers can download them directly.
| Task | End-to（ms） | Model Solution | Model Size |
|:---------------------------------:|:----------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------:|
| Vehicle Detection（high precision） | 25.7ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) | 182M |
| Vehicle Detection（Lightweight） | 13.2ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_ppvehicle.zip) | 27M |
| Vehicle detection (super lightweight) | 10ms(Jetson AGX) | [object detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppvehicle/ppyoloe_plus_crn_t_auxhead_320_60e_ppvehicle.tar.gz) | 17M |
| Vehicle Tracking（high precision） | 40ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) | 182M |
| Vehicle Tracking（Lightweight） | 25ms | [Multi-Object Tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_ppvehicle.zip) | 27M |
| Vehicle tracking (super lightweight) | 13.2ms(Jetson AGX) | [multi-object tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppvehicle/ppyoloe_plus_crn_t_auxhead_320_60e_ppvehicle.tar.gz) | 17M |
| License plate recognition | 4.68ms | [License plate recognition](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz) <br> [License plate character recognition](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz) | Vehicle Detection：3.9M <br> License plate character recognition： 12M |
| Vehicle Attribute Recognition | 7.31ms | [Vehicle Attribute](https://bj.bcebos.com/v1/paddledet/models/pipeline/vehicle_attribute_model.zip) | 7.2M |
| Lane line Segmentation | 47ms | [Lane line Segmentation](https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip) | 47M |
Download the model and unzip it into the `. /output_inference` folder.
In the configuration file, the model path defaults to the download path of the model. If the user does not change it, the corresponding model will be downloaded automatically upon inference.
**Notes：**
- The accuracy of detection tracking model is obtained from the joint dataset PPVehicle (integration of the public dataset BDD100K-MOT and UA-DETRAC). For more details, please refer to [PP-Vehicle](../../../../configs/ppvehicle)
- Inference speed is obtained at T4 with TensorRT FP16 enabled, which includes data pre-processing, model inference and post-processing.
## Configuration
PP-Vehicle related configuration locates in ``deploy/pipeline/config/infer_cfg_ppvehicle.yml``. Developers need to set specific task types to use different features.
The features and corresponding task types are as follows.
| Input | Feature | Task | Config |
| ------------------- | --------------------- | ------------------------------------------- | ---------------- |
| Image | Attribute Recognition | Object Detection Attribute Recognition | DET ATTR |
| Single-camera video | Attribute Recognition | Multi-Object Tracking Attribute Recognition | MOT ATTR |
| Single-camera video | License-plate Recognition | Multi-Object Tracking License-plate Recognition | MOT VEHICLEPLATE |
Take attribute recognition based on video input as an example: Its task type includes multi-object tracking and attributes recognition. The specific configuration is as follows.
```
crop_thresh: 0.5
visual: True
warmup_frame: 50
MOT:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1
enable: True
VEHICLE_ATTR:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/vehicle_attribute_model.zip
batch_size: 8
color_threshold: 0.5
type_threshold: 0.5
enable: True
```
**Notes：**
- If the developer needs to carry out different tasks, set the corresponding enables option to be True in the configuration file.
- If the developer only needs to modify the model file path, run the command line with `-o MOT.model_dir=ppyoloe/` after --config, or manually modify the corresponding model path in the configuration file. For more details, please refer to the following parameter descriptions
## Inference Deployment
1. Use the default configuration directly or the configuration file in examples, or modify the configuration in `infer_cfg_ppvehicle.yml`
```
# Example：In vehicle detection，specify configuration file path and test image
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml --image_file=test_image.jpg --device=gpu
# Example：In license plate recognition，directly configure the examples
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_plate.yml --video_file=test_video.mp4 --device=gpu
```
2. Use the command line to enable functions or change the model path.
```
# Example：In vehicle tracking，specify configuration file path and test video, Turn on the MOT model and modify the model path on the command line, the model path specified on the command line has higher priority than the configuration file
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml -o MOT.enable=True MOT.model_dir=ppyoloe_infer/ --video_file=test_video.mp4 --device=gpu
# Example：In vehicle illegal action analysis，specify configuration file path and test video, 命令行中指定违停区域设置、违停时间判断。
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml \
--video_file=../car_test.mov \
--device=gpu \
--draw_center_traj \
--illegal_parking_time=3 \
--region_type=custom \
--region_polygon 600 300 1300 300 1300 800 600 800
```
### rtsp_stream
The online stream decode based on opencv Capture function, normally support rtsp and rtmp.
- rtsp pull stream
For rtsp pull stream, use --rtsp RTSP [RTSP ...] parameter to specify one or more rtsp streams. Separate the multiple addresses with a space, or replace the video address directly after the video_file with the rtsp stream address), examples as follows
```
# Example: Single video stream for pedestrian attribute recognition
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE] --device=gpu
# Example: Multiple-video stream for pedestrian attribute recognition
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_human_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE1] rtsp://[YOUR_RTSP_SITE2] --device=gpu |
```
- rtsp push stream
For rtsp push stream, use --pushurl rtsp:[IP] parameter to push stream to a IP set, and you can visualize the output video by [VLC Player](https://vlc.onl/) with the `open network` funciton. the whole url path is `rtsp:[IP]/videoname`, the videoname here is the basename of the video file to infer, and the default of videoname is `output` when the video come from local camera and there is no video name.
```
# Example：license plate recognition，in this example the whole url path is: rtsp://[YOUR_SERVER_IP]:8554/test_video
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_plate.yml --video_file=test_video.mp4 --device=gpu --pushurl rtsp://[YOUR_SERVER_IP]:8554
```
Note:
1. rtsp push stream is based on [rtsp-simple-server](https://github.com/aler9/rtsp-simple-server), please enable this serving first.
It's very easy to use: 1) download the [release package](https://github.com/aler9/rtsp-simple-server/releases) which is compatible with your workspace. 2) run command './rtsp-simple-server', which works as a rtsp server.
2. the output visualize will be frozen frequently if the model cost too much time, we suggest to use faster model like ppyoloe_s in tracking, this is simply replace mot_ppyoloe_l_36e_pipeline.zip with mot_ppyoloe_s_36e_pipeline.zip in model config yaml file.
### Nvidia_Jetson
Due to the large gap in computing power of the Jetson platform compared to the server, we suggest:
1. choose a lightweight model, we provide a new model named [PP-YOLOE-Plus Tiny](../../../../configs/ppvehicle/README.md)，which achieve 20fps with four rtsp streams work togather on Jetson AGX.
2. For further speedup, you can set frame skipping of tracking; we recommend 2 or 3: `skip_frame_num: 3`
PP-YOLOE-Plus Tiny module speed test data on AGX：（a single car in the test video）
| module | time cost per frame(ms) | speed(fps) |
|:----------|:----------|:----------|
| tracking | 13 | 77 |
| Attribute | 20.2 | 49.4 |
| Plate | - | - |
### Parameters
#
| Parameters | Necessity | Implications |
| ---------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| --config | Yes | Path to configuration file |
| -o | Option | Overwrite the corresponding configuration in the configuration file |
| --image_file | Option | Images to be predicted |
| --image_dir | Option | Path to the images folder to be predicted |
| --video_file | Option | Video to be predicted, or rtsp stream address (rtsp parameter recommended) |
| --rtsp | Option | rtsp video stream address, supports one or more simultaneous streams input |
| --camera_id | Option | The camera ID for prediction, default is -1 ( for no camera prediction, can be set to 0 - (number of cameras - 1) ), press `q` in the visualization interface during the prediction process to output the prediction result to: output/output.mp4 |
| --device | Option | Running device, options include `CPU/GPU/XPU`, and the default is `CPU`. |
| --pushurl | Option | push the output video to rtsp stream, normaly start with `rtsp://`; this has higher priority than local video save, while this is set, pipeline will not save local visualize video, the default is "", means this will not work now.|
| --output_dir | Option | The root directory for the visualization results, and the default is output/ |
| --run_mode | Option | For GPU, the default is paddle, with (paddle/trt_fp32/trt_fp16/trt_int8) as optional |
| --enable_mkldnn | Option | Whether to enable MKLDNN acceleration in CPU prediction, the default is False |
| --cpu_threads | Option | Set the number of cpu threads, and the default is 1 |
| --trt_calib_mode | Option | Whether TensorRT uses the calibration function, and the default is False; set to True when using TensorRT's int8 function and False when using the PaddleSlim quantized model |
| --do_entrance_counting | Option | Whether to count entrance/exit traffic flows, the default is False |
| --draw_center_traj | Option | Whether to draw center trajectory, the default is False |
| --region_type | Option | 'horizontal' (default), 'vertical': traffic count direction; 'custom': set illegal parking area |
| --region_polygon | Option | Set the coordinates of the polygon multipoint in the illegal parking area. No default. |
| --illegal_parking_time | Option | Set the time threshold for illegal parking in seconds (s), -1 (default) indicates no check |
## Solutions
The overall solution for PP-Vehicle v2 is shown in the graph below:
<div width="1000" align="center">
<img src="https://user-images.githubusercontent.com/31800336/218659932-31f4298c-042d-436d-9845-18879f5d31e3.png"/>
</div>
###
### Vehicle detection
- Take PP-YOLOE L as the object detection model
- For detailed documentation, please refer to [PP-YOLOE](../../../../configs/ppyoloe/) and [Multiple-Object-Tracking](ppvehicle_mot_en.md)
### Vehicle tracking
- Vehicle tracking by SDE solution
- Adopt PP-YOLOE L (high precision) and S (lightweight) for detection models
- Adopt the OC-SORT solution for racking module
- Refer to [OC-SORT](../../../../configs/mot/ocsort) and [Multi-Object Tracking](ppvehicle_mot_en.md) for details
### Attribute Recognition
- Use PP-LCNet provided by PaddleClas to recognize vehicle colours and model attributes.
- For details, please refer to [Attribute Recognition](ppvehicle_attribute_en.md)
### License plate recognition
- Use ch_PP-OCRv3_det+ch_PP-OCRv3_rec model to recognize license plate number
- For details, please refer to [Plate Recognition](ppvehicle_plate_en.md)
### Illegal Parking Detection
- Use vehicle tracking model (high precision) PP-YOLOE L to determine whether the parking is illegal based on the vehicle's trajectory and the designated illegal parking area. If it is illegal parking, display the illegal parking plate number.
- For details, please refer to [Illegal Parking Detection](ppvehicle_illegal_parking_en.md)
#### Vehicle Press Line
- Use segmentation model PP-LiteSeg to get the lane line in frame, combine it with vehicle route to find out the vehicle against traffic.
- For details, please refer to [Vehicle Press Line](ppvehicle_press_en.md)
#### Vehicle Retrograde
- Use segmentation model PP-LiteSeg to get the lane line in frame, combine it with vehicle detection box to juege if the car is pressing on lines.
- For details, please refer to [Vehicle Retrograde](ppvehicle_retrograde_en.md)

272
paddle_detection/deploy/pipeline/docs/tutorials/pphuman_action.md Normal file
View File

@@ -0,0 +1,272 @@
[English](pphuman_action_en.md) | 简体中文
# PP-Human行为识别模块
## 目录
- [基于骨骼点的行为识别](#基于骨骼点的行为识别)
- [基于图像分类的行为识别](#基于图像分类的行为识别)
- [基于检测的行为识别](#基于检测的行为识别)
- [基于行人轨迹的行为识别](#基于行人轨迹的行为识别)
- [基于视频分类的行为识别](#基于视频分类的行为识别)
行为识别在智慧社区，安防监控等方向具有广泛应用，根据行为的不同，PP-Human中集成了基于视频分类、基于检测、基于图像分类，基于行人轨迹以及基于骨骼点的行为识别模块，方便用户根据需求进行选择。
## 基于骨骼点的行为识别
应用行为：摔倒识别
<div align="center">
<img src="https://user-images.githubusercontent.com/22989727/205582385-08a1b6ae-9b1b-465a-ac25-d6427571eb56.gif" width='600'/><br>
<center>数据来源及版权归属：天覆科技，感谢提供并开源实际场景数据，仅限学术研究使用</center>
</div>
### 模型库
基于骨骼点的行为识别包含行人检测/跟踪，关键点检测和摔倒行为识别三个模型，首先需要下载以下预训练模型
| 任务 | 算法 | 精度 | 预测速度(ms) | 模型权重 | 预测部署模型 |
|:---------------------|:---------:|:------:|:------:| :------: |:---------------------------------------------------------------------------------: |
| 行人检测/跟踪 | PP-YOLOE | mAP: 56.3 <br> MOTA: 72.0 | 检测: 16.2ms <br> 跟踪：22.3ms |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.pdparams) |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) |
| 关键点识别 | HRNet | AP: 87.1 | 单人 2.9ms |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/dark_hrnet_w32_256x192.pdparams) |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/dark_hrnet_w32_256x192.zip)|
| 摔倒行为识别 | ST-GCN | 准确率: 96.43 | 单人 2.7ms | - |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/STGCN.zip) |
注：
1. 检测/跟踪模型精度为[MOT17](https://motchallenge.net/)，[CrowdHuman](http://www.crowdhuman.org/)，[HIEVE](http://humaninevents.org/)和部分业务数据融合训练测试得到。
2. 关键点模型使用[COCO](https://cocodataset.org/)，[UAV-Human](https://github.com/SUTDCV/UAV-Human)和部分业务数据融合训练, 精度在业务数据测试集上得到。
3. 摔倒行为识别模型使用[NTU-RGB+D](https://rose1.ntu.edu.sg/dataset/actionRecognition/)，[UR Fall Detection Dataset](http://fenix.univ.rzeszow.pl/~mkepski/ds/uf.html)和部分业务数据融合训练，精度在业务数据测试集上得到。
4. 预测速度为NVIDIA T4 机器上使用TensorRT FP16时的速度, 速度包含数据预处理、模型预测、后处理全流程。
### 配置说明
[配置文件](../../config/infer_cfg_pphuman.yml)中与行为识别相关的参数如下：
```
SKELETON_ACTION: # 基于骨骼点的行为识别模型配置
model_dir: output_inference/STGCN # 模型所在路径
batch_size: 1 # 预测批大小。 当前仅支持为1进行推理
max_frames: 50 # 动作片段对应的帧数。在行人ID对应时序骨骼点结果时达到该帧数后，会通过行为识别模型判断该段序列的动作类型。与训练设置一致时效果最佳。
display_frames: 80 # 显示帧数。当预测结果为摔倒时，在对应人物ID中显示状态的持续时间。
coord_size: [384, 512] # 坐标统一缩放到的尺度大小。与训练设置一致时效果最佳。
enable: False # 是否开启该功能
```
### 使用方法
1. 从`模型库`中下载`行人检测/跟踪`、`关键点识别`、`摔倒行为识别`三个预测部署模型并解压到```./output_inference```路径下;默认自动下载模型，如果手动下载，需要修改模型文件夹为模型存放路径。
2. 目前行为识别模块仅支持视频输入，根据期望开启的行为识别方案类型，设置infer_cfg_pphuman.yml中`SKELETON_ACTION`的enable: True, 然后启动命令如下：
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu \
```
3. 若修改模型路径，有以下两种方式：
- ```./deploy/pipeline/config/infer_cfg_pphuman.yml```下可以配置不同模型路径，关键点模型和摔倒行为识别模型分别对应`KPT`和`SKELETON_ACTION`字段，修改对应字段下的路径为实际期望的路径即可。
- 命令行中--config后面紧跟着增加`-o KPT.model_dir=xxx SKELETON_ACTION.model_dir=xxx `修改模型路径：
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
-o KPT.model_dir=./dark_hrnet_w32_256x192 SKELETON_ACTION.model_dir=./STGCN \
--video_file=test_video.mp4 \
--device=gpu
```
4. 启动命令中的完整参数说明，请参考[参数说明](./PPHuman_QUICK_STARTED.md)。
### 方案说明
1. 使用多目标跟踪获取视频输入中的行人检测框及跟踪ID序号，模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../../configs/ppyoloe/README_cn.md)，跟踪方案为OC-SORT，详细文档参考[OC-SORT](../../../../configs/mot/ocsort)。
2. 通过行人检测框的坐标在输入视频的对应帧中截取每个行人。
3. 使用[关键点识别模型](../../../../configs/keypoint/hrnet/dark_hrnet_w32_256x192.yml)得到对应的17个骨骼特征点。骨骼特征点的顺序及类型与COCO一致，详见[如何准备关键点数据集](../../../../docs/tutorials/data/PrepareKeypointDataSet.md)中的`COCO数据集`部分。
4. 每个跟踪ID对应的目标行人各自累计骨骼特征点结果，组成该人物的时序关键点序列。当累计到预定帧数或跟踪丢失后，使用行为识别模型判断时序关键点序列的动作类型。当前版本模型支持摔倒行为的识别，预测得到的`class id`对应关系为：
```
0: 摔倒，
1: 其他
```
- 摔倒行为识别模型使用了[ST-GCN](https://arxiv.org/abs/1801.07455)，并基于[PaddleVideo](https://github.com/PaddlePaddle/PaddleVideo/blob/develop/docs/zh-CN/model_zoo/recognition/stgcn.md)套件完成模型训练。
## 基于图像分类的行为识别
应用行为：打电话识别
<div align="center">
<img src="https://user-images.githubusercontent.com/22989727/205596971-d92fd24e-977a-4742-91cc-ce5b4802473c.gif" width='600'/><br>
<center>数据来源及版权归属：天覆科技，感谢提供并开源实际场景数据，仅限学术研究使用</center>
</div>
### 模型库
基于图像分类的行为识别包含行人检测/跟踪，打电话识别两个模型，首先需要下载以下预训练模型
| 任务 | 算法 | 精度 | 预测速度(ms) | 模型权重 | 预测部署模型 |
|:---------------------|:---------:|:------:|:------:| :------: |:---------------------------------------------------------------------------------: |
| 行人检测/跟踪 | PP-YOLOE | mAP: 56.3 <br> MOTA: 72.0 | 检测: 16.2ms <br> 跟踪：22.3ms |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.pdparams) |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) |
| 打电话识别 | PP-HGNet | 准确率: 86.85 | 单人 2.94ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_tiny_calling_halfbody.pdparams) | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_tiny_calling_halfbody.zip) |
注：
1. 检测/跟踪模型精度为[MOT17](https://motchallenge.net/)，[CrowdHuman](http://www.crowdhuman.org/)，[HIEVE](http://humaninevents.org/)和部分业务数据融合训练测试得到。
2. 打电话行为识别模型使用[UAV-Human](https://github.com/SUTDCV/UAV-Human)的打电话行为部分进行训练和测试。
3. 预测速度为NVIDIA T4 机器上使用TensorRT FP16时的速度, 速度包含数据预处理、模型预测、后处理全流程。
### 配置说明
[配置文件](../../config/infer_cfg_pphuman.yml)中相关的参数如下：
```
ID_BASED_CLSACTION: # 基于分类的行为识别模型配置
model_dir: output_inference/PPHGNet_tiny_calling_halfbody # 模型所在路径
batch_size: 8 # 预测批大小
threshold: 0.45 #识别为对应行为的阈值
display_frames: 80 # 显示帧数。当识别到对应动作时，在对应人物ID中显示状态的持续时间。
enable: False # 是否开启该功能
```
### 使用方法
1. 从`模型库`中下载`行人检测/跟踪`、`打电话行为识别`两个预测部署模型并解压到`./output_inference`路径下；默认自动下载模型，如果手动下载，需要修改模型文件夹为模型存放路径。
2. 修改配置文件`deploy/pipeline/config/infer_cfg_pphuman.yml`中`ID_BASED_CLSACTION`下的`enable`为`True`；
3. 仅支持输入视频，启动命令如下：
```
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. 启动命令中的完整参数说明，请参考[参数说明](./PPHuman_QUICK_STARTED.md)。
### 方案说明
1. 使用目标检测与多目标跟踪获取视频输入中的行人检测框及跟踪ID序号，模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../../configs/ppyoloe/README_cn.md)，跟踪方案为OC-SORT，详细文档参考[OC-SORT](../../../../configs/mot/ocsort)。
2. 通过行人检测框的坐标在输入视频的对应帧中截取每个行人。
3. 通过在帧级别的行人图像通过图像分类的方式实现。当图片所属类别为对应行为时，即认为在一定时间段内该人物处于该行为状态中。该任务使用[PP-HGNet](https://github.com/PaddlePaddle/PaddleClas/blob/develop/docs/zh_CN/models/PP-HGNet.md)实现，当前版本模型支持打电话行为的识别，预测得到的`class id`对应关系为：
```
0: 打电话，
1: 其他
```
- 基于分类的行为识别基于[PaddleClas](https://github.com/PaddlePaddle/PaddleClas/blob/develop/docs/zh_CN/models/PP-HGNet.md#3.3)完成模型训练。
## 基于检测的行为识别
应用行为：吸烟识别
<div align="center">
<img src="https://user-images.githubusercontent.com/22989727/205599300-380c3805-63d6-43cc-9b77-2687b1328d7b.gif" width='600'/><br>
<center>数据来源及版权归属：天覆科技，感谢提供并开源实际场景数据，仅限学术研究使用</center>
</div>
### 模型库
在这里，我们提供了行人检测/跟踪、吸烟行为识别的预训练模型，用户可以直接下载使用。
| 任务 | 算法 | 精度 | 预测速度(ms) | 模型权重 | 预测部署模型 |
|:---------------------|:---------:|:------:|:------:| :------: |:---------------------------------------------------------------------------------: |
| 行人检测/跟踪 | PP-YOLOE | mAP: 56.3 <br> MOTA: 72.0 | 检测: 16.2ms <br> 跟踪：22.3ms |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.pdparams) |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) |
| 吸烟行为识别 | PP-YOLOE | mAP: 39.7 | 单人 2.0ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppyoloe_crn_s_80e_smoking_visdrone.pdparams) | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppyoloe_crn_s_80e_smoking_visdrone.zip) |
注：
1. 检测/跟踪模型精度为[MOT17](https://motchallenge.net/)，[CrowdHuman](http://www.crowdhuman.org/)，[HIEVE](http://humaninevents.org/)和部分业务数据融合训练测试得到。
2. 抽烟行为识别模型使用业务数据进行训练和测试。
3. 预测速度为NVIDIA T4 机器上使用TensorRT FP16时的速度, 速度包含数据预处理、模型预测、后处理全流程。
### 配置说明
[配置文件](../../config/infer_cfg_pphuman.yml)中相关的参数如下：
```
ID_BASED_DETACTION: # 基于检测的行为识别模型配置
model_dir: output_inference/ppyoloe_crn_s_80e_smoking_visdrone # 模型所在路径
batch_size: 8 # 预测批大小
threshold: 0.4 # 识别为对应行为的阈值
display_frames: 80 # 显示帧数。当识别到对应动作时，在对应人物ID中显示状态的持续时间。
enable: False # 是否开启该功能
```
### 使用方法
1. 从`模型库`中下载`行人检测/跟踪`、`抽烟行为识别`两个预测部署模型并解压到`./output_inference`路径下；默认自动下载模型，如果手动下载，需要修改模型文件夹为模型存放路径。
2. 修改配置文件`deploy/pipeline/config/infer_cfg_pphuman.yml`中`ID_BASED_DETACTION`下的`enable`为`True`；
3. 仅支持输入视频，启动命令如下：
```
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. 启动命令中的完整参数说明，请参考[参数说明](./PPHuman_QUICK_STARTED.md)。
### 方案说明
1. 使用目标检测与多目标跟踪获取视频输入中的行人检测框及跟踪ID序号，模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../../configs/ppyoloe/README_cn.md)，跟踪方案为OC-SORT，详细文档参考[OC-SORT](../../../../configs/mot/ocsort)。
2. 通过行人检测框的坐标在输入视频的对应帧中截取每个行人。
3. 通过在帧级别的行人图像中检测该行为的典型特定目标实现。当检测到特定目标(在这里即烟头）以后，即认为在一定时间段内该人物处于该行为状态中。该任务使用[PP-YOLOE](../../../../configs/ppyoloe/README_cn.md)实现，当前版本模型支持吸烟行为的识别，预测得到的`class id`对应关系为：
```
0: 吸烟，
1: 其他
```
## 基于行人轨迹的行为识别
应用行为：闯入识别
<div align="center">
<img src="https://user-images.githubusercontent.com/22989727/178769370-03ab1965-cfd1-401b-9902-82620a06e43c.gif" width='600'/>
</div>
具体使用请参照[PP-Human检测跟踪模块](pphuman_mot.md)的`5. 区域闯入判断和计数`。
### 方案说明
1. 使用多目标跟踪获取视频输入中的行人检测框及跟踪ID序号，模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../../configs/ppyoloe/README_cn.md)，跟踪方案为OC-SORT，详细文档参考[OC-SORT](../../../../configs/mot/ocsort)。
2. 通过行人检测框的下边界中点在相邻帧位于用户所选区域的内外位置，来识别是否闯入所选区域。
## 基于视频分类的行为识别
应用行为：打架识别
<div align="center">
<img src="https://user-images.githubusercontent.com/22989727/205597198-8b4333b3-6c39-472c-a25c-018dac908867.gif" width='600'/><br>
<center>数据来源及版权归属：Surveillance Camera Fight Dataset。</center>
</div>
该方案关注的场景为监控摄像头下的打架行为识别。打架行为涉及多人，基于骨骼点技术的方案更适用于单人的行为识别。此外，打架行为对时序信息依赖较强，基于检测和分类的方案也不太适用。由于监控场景背景复杂，人的密集程度、光线、拍摄角度等都会对识别造成影响，本方案采用基于视频分类的方式判断视频中是否存在打架行为。针对摄像头距离人较远的情况，通过增大输入图像分辨率优化。由于训练数据有限，采用数据增强的方式提升模型的泛化性能。
### 模型库
在这里，我们提供了打架识别的预训练模型，用户可以直接下载使用。
| 任务 | 算法 | 精度 | 预测速度(ms) | 模型权重 | 预测部署模型 |
|:---------------------|:---------:|:------:|:------:| :------: |:---------------------------------------------------------------------------------: |
| 打架识别 | PP-TSM | 准确率：89.06% | 2s视频 128ms | [下载链接](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.pdparams) | [下载链接](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.zip) |
注：
1. 打架识别模型基于6个公开数据集训练得到：Surveillance Camera Fight Dataset、A Dataset for Automatic Violence Detection in Videos、Hockey Fight Detection Dataset、Video Fight Detection Dataset、Real Life Violence Situations Dataset、UBI Abnormal Event Detection Dataset。
2. 预测速度为NVIDIA T4 机器上使用TensorRT FP16时的速度, 速度包含数据预处理、模型预测、后处理全流程。
### 配置说明
[配置文件](../../config/infer_cfg_pphuman.yml)中与行为识别相关的参数如下：
```
VIDEO_ACTION: # 基于视频分类的行为识别模型配置
model_dir: output_inference/ppTSM # 模型所在路径
batch_size: 1 # 预测批大小。当前仅支持为1进行推理
frame_len: 8 # 累计抽样帧数量，达到该数量后执行一次识别
sample_freq: 7 # 抽样频率，即间隔多少帧抽样一帧
short_size: 340 # 视频帧尺度变换最小边的长度
target_size: 320 # 目标视频帧的大小
enable: False # 是否开启该功能
```
### 使用方法
1. 从上表链接中下载`打架识别`任务的预测部署模型并解压到`./output_inference`路径下；默认自动下载模型，如果手动下载，需要修改模型文件夹为模型存放路径。
2. 修改配置文件`deploy/pphuman/config/infer_cfg_pphuman.yml`中`VIDEO_ACTION`下的`enable`为`True`；
3. 仅支持输入视频，启动命令如下：
```
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. 启动命令中的完整参数说明，请参考[参数说明](./PPHuman_QUICK_STARTED.md)。
### 方案说明
目前打架识别模型使用的是[PP-TSM](https://github.com/PaddlePaddle/PaddleVideo/blob/develop/docs/zh-CN/model_zoo/recognition/pp-tsm.md)，并在PP-TSM视频分类模型训练流程的基础上修改适配，完成模型训练。对于输入的视频或者视频流，进行等间隔抽帧，当视频帧累计到指定数目时，输入到视频分类模型中判断是否存在打架行为。
## 参考文献
```
@inproceedings{stgcn2018aaai,
title = {Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition},
author = {Sijie Yan and Yuanjun Xiong and Dahua Lin},
booktitle = {AAAI},
year = {2018},
}
`````

276
paddle_detection/deploy/pipeline/docs/tutorials/pphuman_action_en.md Normal file
View File

@@ -0,0 +1,276 @@
English | [简体中文](pphuman_action.md)
# Action Recognition Module of PP-Human
Action Recognition is widely used in the intelligent community/smart city, and security monitoring. PP-Human provides the module of video-classification-based, detection-based, image-classification-based and skeleton-based action recognition.
## Model Zoo
There are multiple available pretrained models including pedestrian detection/tracking, keypoint detection, fighting, calling, smoking and fall detection models. Users can download and use them directly.
| Task | Algorithm | Precision | Inference Speed(ms) | Model Weights |Model Inference and Deployment |
|:----------------------------- |:---------:|:-------------------------:|:-----------------------------------:| :-----------------: |:-----------------------------------------------------------------------------------------:|
| Pedestrian Detection/Tracking | PP-YOLOE | mAP: 56.3 <br> MOTA: 72.0 | Detection: 28ms <br>Tracking：33.1ms |[Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.pdparams) |[Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) |
| Calling Recognition | PP-HGNet | Precision Rate: 86.85 | Single Person 2.94ms | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_tiny_calling_halfbody.pdparams) | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_tiny_calling_halfbody.zip) |
| Smoking Recognition | PP-YOLOE | mAP: 39.7 | Single Person 2.0ms | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppyoloe_crn_s_80e_smoking_visdrone.pdparams) | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/ppyoloe_crn_s_80e_smoking_visdrone.zip) |
| Keypoint Detection | HRNet | AP: 87.1 | Single Person 2.9ms |[Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/dark_hrnet_w32_256x192.pdparams) |[Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/dark_hrnet_w32_256x192.zip) |
| Falling Recognition | ST-GCN | Precision Rate: 96.43 | Single Person 2.7ms | - |[Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/STGCN.zip) |
| Fighting Recognition | PP-TSM | Precision Rate: 89.06% | 128ms for a 2sec video | [Link](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.pdparams) | [Link](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.zip) |
Note:
1. The precision of the pedestrian detection/ tracking model is obtained by trainning and testing on [MOT17](https://motchallenge.net/), [CrowdHuman](http://www.crowdhuman.org/), [HIEVE](http://humaninevents.org/) and some business data.
2. The keypoint detection model is trained on [COCO](https://cocodataset.org/), [UAV-Human](https://github.com/SUTDCV/UAV-Human), and some business data, and the precision is obtained on test sets of business data.
3. The falling action recognition model is trained on [NTU-RGB+D](https://rose1.ntu.edu.sg/dataset/actionRecognition/), [UR Fall Detection Dataset](http://fenix.univ.rzeszow.pl/~mkepski/ds/uf.html), and some business data, and the precision is obtained on the testing set of business data.
4. The calling action recognition model is trained and tested on [UAV-Human](https://github.com/SUTDCV/UAV-Human), by using video frames of calling in this dataset.
5. The smoking action recognition model is trained and tested on business data.
6. The fighting action recognition model is trained and tested on 6 public datasets, including Surveillance Camera Fight Dataset, A Dataset for Automatic Violence Detection in Videos, Hockey Fight Detection Dataset, Video Fight Detection Dataset, Real Life Violence Situations Dataset, UBI Abnormal Event Detection Dataset.
7. The inference speed is the speed of using TensorRT FP16 on NVIDIA T4, including the total time of data pre-training, model inference, and post-processing.
## Skeleton-based action recognition -- falling detection
<div align="center"> <img src="https://user-images.githubusercontent.com/22989727/205582385-08a1b6ae-9b1b-465a-ac25-d6427571eb56.gif" width='600'/><br> <center>Data source and copyright owner：Skyinfor
Technology. Thanks for the provision of actual scenario data, which are only
used for academic research here. </center>
</div>
### Description of Configuration
Parameters related to action recognition in the [config file](../../config/infer_cfg_pphuman.yml) are as follow:
```
SKELETON_ACTION: # Config for skeleton-based action recognition model
model_dir: output_inference/STGCN # Path of the model
batch_size: 1 # The size of the inference batch. Current now only support 1.
max_frames: 50 # The number of frames of action segments. When frames of time-ordered skeleton keypoints of each pedestrian ID achieve the max value,the action type will be judged by the action recognition model. If the setting is the same as the training, there will be an ideal inference result.
display_frames: 80 # The number of display frames. When the inferred action type is falling down, the time length of the act will be displayed in the ID.
coord_size: [384, 512] # The unified size of the coordinate, which is the best when it is the same as the training setting.
enable: False # Whether to enable this function
```
## How to Use
1. Download models `Pedestrian Detection/Tracking`, `Keypoint Detection` and `Falling Recognition` from the links in the Model Zoo and unzip them to ```./output_inference```. The models are automatically downloaded by default. If you download them manually, you need to modify the `model_dir` as the model storage path.
2. Now the only available input is the video input in the action recognition module. set the "enable: True" of `SKELETON_ACTION` in infer_cfg_pphuman.yml. And then run the command:
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
3. There are two ways to modify the model path:
- In ```./deploy/pipeline/config/infer_cfg_pphuman.yml```, you can configurate different model paths，which is proper only if you match keypoint models and action recognition models with the fields of `KPT` and `SKELETON_ACTION` respectively, and modify the corresponding path of each field into the expected path.
- Add `-o KPT.model_dir=xxx SKELETON_ACTION.model_dir=xxx ` in the command line following the --config to change the model path：
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
-o KPT.model_dir=./dark_hrnet_w32_256x192 SKELETON_ACTION.model_dir=./STGCN \
--video_file=test_video.mp4 \
--device=gpu
```
4. For detailed parameter description, please refer to [Parameter Description](./PPHuman_QUICK_STARTED.md)
### Introduction to the Solution
1. Get the pedestrian detection box and the tracking ID number of the video input through object detection and multi-object tracking. The adopted model is PP-YOLOE, and for details, please refer to [PP-YOLOE](../../../../configs/ppyoloe).
2. Capture every pedestrian in frames of the input video accordingly by using the coordinate of the detection box.
3. In this strategy, we use the [keypoint detection model](../../../../configs/keypoint/hrnet/dark_hrnet_w32_256x192.yml) to obtain 17 skeleton keypoints. Their sequences and types are identical to those of COCO. For details, please refer to the `COCO dataset` part of [how to prepare keypoint datasets](../../../../docs/tutorials/data/PrepareKeypointDataSet_en.md).
4. Each target pedestrian with a tracking ID has their own accumulation of skeleton keypoints, which is used to form a keypoint sequence in time order. When the number of accumulated frames reach a preset threshold or the tracking is lost, the action recognition model will be applied to judging the action type of the time-ordered keypoint sequence. The current model only supports the recognition of the act of falling down, and the relationship between the action type and `class id` is：
```
0: Fall down
1: Others
```
- The falling action recognition model uses [ST-GCN](https://arxiv.org/abs/1801.07455), and employ the [PaddleVideo](https://github.com/PaddlePaddle/PaddleVideo/blob/develop/docs/zh-CN/model_zoo/recognition/stgcn.md) toolkit to complete model training.
## Image-Classification-Based Action Recognition -- Calling Recognition
<div align="center"> <img src="https://user-images.githubusercontent.com/22989727/205596971-d92fd24e-977a-4742-91cc-ce5b4802473c.gif" width='600'/><br> <center>Data source and copyright owner：Skyinfor
Technology. Thanks for the provision of actual scenario data, which are only
used for academic research here. </center>
</div>
### Description of Configuration
Parameters related to action recognition in the [config file](../../config/infer_cfg_pphuman.yml) are as follow:
```
ID_BASED_CLSACTION: # config for classfication-based action recognition model
model_dir: output_inference/PPHGNet_tiny_calling_halfbody # Path of the model
batch_size: 8 # The size of the inference batch
threshold: 0.45 # Threshold for corresponding behavior
display_frames: 80 # The number of display frames. When the corresponding action is detected, the time length of the act will be displayed in the ID.
enable: False # Whether to enable this function
```
### How to Use
1. Download models `Pedestrian Detection/Tracking` and `Calling Recognition` from the links in `Model Zoo` and unzip them to ```./output_inference```. The models are automatically downloaded by default. If you download them manually, you need to modify the `model_dir` as the model storage path.
2. Now the only available input is the video input in the action recognition module. Set the "enable: True" of `ID_BASED_CLSACTION` in infer_cfg_pphuman.yml.
3. Run this command:
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. For detailed parameter description, please refer to [Parameter Description](./PPHuman_QUICK_STARTED.md)
### Introduction to the Solution
1. Get the pedestrian detection box and the tracking ID number of the video input through object detection and multi-object tracking. The adopted model is PP-YOLOE, and for details, please refer to [PP-YOLOE](../../../configs/ppyoloe).
2. Capture every pedestrian in frames of the input video accordingly by using the coordinate of the detection box.
3. With image classification through pedestrian images at the frame level, when the category to which the image belongs is the corresponding behavior, it is considered that the character is in the behavior state for a certain period of time. This task is implemented with [PP-HGNet](https://github.com/PaddlePaddle/PaddleClas/blob/develop/docs/zh_CN/models/PP-HGNet.md). In current version, the behavior of calling is supported and the relationship between the action type and `class id` is:
```
0: Calling
1: Others
```
## Detection-based Action Recognition -- Smoking Detection
<div align="center"> <img src="https://user-images.githubusercontent.com/22989727/205599300-380c3805-63d6-43cc-9b77-2687b1328d7b.gif" width='600'/><br> <center>Data source and copyright owner：Skyinfor
Technology. Thanks for the provision of actual scenario data, which are only
used for academic research here. </center>
</div>
### Description of Configuration
Parameters related to action recognition in the [config file](../../config/infer_cfg_pphuman.yml) are as follow:
```
ID_BASED_DETACTION: # Config for detection-based action recognition model
model_dir: output_inference/ppyoloe_crn_s_80e_smoking_visdrone # Path of the model
batch_size: 8 # The size of the inference batch
threshold: 0.4 # Threshold for corresponding behavior.
display_frames: 80 # The number of display frames. When the corresponding action is detected, the time length of the act will be displayed in the ID.
enable: False # Whether to enable this function
```
### How to Use
1. Download models `Pedestrian Detection/Tracking` and `Smoking Recognition` from the links in `Model Zoo` and unzip them to ```./output_inference```. The models are automatically downloaded by default. If you download them manually, you need to modify the `model_dir` as the model storage path.
2. Now the only available input is the video input in the action recognition module. set the "enable: True" of `ID_BASED_DETACTION` in infer_cfg_pphuman.yml.
3. Run this command:
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. For detailed parameter description, please refer to [Parameter Description](./PPHuman_QUICK_STARTED.md)
### Introduction to the Solution
1. Get the pedestrian detection box and the tracking ID number of the video input through object detection and multi-object tracking. The adopted model is PP-YOLOE, and for details, please refer to [PP-YOLOE](../../../../configs/ppyoloe).
2. Capture every pedestrian in frames of the input video accordingly by using the coordinate of the detection box.
3. We detecting the typical specific target of this behavior in frame-level pedestrian images. When a specific target (in this case, cigarette is the target) is detected, it is considered that the character is in the behavior state for a certain period of time. This task is implemented by [PP-YOLOE](../../../../configs/ppyoloe/). In current version, the behavior of smoking is supported and the relationship between the action type and `class id` is:
```
0: Smoking
1: Others
```
## Video-Classification-Based Action Recognition -- Fighting Detection
With wider and wider deployment of surveillance cameras, it is time-consuming and labor-intensive and inefficient to manually check whether there are abnormal behaviors such as fighting. AI + security assistant smart security. A fight recognition module is integrated into PP-Human to identify whether there is fighting in the video. We provide pre-trained models that users can download and use directly.
| Task | Model | Acc. | Speed(ms) | Weight | Deploy Model |
| ---- | ---- | ---------- | ---- | ---- | ---------- |
| Fighting Detection | PP-TSM | 89.06% | 128ms for a 2-sec video| [Link](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.pdparams) | [Link](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.zip) |
The model is trained with 6 public dataset, including Surveillance Camera Fight Dataset、A Dataset for Automatic Violence Detection in Videos、Hockey Fight Detection Dataset、Video Fight Detection Dataset、Real Life Violence Situations Dataset、UBI Abnormal Event Detection Dataset.
This project focuses on is the identification of fighting behavior under surveillance cameras. Fighting behavior involves multiple people, and the skeleton-based technology is more suitable for single-person behavior recognition. In addition, fighting behavior is strongly dependent on timing information, and the detection and classification-based scheme is not suitable. Due to the complex background of the monitoring scene, the density of people, light, filming angle may affect the accuracy. This solution uses video-classification-based method to determine whether there is fighting in the video.
For the case where the camera is far away from the person, it is optimized by increasing the resolution of the input image. Due to the limited training data, data augmentation is used to improve the generalization performance of the model.
### Description of Configuration
Parameters related to action recognition in the [config file](../../config/infer_cfg_pphuman.yml) are as follow:
```
VIDEO_ACTION: # Config for detection-based action recognition model
model_dir: output_inference/ppTSM # Path of the model
batch_size: 1 # The size of the inference batch. Current now only support 1.
frame_len: 8 # Accumulate the number of sampling frames. Inference will be executed when sampled frames reached this value.
sample_freq: 7 # Sampling frequency. It means how many frames to sample one frame.
short_size: 340 # The shortest length for video frame scaling transforms.
target_size: 320 # Target size for input video
enable: False # Whether to enable this function
```
### How to Use
1. Download model `Fighting Detection` from the links of the above table and unzip it to ```./output_inference```. The models are automatically downloaded by default. If you download them manually, you need to modify the `model_dir` as the model storage path.
2. Modify the file names in the `ppTSM` folder to `model.pdiparams, model.pdiparams.info and model.pdmodel`;
3. Now the only available input is the video input in the action recognition module. set the "enable: True" of `VIDEO_ACTION` in infer_cfg_pphuman.yml.
4. Run this command:
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
5. For detailed parameter description, please refer to [Parameter Description](./PPHuman_QUICK_STARTED.md).
The result is shown as follow:
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205597198-8b4333b3-6c39-472c-a25c-018dac908867.gif"/>
</div>
Data source and copyright owner: Surveillance Camera Fight Dataset.
### Introduction to the Solution
The current fight recognition model is using [PP-TSM](https://github.com/PaddlePaddle/PaddleVideo/blob/develop/docs/zh-CN/model_zoo/recognition/pp-tsm.md), and adaptated to complete the model training. For the input video or video stream, we extraction frame at a certain interval. When the video frame accumulates to the specified number, it is input into the video classification model to determine whether there is fighting.
## Custom Training
The pretrained models are provided and can be used directly, including pedestrian detection/ tracking, keypoint detection, smoking, calling and fighting recognition. If users need to train custom action or optimize the model performance, please refer the link below.
| Task | Model | Development Document |
| ---- | ---- | -------- |
| pedestrian detection/tracking | PP-YOLOE | [doc](../../../../configs/ppyoloe/README.md#getting-start) |
| keypoint detection | HRNet | [doc](../../../../configs/keypoint/README_en.md#3training-and-testing) |
| action recognition (fall down) | ST-GCN | [doc](../../../../docs/advanced_tutorials/customization/action_recognotion/skeletonbased_rec.md) |
| action recognition (smoking) | PP-YOLOE | [doc](../../../../docs/advanced_tutorials/customization/action_recognotion/idbased_det.md) |
| action recognition (calling) | PP-HGNet | [doc](../../../../docs/advanced_tutorials/customization/action_recognotion/idbased_clas.md) |
| action recognition (fighting) | PP-TSM | [doc](../../../../docs/advanced_tutorials/customization/action_recognotion/videobased_rec.md) |
## Reference
```
@inproceedings{stgcn2018aaai,
title = {Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition},
author = {Sijie Yan and Yuanjun Xiong and Dahua Lin},
booktitle = {AAAI},
year = {2018},
}
```

111
paddle_detection/deploy/pipeline/docs/tutorials/pphuman_attribute.md Normal file
View File

@@ -0,0 +1,111 @@
[English](pphuman_attribute_en.md) | 简体中文
# PP-Human属性识别模块
行人属性识别在智慧社区，工业巡检，交通监控等方向都具有广泛应用，PP-Human中集成了属性识别模块，属性包含性别、年龄、帽子、眼镜、上衣下衣款式等。我们提供了预训练模型，用户可以直接下载使用。
| 任务 | 算法 | 精度 | 预测速度(ms) |下载链接 |
|:---------------------|:---------:|:------:|:------:| :---------------------------------------------------------------------------------: |
| 行人检测/跟踪 | PP-YOLOE | mAP: 56.3 <br> MOTA: 72.0 | 检测: 16.2ms <br> 跟踪：22.3ms |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) |
| 行人属性高精度模型 | PP-HGNet_small | mA: 95.4 | 单人 1.54ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_small_person_attribute_954_infer.zip) |
| 行人属性轻量级模型 | PP-LCNet_x1_0 | mA: 94.5 | 单人 0.54ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPLCNet_x1_0_person_attribute_945_infer.zip) |
| 行人属性精度与速度均衡模型 | PP-HGNet_tiny | mA: 95.2 | 单人 1.14ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_tiny_person_attribute_952_infer.zip) |
1. 检测/跟踪模型精度为[MOT17](https://motchallenge.net/)，[CrowdHuman](http://www.crowdhuman.org/)，[HIEVE](http://humaninevents.org/)和部分业务数据融合训练测试得到。
2. 行人属性分析精度为[PA100k](https://github.com/xh-liu/HydraPlus-Net#pa-100k-dataset)，[RAPv2](http://www.rapdataset.com/rapv2.html)，[PETA](http://mmlab.ie.cuhk.edu.hk/projects/PETA.html)和部分业务数据融合训练测试得到
3. 预测速度为V100 机器上使用TensorRT FP16时的速度, 该处测速速度为模型预测速度
4. 属性模型应用依赖跟踪模型结果，请在[跟踪模型页面](./pphuman_mot.md)下载跟踪模型，依自身需求选择高精或轻量级下载。
5. 模型下载后解压放置在PaddleDetection/output_inference/目录下。
## 使用方法
1. 从上表链接中下载模型并解压到```PaddleDetection/output_inference```路径下，并修改配置文件中模型路径，也可默认自动下载模型。设置```deploy/pipeline/config/infer_cfg_pphuman.yml```中`ATTR`的enable: True
`infer_cfg_pphuman.yml`中配置项说明：
```
ATTR: #模块名称
model_dir: output_inference/PPLCNet_x1_0_person_attribute_945_infer/ #模型路径
batch_size: 8 #推理最大batchsize
enable: False #功能是否开启
```
2. 图片输入时，启动命令如下(更多命令参数说明，请参考[快速开始-参数说明](./PPHuman_QUICK_STARTED.md#41-参数说明))。
```python
#单张图片
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--image_file=test_image.jpg \
--device=gpu \
#图片文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--image_dir=images/ \
--device=gpu \
```
3. 视频输入时，启动命令如下
```python
#单个视频文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu \
#视频文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_dir=test_videos/ \
--device=gpu \
```
4. 若修改模型路径，有以下两种方式：
- 方法一：```./deploy/pipeline/config/infer_cfg_pphuman.yml```下可以配置不同模型路径，属性识别模型修改ATTR字段下配置
- 方法二：命令行中--config后面紧跟着增加`-o ATTR.model_dir`修改模型路径：
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml
-o ATTR.model_dir=output_inference/PPLCNet_x1_0_person_attribute_945_infer/\
--video_file=test_video.mp4 \
--device=gpu
```
测试效果如下：
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205597518-7a602bd5-e643-44a1-a4ca-03c9ffecd918.gif"/>
</div>
数据来源及版权归属：天覆科技，感谢提供并开源实际场景数据，仅限学术研究使用
## 方案说明
1. 目标检测/多目标跟踪获取图片/视频输入中的行人检测框，模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../configs/ppyoloe/README_cn.md)
2. 通过行人检测框的坐标在输入图像中截取每个行人
3. 使用属性识别分析每个行人对应属性，属性类型与PA100k数据集相同，具体属性列表如下：
```
- 性别：男、女
- 年龄：小于18、18-60、大于60
- 朝向：朝前、朝后、侧面
- 配饰：眼镜、帽子、无
- 正面持物：是、否
- 包：双肩包、单肩包、手提包
- 上衣风格：带条纹、带logo、带格子、拼接风格
- 下装风格：带条纹、带图案
- 短袖上衣：是、否
- 长袖上衣：是、否
- 长外套：是、否
- 长裤：是、否
- 短裤：是、否
- 短裙&裙子：是、否
- 穿靴：是、否
```
4. 属性识别模型方案为[StrongBaseline](https://arxiv.org/pdf/2107.03576.pdf)，模型结构更改为基于PP-HGNet、PP-LCNet的多分类网络结构，引入Weighted BCE loss提升模型效果。
## 参考文献
```
@article{jia2020rethinking,
title={Rethinking of pedestrian attribute recognition: Realistic datasets with efficient method},
author={Jia, Jian and Huang, Houjing and Yang, Wenjie and Chen, Xiaotang and Huang, Kaiqi},
journal={arXiv preprint arXiv:2005.11909},
year={2020}
}
```

107
paddle_detection/deploy/pipeline/docs/tutorials/pphuman_attribute_en.md Normal file
View File

@@ -0,0 +1,107 @@
English | [简体中文](pphuman_attribute.md)
# Attribute Recognition Modules of PP-Human
Pedestrian attribute recognition has been widely used in the intelligent community, industrial, and transportation monitoring. Many attribute recognition modules have been gathered in PP-Human, including gender, age, hats, eyes, clothing and up to 26 attributes in total. Also, the pre-trained models are offered here and users can download and use them directly.
| Task | Algorithm | Precision | Inference Speed(ms) | Download Link |
|:---------------------|:---------:|:------:|:------:| :---------------------------------------------------------------------------------: |
| High-Precision Model | PP-HGNet_small | mA: 95.4 | per person 1.54ms | [Download](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_small_person_attribute_954_infer.tar) |
| Fast Model | PP-LCNet_x1_0 | mA: 94.5 | per person 0.54ms | [Download](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPLCNet_x1_0_person_attribute_945_infer.tar) |
| Balanced Model | PP-HGNet_tiny | mA: 95.2 | per person 1.14ms | [Download](https://bj.bcebos.com/v1/paddledet/models/pipeline/PPHGNet_tiny_person_attribute_952_infer.tar) |
1. The precision of pedestiran attribute analysis is obtained by training and testing on the dataset consist of [PA100k](https://github.com/xh-liu/HydraPlus-Net#pa-100k-dataset)，[RAPv2](http://www.rapdataset.com/rapv2.html)，[PETA](http://mmlab.ie.cuhk.edu.hk/projects/PETA.html) and some business data.
2. The inference speed is V100, the speed of using TensorRT FP16.
3. This model of Attribute is based on the result of tracking, please download tracking model in the [Page of Mot](./pphuman_mot_en.md). The High precision and Faster model are both available.
4. You should place the model unziped in the directory of `PaddleDetection/output_inference/`.
## Instruction
1. Download the model from the link in the above table, and unzip it to```./output_inference```, and set the "enable: True" in ATTR of infer_cfg_pphuman.yml
The meaning of configs of `infer_cfg_pphuman.yml`：
```
ATTR: #module name
model_dir: output_inference/PPLCNet_x1_0_person_attribute_945_infer/ #model path
batch_size: 8 #maxmum batchsize when inference
enable: False #whether to enable this model
```
2. When inputting the image, run the command as follows (please refer to [QUICK_STARTED-Parameters](./PPHuman_QUICK_STARTED.md#41-参数说明) for more details):
```python
#single image
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--image_file=test_image.jpg \
--device=gpu \
#image directory
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--image_dir=images/ \
--device=gpu \
```
3. When inputting the video, run the command as follows:
```python
#a single video file
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu \
#directory of videos
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_dir=test_videos/ \
--device=gpu \
```
4. If you want to change the model path, there are two methods：
- The first: In ```./deploy/pipeline/config/infer_cfg_pphuman.yml``` you can configurate different model paths. In attribute recognition models, you can modify the configuration in the field of ATTR.
- The second: Add `-o ATTR.model_dir` in the command line following the --config to change the model path：
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
-o ATTR.model_dir=output_inference/PPLCNet_x1_0_person_attribute_945_infer/\
--video_file=test_video.mp4 \
--device=gpu
```
The test result is：
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/48054808/159898428-5bda0831-7249-4889-babd-9165f26f664d.gif"/>
</div>
Data Source and Copyright：Skyinfor Technology. Thanks for the provision of actual scenario data, which are only used for academic research here.
## Introduction to the Solution
1. The PP-YOLOE model is used to handle detection boxs of input images/videos from object detection/ multi-object tracking. For details, please refer to the document [PP-YOLOE](../../../configs/ppyoloe).
2. Capture every pedestrian in the input images with the help of coordiantes of detection boxes.
3. Analyze the listed labels of pedestirans through attribute recognition. They are the same as those in the PA100k dataset. The label list is as follows:
```
- Gender
- Age: Less than 18; 18-60; Over 60
- Orientation: Front; Back; Side
- Accessories: Glasses; Hat; None
- HoldObjectsInFront: Yes; No
- Bag: BackPack; ShoulderBag; HandBag
- TopStyle: UpperStride; UpperLogo; UpperPlaid; UpperSplice
- BottomStyle: LowerStripe; LowerPattern
- ShortSleeve: Yes; No
- LongSleeve: Yes; No
- LongCoat: Yes; No
- Trousers: Yes; No
- Shorts: Yes; No
- Skirt&Dress: Yes; No
- Boots: Yes; No
```
4. The model adopted in the attribute recognition is [StrongBaseline](https://arxiv.org/pdf/2107.03576.pdf), where the structure is the multi-class network structure based on PP-HGNet、PP-LCNet, and Weighted BCE loss is introduced for effect optimization.
## Reference
```
@article{jia2020rethinking,
title={Rethinking of pedestrian attribute recognition: Realistic datasets with efficient method},
author={Jia, Jian and Huang, Houjing and Yang, Wenjie and Chen, Xiaotang and Huang, Kaiqi},
journal={arXiv preprint arXiv:2005.11909},
year={2020}
}
```

116
paddle_detection/deploy/pipeline/docs/tutorials/pphuman_mot.md Normal file
View File

@@ -0,0 +1,116 @@
[English](pphuman_mot_en.md) | 简体中文
# PP-Human检测跟踪模块
行人检测与跟踪在智慧社区，工业巡检，交通监控等方向都具有广泛应用，PP-Human中集成了检测跟踪模块，是关键点检测、属性行为识别等任务的基础。我们提供了预训练模型，用户可以直接下载使用。
| 任务 | 算法 | 精度 | 预测速度(ms) |下载链接 |
|:---------------------|:---------:|:------:|:------:| :---------------------------------------------------------------------------------: |
| 行人检测/跟踪 | PP-YOLOE-l | mAP: 57.8 <br> MOTA: 82.2 | 检测: 25.1ms <br> 跟踪：31.8ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) |
| 行人检测/跟踪 | PP-YOLOE-s | mAP: 53.2 <br> MOTA: 73.9 | 检测: 16.2ms <br> 跟踪：21.0ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_pipeline.zip) |
1. 检测/跟踪模型精度为[COCO-Person](http://cocodataset.org/), [CrowdHuman](http://www.crowdhuman.org/), [HIEVE](http://humaninevents.org/) 和部分业务数据融合训练测试得到，验证集为业务数据
2. 预测速度为T4 机器上使用TensorRT FP16时的速度, 速度包含数据预处理、模型预测、后处理全流程
## 使用方法
1. 从上表链接中下载模型并解压到```./output_inference```路径下，并修改配置文件中模型路径。默认为自动下载模型，无需做改动。
2. 图片输入时，是纯检测任务，启动命令如下
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--image_file=test_image.jpg \
--device=gpu
```
3. 视频输入时，是跟踪任务，注意首先设置infer_cfg_pphuman.yml中的MOT配置的`enable=True`，如果希望跳帧加速检测跟踪流程，可以设置`skip_frame_num: 2`，建议跳帧帧数最大不超过3：
```
MOT:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1
skip_frame_num: 2
enable: True
```
然后启动命令如下
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. 若修改模型路径，有以下两种方式：
- ```./deploy/pipeline/config/infer_cfg_pphuman.yml```下可以配置不同模型路径，检测和跟踪模型分别对应`DET`和`MOT`字段，修改对应字段下的路径为实际期望的路径即可。
- 命令行中--config后面紧跟着增加`-o MOT.model_dir`修改模型路径：
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
-o MOT.model_dir=ppyoloe/\
--video_file=test_video.mp4 \
--device=gpu \
--region_type=horizontal \
--do_entrance_counting \
--draw_center_traj
```
**注意:**
- `--do_entrance_counting`表示是否统计出入口流量，不设置即默认为False。
- `--draw_center_traj`表示是否绘制跟踪轨迹，不设置即默认为False。注意绘制跟踪轨迹的测试视频最好是静止摄像头拍摄的。
- `--region_type`表示流量计数的区域，当设置`--do_entrance_counting`时可选择`horizontal`或者`vertical`，默认是`horizontal`，表示以视频图片的中心水平线为出入口，同一物体框的中心点在相邻两秒内分别在区域中心水平线的两侧，即完成计数加一。
测试效果如下：
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205599943-8da89ce8-f6d1-47e5-adc8-6d199b76d167.gif"/>
</div>
数据来源及版权归属：天覆科技，感谢提供并开源实际场景数据，仅限学术研究使用
5. 区域闯入判断和计数
注意首先设置infer_cfg_pphuman.yml中的MOT配置的enable=True，然后启动命令如下
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu \
--draw_center_traj \
--do_break_in_counting \
--region_type=custom \
--region_polygon 200 200 400 200 300 400 100 400
```
**注意:**
- 区域闯入的测试视频必须是静止摄像头拍摄的，镜头不能抖动或移动。
- `--do_break_in_counting`表示是否进行区域出入后计数，不设置即默认为False。
- `--region_type`表示流量计数的区域，当设置`--do_break_in_counting`时仅可选择`custom`，默认是`custom`，表示以用户自定义区域为出入口，同一物体框的下边界中点坐标在相邻两秒内从区域外到区域内，即完成计数加一。
- `--region_polygon`表示用户自定义区域的多边形的点坐标序列，每两个为一对点坐标(x,y)，**按顺时针顺序**连成一个**封闭区域**，至少需要3对点也即6个整数，默认值是`[]`，需要用户自行设置点坐标，如是四边形区域，坐标顺序是`左上、右上、右下、左下`。用户可以运行[此段代码](../../tools/get_video_info.py)获取所测视频的分辨率帧数，以及可以自定义画出自己想要的多边形区域的可视化并自己调整。
自定义多边形区域的可视化代码运行如下：
```python
python get_video_info.py --video_file=demo.mp4 --region_polygon 200 200 400 200 300 400 100 400
```
快速画出想要的区域的小技巧：先任意取点得到图片，用画图工具打开，鼠标放到想要的区域点上会显示出坐标，记录下来并取整，作为这段可视化代码的region_polygon参数，并再次运行可视化，微调点坐标参数直至满意。
测试效果如下：
<div align="center">
<img src="https://user-images.githubusercontent.com/22989727/178769370-03ab1965-cfd1-401b-9902-82620a06e43c.gif" width='600'/>
</div>
## 方案说明
1. 使用目标检测/多目标跟踪技术来获取图片/视频输入中的行人检测框，检测模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../../configs/ppyoloe)。
2. 多目标跟踪模型方案采用[ByteTrack](https://arxiv.org/pdf/2110.06864.pdf)和[OC-SORT](https://arxiv.org/pdf/2203.14360.pdf)，采用PP-YOLOE替换原文的YOLOX作为检测器，采用BYTETracker和OCSORTTracker作为跟踪器，详细文档参考[ByteTrack](../../../../configs/mot/bytetrack)和[OC-SORT](../../../../configs/mot/ocsort)。
## 参考文献
```
@article{zhang2021bytetrack,
title={ByteTrack: Multi-Object Tracking by Associating Every Detection Box},
author={Zhang, Yifu and Sun, Peize and Jiang, Yi and Yu, Dongdong and Yuan, Zehuan and Luo, Ping and Liu, Wenyu and Wang, Xinggang},
journal={arXiv preprint arXiv:2110.06864},
year={2021}
}
@article{cao2022observation,
title={Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking},
author={Cao, Jinkun and Weng, Xinshuo and Khirodkar, Rawal and Pang, Jiangmiao and Kitani, Kris},
journal={arXiv preprint arXiv:2203.14360},
year={2022}
}
```

111
paddle_detection/deploy/pipeline/docs/tutorials/pphuman_mot_en.md Normal file
View File

@@ -0,0 +1,111 @@
English | [简体中文](pphuman_mot.md)
# Detection and Tracking Module of PP-Human
Pedestrian detection and tracking is widely used in the intelligent community, industrial inspection, transportation monitoring and so on. PP-Human has the detection and tracking module, which is fundamental to keypoint detection, attribute action recognition, etc. Users enjoy easy access to pretrained models here.
| Task | Algorithm | Precision | Inference Speed(ms) | Download Link |
|:---------------------|:---------:|:------:|:------:| :---------------------------------------------------------------------------------: |
| Pedestrian Detection/ Tracking | PP-YOLOE-l | mAP: 57.8 <br> MOTA: 82.2 | Detection: 25.1ms <br> Tracking：31.8ms | [Download](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip) |
| Pedestrian Detection/ Tracking | PP-YOLOE-s | mAP: 53.2 <br> MOTA: 73.9 | Detection: 16.2ms <br> Tracking：21.0ms | [Download](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_pipeline.zip) |
1. The precision of the pedestrian detection/ tracking model is obtained by trainning and testing on [COCO-Person](http://cocodataset.org/), [CrowdHuman](http://www.crowdhuman.org/), [HIEVE](http://humaninevents.org/) and some business data.
2. The inference speed is the speed of using TensorRT FP16 on T4, the total number of data pre-training, model inference, and post-processing.
## How to Use
1. Download models from the links of the above table and unizp them to ```./output_inference```.
2. When use the image as input, it's a detection task, the start command is as follows:
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--image_file=test_image.jpg \
--device=gpu
```
3. When use the video as input, it's a tracking task, first you should set the "enable: True" in MOT of infer_cfg_pphuman.yml. If you want skip some frames speed up the detection and tracking process, you can set `skip_frame_num: 2`, it is recommended that the maximum number of skip_frame_num should not exceed 3:
```
MOT:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1
skip_frame_num: 2
enable: True
```
and then the start command is as follows:
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. There are two ways to modify the model path:
- In `./deploy/pipeline/config/infer_cfg_pphuman.yml`, you can configurate different model paths，which is proper only if you match keypoint models and action recognition models with the fields of `DET` and `MOT` respectively, and modify the corresponding path of each field into the expected path.
- Add `-o MOT.model_dir` in the command line following the --config to change the model path：
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
-o MOT.model_dir=ppyoloe/\
--video_file=test_video.mp4 \
--device=gpu \
--region_type=horizontal \
--do_entrance_counting \
--draw_center_traj
```
**Note:**
- `--do_entrance_counting` is whether to calculate flow at the gateway, and the default setting is False.
- `--draw_center_traj` means whether to draw the track, and the default setting is False. It's worth noting that the test video of track drawing should be filmed by the still camera.
- `--region_type` means the region type of flow counting. When set `--do_entrance_counting`, you can select from `horizontal` or `vertical`, the default setting is `horizontal`, means that the central horizontal line of the video picture is used as the entrance and exit, and when the central point of the same object box is on both sides of the central horizontal line of the area in two adjacent seconds, the counting plus one is completed.
The test result is：
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205599943-8da89ce8-f6d1-47e5-adc8-6d199b76d167.gif"/>
</div>
Data source and copyright owner：Skyinfor Technology. Thanks for the provision of actual scenario data, which are only used for academic research here.
5. Break in and counting
Please set the "enable: True" in MOT of infer_cfg_pphuman.yml at first, and then the start command is as follows:
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml \
--video_file=test_video.mp4 \
--device=gpu \
--draw_center_traj \
--do_break_in_counting \
--region_type=custom \
--region_polygon 200 200 400 200 300 400 100 400
```
**Note:**
- `--do_break_in_counting` is whether to calculate flow when break in the user-defined region, and the default setting is False.
- `--region_type` means the region type of flow counting. When set `--do_break_in_counting`, only `custom` can be selected, and the default is `custom`, which means that the user-defined region is used as the entrance and exit, and when the midpoint coords of the bottom boundary of the same object moves from outside to inside the region within two adjacent seconds, the counting plus one is completed.
- `--region_polygon` means the point coords sequence of the polygon in the user-defined region. Every two integers are a pair of point coords (x,y), which are connected into a closed area in clockwise order. At least 3 pairs of points, that is, 6 integers, are required. The default value is `[]`, and the user needs to set the point coords by himself. Users can run this [code](../../tools/get_video_info.py) to obtain the resolution and frame number of the measured video, and can customize the visualization of drawing the polygon area they want and adjust it by themselves.
The visualization code of the custom polygon region runs as follows:
```python
python get_video_info.py --video_file=demo.mp4 --region_polygon 200 200 400 200 300 400 100 400
```
The test result is：
<div align="center">
<img src="https://user-images.githubusercontent.com/22989727/178769370-03ab1965-cfd1-401b-9902-82620a06e43c.gif" width='600'/>
</div>
## Introduction to the Solution
1. Get the pedestrian detection box of the image/ video input through object detection and multi-object tracking. The detection model is PP-YOLOE, please refer to [PP-YOLOE](../../../../configs/ppyoloe) for details.
2. The multi-object tracking solution is based on [ByteTrack](https://arxiv.org/pdf/2110.06864.pdf) and [OC-SORT](https://arxiv.org/pdf/2203.14360.pdf), and replace the original YOLOX with PP-YOLOE as the detector，and BYTETracker or OC-SORT Tracker as the tracker, please refer to [ByteTrack](../../../../configs/mot/bytetrack) and [OC-SORT](../../../../configs/mot/ocsort).
## Reference
```
@article{zhang2021bytetrack,
title={ByteTrack: Multi-Object Tracking by Associating Every Detection Box},
author={Zhang, Yifu and Sun, Peize and Jiang, Yi and Yu, Dongdong and Yuan, Zehuan and Luo, Ping and Liu, Wenyu and Wang, Xinggang},
journal={arXiv preprint arXiv:2110.06864},
year={2021}
}
```

90
paddle_detection/deploy/pipeline/docs/tutorials/pphuman_mtmct.md Normal file
View File

@@ -0,0 +1,90 @@
[English](pphuman_mtmct_en.md) | 简体中文
# PP-Human跨镜头跟踪模块
跨镜头跟踪任务，是在单镜头跟踪的基础上，实现不同摄像头中人员的身份匹配关联。在安放、智慧零售等方向有较多的应用。
PP-Human跨镜头跟踪模块主要目的在于提供一套简洁、高效的跨镜跟踪Pipeline，REID模型完全基于开源数据集训练。
## 使用方法
1. 下载模型 [行人跟踪](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_pipeline.zip)和[REID模型](https://bj.bcebos.com/v1/paddledet/models/pipeline/reid_model.zip) 并解压到```./output_inference```路径下，修改配置文件中模型路径。也可简单起见直接用默认配置，自动下载模型。 MOT模型请参考[mot说明](./pphuman_mot.md)文件下载。
2. 跨镜头跟踪模式下，要求输入的多个视频放在同一目录下，同时开启infer_cfg_pphuman.yml 中的REID选择中的enable=True, 命令如下：
```python
python3 deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml --video_dir=[your_video_file_directory] --device=gpu
```
3. 相关配置在`./deploy/pipeline/config/infer_cfg_pphuman.yml`文件中修改：
```python
python3 deploy/pipeline/pipeline.py
--config deploy/pipeline/config/infer_cfg_pphuman.yml -o REID.model_dir=reid_best/
--video_dir=[your_video_file_directory]
--device=gpu
```
## 方案说明
跨镜头跟踪模块，主要由跨镜头跟踪Pipeline及REID模型两部分组成。
1. 跨镜头跟踪Pipeline
```
单镜头跟踪[id+bbox]
│
根据bbox截取原图中目标——│
│ │
REID模型 质量评估(遮挡、完整度、亮度等)
│ │
[feature] [quality]
│ │
datacollector—————│
│
特征排序、筛选
│
多视频各id相似度计算
│
id聚类、重新分配id
```
2. 模型方案为[reid-strong-baseline](https://github.com/michuanhaohao/reid-strong-baseline), Backbone为ResNet50, 主要特色为模型结构简单。
本跨镜跟踪中所用REID模型在上述基础上，整合多个开源数据集并压缩模型特征到128维以提升泛化性能。大幅提升了在实际应用中的泛化效果。
### 其他建议
- 提供的REID模型基于开源数据集训练得到，建议加入自有数据，训练更加强有力的REID模型，将非常明显提升跨镜跟踪效果。
- 质量评估部分基于简单逻辑+OpenCV实现，效果有限，如果有条件建议针对性训练质量判断模型。
### 示例效果
- camera 1:
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205595795-fd859feb-8218-450f-a109-91c27713a662.gif"/>
</div>
- camera 2:
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205595826-18ab5f0e-a572-4950-a502-96e6eb904a1e.gif"/>
</div>
## 参考文献
```
@InProceedings{Luo_2019_CVPR_Workshops,
author = {Luo, Hao and Gu, Youzhi and Liao, Xingyu and Lai, Shenqi and Jiang, Wei},
title = {Bag of Tricks and a Strong Baseline for Deep Person Re-Identification},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2019}
}
@ARTICLE{Luo_2019_Strong_TMM,
author={H. {Luo} and W. {Jiang} and Y. {Gu} and F. {Liu} and X. {Liao} and S. {Lai} and J. {Gu}},
journal={IEEE Transactions on Multimedia},
title={A Strong Baseline and Batch Normalization Neck for Deep Person Re-identification},
year={2019},
pages={1-1},
doi={10.1109/TMM.2019.2958756},
ISSN={1941-0077},
}
```

93
paddle_detection/deploy/pipeline/docs/tutorials/pphuman_mtmct_en.md Normal file
View File

@@ -0,0 +1,93 @@
English | [简体中文](pphuman_mtmct.md)
# Multi-Target Multi-Camera Tracking Module of PP-Human
Multi-target multi-camera tracking, or MTMCT, matches the identity of a person in different cameras based on the single-camera tracking. MTMCT is usually applied to the security system and the smart retailing.
The MTMCT module of PP-Human aims to provide a multi-target multi-camera pipleline which is simple, and efficient.
## How to Use
1. Download [REID model](https://bj.bcebos.com/v1/paddledet/models/pipeline/reid_model.zip) and unzip it to ```./output_inference```. For the MOT model, please refer to [mot description](./pphuman_mot.md).
2. In the MTMCT mode, input videos are required to be put in the same directory. set the REID "enable: True" in the infer_cfg_pphuman.yml. The command line is:
```python
python3 deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_pphuman.yml --video_dir=[your_video_file_directory] --device=gpu
```
3. Configuration can be modified in `./deploy/pipeline/config/infer_cfg_pphuman.yml`.
```python
python3 deploy/pipeline/pipeline.py
--config deploy/pipeline/config/infer_cfg_pphuman.yml -o REID.model_dir=reid_best/
--video_dir=[your_video_file_directory]
--device=gpu
```
## Intorduction to the Solution
MTMCT module consists of the multi-target multi-camera tracking pipeline and the REID model.
1. Multi-Target Multi-Camera Tracking Pipeline
```
single-camera tracking[id+bbox]
│
capture the target in the original image according to bbox——│
│ │
REID model quality assessment (covered or not, complete or not, brightness, etc.)
│ │
[feature] [quality]
│ │
datacollector—————│
│
sort out and filter features
│
calculate the similarity of IDs in the videos
│
make the IDs cluster together and rearrange them
```
2. The model solution is [reid-strong-baseline](https://github.com/michuanhaohao/reid-strong-baseline), with ResNet50 as the backbone.
Under the above circumstances, the REID model used in MTMCT integrates open-source datasets and compresses model features to 128-dimensional features to optimize the generalization. In this way, the actual generalization result becomes much better.
### Other Suggestions
- The provided REID model is obtained from open-source dataset training. It is recommended to add your own data to get a more powerful REID model, notably improving the MTMCT effect.
- The quality assessment is based on simple logic +OpenCV, whose effect is limited. If possible, it is advisable to conduct specific training on the quality assessment model.
### Example
- camera 1:
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205595795-fd859feb-8218-450f-a109-91c27713a662.gif"/>
</div>
- camera 2:
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205595826-18ab5f0e-a572-4950-a502-96e6eb904a1e.gif"/>
</div>
## Reference
```
@InProceedings{Luo_2019_CVPR_Workshops,
author = {Luo, Hao and Gu, Youzhi and Liao, Xingyu and Lai, Shenqi and Jiang, Wei},
title = {Bag of Tricks and a Strong Baseline for Deep Person Re-Identification},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2019}
}
@ARTICLE{Luo_2019_Strong_TMM,
author={H. {Luo} and W. {Jiang} and Y. {Gu} and F. {Liu} and X. {Liao} and S. {Lai} and J. {Gu}},
journal={IEEE Transactions on Multimedia},
title={A Strong Baseline and Batch Normalization Neck for Deep Person Re-identification},
year={2019},
pages={1-1},
doi={10.1109/TMM.2019.2958756},
ISSN={1941-0077},
}
```

117
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_attribute.md Normal file
View File

@@ -0,0 +1,117 @@
[English](ppvehicle_attribute_en.md) | 简体中文
# PP-Vehicle属性识别模块
车辆属性识别在智慧城市，智慧交通等方向具有广泛应用。在PP-Vehicle中，集成了车辆属性识别模块，可识别车辆颜色及车型属性的识别。
| 任务 | 算法 | 精度 | 预测速度 | 下载链接|
|-----------|------|-----------|----------|---------------|
| 车辆检测/跟踪 | PP-YOLOE | mAP 63.9 | 38.67ms | [预测部署模型](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| 车辆属性识别 | PPLCNet | 90.81 | 7.31 ms | [预测部署模型](https://bj.bcebos.com/v1/paddledet/models/pipeline/vehicle_attribute_model.zip) |
注意：
1. 属性模型预测速度是基于NVIDIA T4, 开启TensorRT FP16得到。模型预测速度包含数据预处理、模型预测、后处理部分。
2. 关于PP-LCNet的介绍可以参考[PP-LCNet](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/zh_CN/models/PP-LCNet.md)介绍，相关论文可以查阅[PP-LCNet paper](https://arxiv.org/abs/2109.15099)。
3. 属性模型的训练和精度测试均基于[VeRi数据集](https://www.v7labs.com/open-datasets/veri-dataset)。
- 当前提供的预训练模型支持识别10种车辆颜色及9种车型，同VeRi数据集，具体如下：
```yaml
# 车辆颜色
- "yellow"
- "orange"
- "green"
- "gray"
- "red"
- "blue"
- "white"
- "golden"
- "brown"
- "black"
# 车型
- "sedan"
- "suv"
- "van"
- "hatchback"
- "mpv"
- "pickup"
- "bus"
- "truck"
- "estate"
```
## 使用方法
### 配置项说明
[配置文件](../../config/infer_cfg_ppvehicle.yml)中与属性相关的参数如下：
```
VEHICLE_ATTR:
model_dir: output_inference/vehicle_attribute_infer/ # 车辆属性模型调用路径
batch_size: 8 # 模型预测时的batch_size大小
color_threshold: 0.5 # 颜色属性阈值，需要置信度达到此阈值才会确定具体颜色，否则为'Unknown‘
type_threshold: 0.5 # 车型属性阈值，需要置信度达到此阈值才会确定具体属性，否则为'Unknown‘
enable: False # 是否开启该功能
```
### 使用命令
1. 从模型库下载`车辆检测/跟踪`, `车辆属性识别`两个预测部署模型并解压到`./output_inference`路径下；默认会自动下载模型，如果手动下载，需要修改模型文件夹为模型存放路径。
2. 修改配置文件中`VEHICLE_ATTR`项的`enable: True`，以启用该功能。
3. 图片输入时，启动命令如下(更多命令参数说明，请参考[快速开始-参数说明](./PPVehicle_QUICK_STARTED.md))：
```bash
# 预测单张图片文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_file=test_image.jpg \
--device=gpu
# 预测包含一张或多张图片的文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_dir=images/ \
--device=gpu
```
4. 视频输入时，启动命令如下：
```bash
#预测单个视频文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu
#预测包含一个或多个视频的文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_dir=test_videos/ \
--device=gpu
```
5. 若修改模型路径，有以下两种方式：
- 方法一：`./deploy/pipeline/config/infer_cfg_ppvehicle.yml`下可以配置不同模型路径，属性识别模型修改`VEHICLE_ATTR`字段下配置
- 方法二：直接在命令行中增加`-o`，以覆盖配置文件中的默认模型路径：
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
-o VEHICLE_ATTR.model_dir=output_inference/vehicle_attribute_infer
```
测试效果如下：
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205599146-56abd72f-6e0a-4a21-bd11-f8bb421f2887.gif"/>
</div>
## 方案说明
车辆属性识别模型使用了[PaddleClas](https://github.com/PaddlePaddle/PaddleClas) 的超轻量图像分类方案(PULC，Practical Ultra Lightweight image Classification)。关于该模型的数据准备、训练、测试等详细内容，请见[PULC 车辆属性识别模型](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/zh_CN/PULC/PULC_vehicle_attribute.md).
车辆属性识别模型选用了轻量级、高精度的PPLCNet。并在该模型的基础上，进一步使用了以下优化方案：
- 使用SSLD预训练模型，在不改变推理速度的前提下，精度可以提升约0.5个百分点
- 融合EDA数据增强策略，精度可以再提升0.52个百分点
- 使用SKL-UGI知识蒸馏, 精度可以继续提升0.23个百分点

121
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_attribute_en.md Normal file
View File

@@ -0,0 +1,121 @@
English | [简体中文](ppvehicle_attribute.md)
# Attribute Recognition Module of PP-Vehicle
Vehicle attribute recognition is widely used in smart cities, smart transportation and other scenarios. In PP-Vehicle, a vehicle attribute recognition module is integrated, which can identify vehicle color and model.
| Task | Algorithm | Precision | Inference Speed | Download |
|-----------|------|-----------|----------|---------------------|
| Vehicle Detection/Tracking | PP-YOLOE | mAP 63.9 | 38.67ms | [Inference and Deployment Model](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| Vehicle Attribute Recognition | PPLCNet | 90.81 | 7.31 ms | [Inference and Deployment Model](https://bj.bcebos.com/v1/paddledet/models/pipeline/vehicle_attribute_model.zip) |
Note:
1. The inference speed of the attribute model is obtained from the test on NVIDIA T4, with TensorRT FP16. The time includes data pre-process, model inference and post-process.
2. For introductions, please refer to [PP-LCNet Series](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/en/models/PP-LCNet_en.md). Related paper is available on PP-LCNet paper
3. The training and test phase of vehicle attribute recognition model are both obtained from [VeRi dataset](https://www.v7labs.com/open-datasets/veri-dataset).
- The provided pre-trained model supports 10 colors and 9 models, which is the same with VeRi dataset. The details are as follows:
```yaml
# Vehicle Colors
- "yellow"
- "orange"
- "green"
- "gray"
- "red"
- "blue"
- "white"
- "golden"
- "brown"
- "black"
# Vehicle Models
- "sedan"
- "suv"
- "van"
- "hatchback"
- "mpv"
- "pickup"
- "bus"
- "truck"
- "estate"
```
## Instructions
### Description of Configuration
Parameters related to vehicle attribute recognition in the [config file](../../config/infer_cfg_ppvehicle.yml) are as follows:
```yaml
VEHICLE_ATTR:
model_dir: output_inference/vehicle_attribute_infer/ # Path of the model
batch_size: 8 # The size of the inference batch
color_threshold: 0.5 # Threshold of color. Confidence is required to reach this threshold to determine the specific attribute, otherwise it will be 'Unknown‘.
type_threshold: 0.5 # Threshold of vehicle model. Confidence is required to reach this threshold to determine the specific attribute, otherwise it will be 'Unknown‘.
enable: False # Whether to enable this function
```
### How to Use
1. Download models `Vehicle Detection/Tracking` and `Vehicle Attribute Recognition` from the links in `Model Zoo` and unzip them to ```./output_inference```. The models are automatically downloaded by default. If you download them manually, you need to modify the `model_dir` as the model storage path to use this function.
2. Set the "enable: True" of `VEHICLE_ATTR` in infer_cfg_ppvehicle.yml.
3. For image input, please run these commands. (Description of more parameters, please refer to [QUICK_STARTED - Parameter_Description](./PPVehicle_QUICK_STARTED.md).
```bash
# For single image
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_file=test_image.jpg \
--device=gpu
# For folder contains one or multiple images
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_dir=images/ \
--device=gpu
```
4. For video input, please run these commands.
```bash
# For single video
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu
# For folder contains one or multiple videos
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_dir=test_videos/ \
--device=gpu
```
5. There are two ways to modify the model path:
- Method 1：Set paths of each model in `./deploy/pipeline/config/infer_cfg_ppvehicle.yml`. For vehicle attribute recognition, the path should be modified under the `VEHICLE_ATTR` field.
- Method 2: Directly add `-o` in command line to override the default model path in the configuration file:
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
-o VEHICLE_ATTR.model_dir=output_inference/vehicle_attribute_infer
```
The result is shown as follow:
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205599146-56abd72f-6e0a-4a21-bd11-f8bb421f2887.gif"/>
</div>
### Features to the Solution
The vehicle attribute recognition model adopts PULC, Practical Ultra Lightweight image Classification from [PaddleClas](https://github.com/PaddlePaddle/PaddleClas). For details on data preparation, training, and testing of the model, please refer to [PULC Recognition Model of Vehicle Attribute](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/en/PULC/PULC_vehicle_attribute_en.md).
The vehicle attribute recognition model adopts the lightweight and high-precision PPLCNet. And on top of PPLCNet, our model optimized via::
- Improved about 0.5 percentage points accuracy by using the SSLD pre-trained model without changing the inference speed.
- Improved 0.52 percentage points accuracy further by integrating EDA data augmentation strategy.
- Improved 0.23 percentage points accuracy by using SKL-UGI knowledge distillation.

92
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_illegal_parking.md Normal file
View File

@@ -0,0 +1,92 @@
# PP-Vehicle违章停车识别模块
禁停区域违章停车识别在车辆应用场景中有着非常广泛的应用，借助AI的力量可以减轻人力投入，精准快速识别出违停车辆并进一步采取如广播驱离行为。PP-Vehicle中基于车辆跟踪模型、车牌检测模型和车牌识别模型实现了违章停车识别功能，具体模型信息如下：
| 任务 | 算法 | 精度 | 预测速度(ms) |预测模型下载链接 |
|:---------------------|:---------:|:------:|:------:| :---------------------------------------------------------------------------------: |
| 车辆检测/跟踪 | PP-YOLOE-l | mAP: 63.9 | - |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| 车牌检测模型 | ch_PP-OCRv3_det | hmean: 0.979 | - | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz) |
| 车牌识别模型 | ch_PP-OCRv3_rec | acc: 0.773 | - | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz) |
1. 跟踪模型使用PPVehicle数据集（整合了BDD100K-MOT和UA-DETRAC），是将BDD100K-MOT中的car, truck, bus, van和UA-DETRAC中的car, bus, van都合并为1类vehicle(1)后的数据集。
2. 车牌检测、识别模型使用PP-OCRv3模型在CCPD2019、CCPD2020混合车牌数据集上fine-tune得到。
## 使用方法
1. 用户可从上表链接中下载模型并解压到```PaddleDetection/output_inference```路径下，并修改配置文件中模型路径，也可默认自动下载模型。在```deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml```中可手动设置三个模型的模型路径。
`infer_cfg_illegal_parking.yml`中配置项说明：
```
MOT: # 跟踪模块
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip # 跟踪模型路径
tracker_config: deploy/pipeline/config/tracker_config.yml # 跟踪配置文件路径
batch_size: 1 # 跟踪batch size
enable: True # 是否开启跟踪功能
VEHICLE_PLATE: # 车牌识别模块
det_model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz # 车牌检测模型路径
det_limit_side_len: 480 # 检测模型单边输入尺寸
det_limit_type: "max" # 检测模型输入尺寸长短边选择，"max"表示长边
rec_model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz # 车牌识别模型路径
rec_image_shape: [3, 48, 320] # 车牌识别模型输入尺寸
rec_batch_num: 6 # 车牌识别batch size
word_dict_path: deploy/pipeline/ppvehicle/rec_word_dict.txt # OCR模型查询字典
enable: True # 是否开启车牌识别功能
```
2. 输入视频，启动命令如下
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml \
--video_file=test_video.mp4 \
--device=gpu \
--draw_center_traj \
--illegal_parking_time=5 \
--region_type=custom \
--region_polygon 100 1000 1000 1000 900 1700 0 1700
```
参数说明如下：
- config：配置文件路径；
- video_file：测试视频路径；
- device：推理设备配置；
- draw_center_traj：画出车辆中心运动轨迹；
- illegal_parking_time：非法停车时间，单位为秒；
- region_type：非法停车区域类型，custom表示自定义；
- region_polygon：自定义非法停车多边形，至少为3个点。
**注意:**
- 违章停车的测试视频必须是静止摄像头拍摄的，镜头不能抖动或移动。
- 判断车辆是否在违停区域内是**以车辆的中心点**作为参考，车辆擦边而过等场景不算作违章停车。
- `--region_polygon`表示用户自定义区域的多边形的点坐标序列，每两个为一对点坐标(x,y)，**按顺时针顺序**连成一个**封闭区域**，至少需要3对点也即6个整数，默认值是`[]`，需要用户自行设置点坐标，如是四边形区域，坐标顺序是`左上、右上、右下、左下`。用户可以运行[此段代码](../../tools/get_video_info.py)获取所测视频的分辨率帧数，以及可以自定义画出自己想要的多边形区域的可视化并自己调整。
自定义多边形区域的可视化代码运行如下：
```python
python get_video_info.py --video_file=demo.mp4 --region_polygon 200 200 400 200 300 400 100 400
```
快速画出想要的区域的小技巧：先任意取点得到图片，用画图工具打开，鼠标放到想要的区域点上会显示出坐标，记录下来并取整，作为这段可视化代码的region_polygon参数，并再次运行可视化，微调点坐标参数直至满意。
3. 若修改模型路径，有以下两种方式：
- 方法一：```./deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml```下可以配置不同模型路径；
- 方法二：命令行中--config配置项后面增加`-o VEHICLE_PLATE.det_model_dir=[YOUR_DETMODEL_PATH] VEHICLE_PLATE.rec_model_dir=[YOUR_RECMODEL_PATH]`修改模型路径。
测试效果如下：
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205598624-bcf5165c-990c-4fe4-8cde-eb1d45298d8f.gif"/>
</div>
可视化视频中左上角num后面的数值表示当前帧中车辆的数目；Total count表示画面中出现的车辆的总数，包括出现又消失的车辆。
## 方案说明
1. 目标检测/多目标跟踪获取图片/视频输入中的车辆检测框，模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../configs/ppyoloe/README_cn.md)
2. 基于跟踪算法获取每辆车的轨迹，如果车辆中心在违停区域内且在指定时间内未发生移动，则视为违章停车；
3. 使用车牌识别模型得到违章停车车牌并可视化。
## 参考资料
1. PaddeDetection特色检测模型[PP-YOLOE](../../../../configs/ppyoloe)。
2. Paddle字符识别模型库[PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)。

80
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_illegal_parking_en.md Normal file
View File

@@ -0,0 +1,80 @@
# PP-Vehicle Illegal Parking Recognition Module
Illegal parking recognition in no-parking areas has a very wide range of applications in vehicle application scenarios. With the help of AI, human input can be reduced, and illegally parked vehicles can be accurately and quickly identified, and further behaviors such as broadcasting to expel the vehicles can be performed. Based on the vehicle tracking model, license plate detection model and license plate recognition model, the PP-Vehicle realizes the illegal parking recognition function. The specific model information is as follows:
| Task | Algorithm | Precision | Inference Speed(ms) |Inference Model Download Link |
|:---------------------|:---------:|:------:|:------:| :---------------------------------------------------------------------------------: |
| Vehicle Tracking | PP-YOLOE-l | mAP: 63.9 | - |[Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| Plate Detection | ch_PP-OCRv3_det | hmean: 0.979 | - | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz) |
| Plate Recognition | ch_PP-OCRv3_rec | acc: 0.773 | - | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz) |
1. The tracking model uses the PPVehicle dataset (integrating BDD100K-MOT and UA-DETRAC), which combines car, truck, bus, van in BDD100K-MOT and car, bus, and van in UA-DETRAC into one class which named vehicle (1).
2. The license plate detection and recognition model is fine-tuned on the CCPD2019 and CCPD2020 using the PP-OCRv3 model.
## Instructions
1. Users can download the model from the link in the table above and unzip it to the ``PaddleDetection/output_inference``` path, and modify the model path in the configuration file, or download the model automatically by default. The model paths for the three models can be manually set in ``deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml```.
Description of configuration items in `infer_cfg_illegal_parking.yml`:
```
MOT: # Tracking Module
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip # Path of Tracking Model
tracker_config: deploy/pipeline/config/tracker_config.yml # Config Path of Tracking
batch_size: 1 # Tracking batch size
enable: True # Whether to Enable Tracking Function
VEHICLE_PLATE: # Plate Recognition Module
det_model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz # Path of Plate Detection Model
det_limit_side_len: 480 # Single Side Size of Detection Model
det_limit_type: "max" # Detection model Input Size Selection of Long and Short Sides, "max" Represents the Long Side
rec_model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz # Path of Plate Recognition Model
rec_image_shape: [3, 48, 320] # The Input Size of Plate Recognition Model
rec_batch_num: 6 # Plate Recognition batch size
word_dict_path: deploy/pipeline/ppvehicle/rec_word_dict.txt # OCR Model Look-up Table
enable: True # Whether to Enable Plate Recognition Function
```
2. Input video, the command is as follows:
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml \
--video_file=test_video.mp4 \
--device=gpu \
--draw_center_traj \
--illegal_parking_time=5 \
--region_type=custom \
--region_polygon 100 1000 1000 1000 900 1700 0 1700
The parameter description:
- config: config path;
- video_file: video path to be tested;
- device: device to infe;
- draw_center_traj: draw the trajectory of the center of the vehicle;
- illegal_parking_time: illegal parking time, in seconds;
- region_type: illegal parking region type, 'custom' means the region is customized;
- region_polygon: customized illegal parking region which includes three points at least.
3. Methods to modify the path of model:
- Method 1: Configure different model paths in ```./deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml``` file;
- Method2: In the command line, add `-o VEHICLE_PLATE.det_model_dir=[YOUR_DETMODEL_PATH] VEHICLE_PLATE.rec_model_dir=[YOUR_RECMODEL_PATH]` after the --config configuration item to modify the model path.
Test Result:
<div width="600" align="center">
<img src="https://user-images.githubusercontent.com/22989727/205598624-bcf5165c-990c-4fe4-8cde-eb1d45298d8f.gif"/>
</div>
## Method Description
1. Target multi-target tracking obtains the vehicle detection frame in the picture/video input. The model scheme is PP-YOLOE. For detailed documentation, refer to [PP-YOLOE](../../../configs/ppyoloe/README_cn. md)
2. Obtain the trajectory of each vehicle based on the tracking algorithm. If the center of the vehicle is in the illegal parking area and does not move within the specified time, it is considered illegal parking;
3. Use the license plate recognition model to get the illegal parking license plate and visualize it.
## References
1. Detection Model in PaddeDetection:[PP-YOLOE](../../../../configs/ppyoloe).
2. Character Recognition Model Library in Paddle: [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR).

120
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_mot.md Normal file
View File

@@ -0,0 +1,120 @@
[English](ppvehicle_mot_en.md) | 简体中文
# PP-Vehicle车辆跟踪模块
【应用介绍】
车辆检测与跟踪在交通监控、自动驾驶等方向都具有广泛应用，PP-Vehicle中集成了检测跟踪模块，是车牌检测、车辆属性识别等任务的基础。我们提供了预训练模型，用户可以直接下载使用。
【模型下载】
| 任务 | 算法 | 精度 | 预测速度(ms) |下载链接 |
|:---------------------|:---------:|:------:|:------:| :---------------------------------------------------------------------------------: |
| 车辆检测/跟踪 | PP-YOLOE-l | mAP: 63.9 <br> MOTA: 50.1 | 检测: 25.1ms <br> 跟踪：31.8ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| 车辆检测/跟踪 | PP-YOLOE-s | mAP: 61.3 <br> MOTA: 46.8 | 检测: 16.2ms <br> 跟踪：21.0ms | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_ppvehicle.zip) |
1. 检测/跟踪模型精度为PPVehicle数据集训练得到，整合了BDD100K-MOT和UA-DETRAC，是将BDD100K-MOT中的`car, truck, bus, van`和UA-DETRAC中的`car, bus, van`都合并为1类`vehicle(1)`后的数据集，检测精度mAP是PPVehicle的验证集上测得，跟踪精度MOTA是在BDD100K-MOT的验证集上测得(`car, truck, bus, van`合并为1类`vehicle`)。训练具体流程请参照[ppvehicle](../../../../configs/ppvehicle)。
2. 预测速度为T4 机器上使用TensorRT FP16时的速度, 速度包含数据预处理、模型预测、后处理全流程。
## 使用方法
【配置项说明】
配置文件中与属性相关的参数如下：
```
DET:
model_dir: output_inference/mot_ppyoloe_l_36e_ppvehicle/ # 车辆检测模型调用路径
batch_size: 1 # 模型预测时的batch_size大小
MOT:
model_dir: output_inference/mot_ppyoloe_l_36e_ppvehicle/ # 车辆跟踪模型调用路径
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1 # 模型预测时的batch_size大小, 跟踪任务只能设置为1
skip_frame_num: -1 # 跳帧预测的帧数，-1表示不进行跳帧，建议跳帧帧数最大不超过3
enable: False # 是否开启该功能，使用跟踪前必须确保设置为True
```
【使用命令】
1. 从上表链接中下载模型并解压到```./output_inference```路径下，并修改配置文件中模型路径。默认为自动下载模型，无需做改动。
2. 图片输入时，是纯检测任务，启动命令如下
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_file=test_image.jpg \
--device=gpu
```
3. 视频输入时，是跟踪任务，注意首先设置infer_cfg_ppvehicle.yml中的MOT配置的`enable=True`，如果希望跳帧加速检测跟踪流程，可以设置`skip_frame_num: 2`，建议跳帧帧数最大不超过3：
```
MOT:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1
skip_frame_num: 2
enable: True
```
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. 若修改模型路径，有以下两种方式：
- 方法一：```./deploy/pipeline/config/infer_cfg_ppvehicle.yml```下可以配置不同模型路径，检测和跟踪模型分别对应`DET`和`MOT`字段，修改对应字段下的路径为实际期望的路径即可。
- 方法二：命令行中--config配置项后面增加`-o MOT.model_dir=[YOUR_DETMODEL_PATH]`修改模型路径。
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
--region_type=horizontal \
--do_entrance_counting \
--draw_center_traj \
-o MOT.model_dir=ppyoloe/
```
**注意:**
- `--do_entrance_counting`表示是否统计出入口流量，不设置即默认为False。
- `--draw_center_traj`表示是否绘制跟踪轨迹，不设置即默认为False。注意绘制跟踪轨迹的测试视频最好是静止摄像头拍摄的。
- `--region_type`表示流量计数的区域，当设置`--do_entrance_counting`时可选择`horizontal`或者`vertical`，默认是`horizontal`，表示以视频图片的中心水平线为出入口，同一物体框的中心点在相邻两秒内分别在区域中心水平线的两侧，即完成计数加一。
5. 区域闯入判断和计数
注意首先设置infer_cfg_ppvehicle.yml中的MOT配置的enable=True，然后启动命令如下
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
--draw_center_traj \
--do_break_in_counting \
--region_type=custom \
--region_polygon 200 200 400 200 300 400 100 400
```
**注意:**
- 区域闯入的测试视频必须是静止摄像头拍摄的，镜头不能抖动或移动。
- `--do_break_in_counting`表示是否进行区域出入后计数，不设置即默认为False。
- `--region_type`表示流量计数的区域，当设置`--do_break_in_counting`时仅可选择`custom`，默认是`custom`，表示以用户自定义区域为出入口，同一物体框的下边界中点坐标在相邻两秒内从区域外到区域内，即完成计数加一。
- `--region_polygon`表示用户自定义区域的多边形的点坐标序列，每两个为一对点坐标(x,y)，**按顺时针顺序**连成一个**封闭区域**，至少需要3对点也即6个整数，默认值是`[]`，需要用户自行设置点坐标，如是四边形区域，坐标顺序是`左上、右上、右下、左下`。用户可以运行[此段代码](../../tools/get_video_info.py)获取所测视频的分辨率帧数，以及可以自定义画出自己想要的多边形区域的可视化并自己调整。
自定义多边形区域的可视化代码运行如下：
```python
python get_video_info.py --video_file=demo.mp4 --region_polygon 200 200 400 200 300 400 100 400
```
快速画出想要的区域的小技巧：先任意取点得到图片，用画图工具打开，鼠标放到想要的区域点上会显示出坐标，记录下来并取整，作为这段可视化代码的region_polygon参数，并再次运行可视化，微调点坐标参数直至满意。
【效果展示】
<div width="600" align="center">
<img src="../images/mot_vehicle.gif"/>
</div>
## 方案说明
【实现方案及特色】
1. 使用目标检测/多目标跟踪技术来获取图片/视频输入中的车辆检测框，检测模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../../configs/ppyoloe)和[ppvehicle](../../../../configs/ppvehicle)。
2. 多目标跟踪模型方案采用[OC-SORT](https://arxiv.org/pdf/2203.14360.pdf)，采用PP-YOLOE替换原文的YOLOX作为检测器，采用OCSORTTracker作为跟踪器，详细文档参考[OC-SORT](../../../../configs/mot/ocsort)。
## 参考文献
```
@article{cao2022observation,
title={Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking},
author={Cao, Jinkun and Weng, Xinshuo and Khirodkar, Rawal and Pang, Jiangmiao and Kitani, Kris},
journal={arXiv preprint arXiv:2203.14360},
year={2022}
}
```

143
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_mot_en.md Normal file
View File

@@ -0,0 +1,143 @@
English | [简体中文](ppvehicle_mot.md)
# PP-Vehicle Vehicle Tracking Module
【Application Introduction】
Vehicle detection and tracking are widely used in traffic monitoring and autonomous driving. The detection and tracking module is integrated in PP-Vehicle, providing a solid foundation for tasks including license plate detection and vehicle attribute recognition. We provide pre-trained models that can be directly used by developers.
【Model Download】
| Task | Algorithm | Accuracy | Inference speed(ms) | Download Link |
| -------------------------- | ---------- | ----------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------ |
| Vehicle Detection/Tracking | PP-YOLOE-l | mAP: 63.9<br>MOTA: 50.1 | Detection: 25.1ms<br>Tracking：31.8ms | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| Vehicle Detection/Tracking | PP-YOLOE-s | mAP: 61.3<br>MOTA: 46.8 | Detection: 16.2ms<br>Tracking：21.0ms | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_ppvehicle.zip) |
1. The detection/tracking model uses the PPVehicle dataset ( which integrates BDD100K-MOT and UA-DETRAC). The dataset merged car, truck, bus, van from BDD100K-MOT and car, bus, van from UA-DETRAC all into 1 class vehicle(1). The detection accuracy mAP was tested on the test set of PPVehicle, and the tracking accuracy MOTA was obtained on the test set of BDD100K-MOT (`car, truck, bus, van` were combined into 1 class `vehicle`). For more details about the training procedure, please refer to [ppvehicle](../../../../configs/ppvehicle).
2. Inference speed is obtained at T4 with TensorRT FP16 enabled, which includes data pre-processing, model inference and post-processing.
## How To Use
【Config】
The parameters associated with the attributes in the configuration file are as follows.
```
DET:
model_dir: output_inference/mot_ppyoloe_l_36e_ppvehicle/ # Vehicle detection model path
batch_size: 1 # Batch_size size for model inference
MOT:
model_dir: output_inference/mot_ppyoloe_l_36e_ppvehicle/ # Vehicle tracking model path
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1 # Batch_size size for model inference, 1 only for tracking task.
skip_frame_num: -1 # Number of frames to skip, -1 means no skipping, the maximum skipped frames are recommended to be 3
enable: False # Whether or not to enable this function, please make sure it is set to True before tracking
```
【Usage】
1. Download the model from the link in the table above and unzip it to ``. /output_inference`` and change the model path in the configuration file. The default is to download the model automatically, no changes are needed.
2. The image input will start a pure detection task, and the start command is as follows
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_file=test_image.jpg \
--device=gpu
```
3. Video input will start a tracking task. Please set `enable=True` for the MOT configuration in infer_cfg_ppvehicle.yml. If skip frames are needed for faster detection and tracking, it is recommended to set `skip_frame_num: 2` the maximum should not exceed 3.
```
MOT:
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip
tracker_config: deploy/pipeline/config/tracker_config.yml
batch_size: 1
skip_frame_num: 2
enable: True
```
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu
```
4. There are two ways to modify the model path
- Config different model path in```./deploy/pipeline/config/infer_cfg_ppvehicle.yml```. The detection and tracking models correspond to the `DET` and `MOT` fields respectively. Modify the path under the corresponding field to the actual path.
- **[Recommand]** Add`-o MOT.model_dir=[YOUR_DETMODEL_PATH]` after the config in the command line to modify model path
```python
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
--region_type=horizontal \
--do_entrance_counting \
--draw_center_traj \
-o MOT.model_dir=ppyoloe/
```
**Note:**
- `--do_entrance_counting` : Whether to count entrance/exit traffic flows, the default is False
- `--draw_center_traj` : Whether to draw center trajectory, the default is False. Its input video is preferably taken from a still camera
- `--region_type` : The region for traffic counting. When setting `--do_entrance_counting`, there are two options: `horizontal` or `vertical`. The default is `horizontal`, which means the center horizontal line of the video picture is the entrance and exit. When the center point of the same object frame is on both sides of the centre horizontal line of the region in two adjacent seconds, i.e. the count adds 1.
5. Regional break-in and counting
Please set the MOT config: enable=True in `infer_cfg_ppvehicle.yml` before running the starting command:
```
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
--draw_center_traj \
--do_break_in_counting \
--region_type=custom \
--region_polygon 200 200 400 200 300 400 100 400
```
**Note:**
- Test video of area break-ins must be taken from a still camera, with no shaky or moving footage.
- `--do_break_in_counting`Indicates whether or not to count the entrance and exit of the area. The default is False.
- `--region_type` indicates the region for traffic counting, when setting `--do_break_in_counting` only `custom` can be selected, and the default is `custom`. It means that the customized region is used as the entry and exit. When the coordinates of the lower boundary midpoint of the same object frame goes to the inside of the region within two adjacent seconds, i.e. the count adds one.
- `--region_polygon` indicates a sequence of point coordinates for a polygon in a customized region. Every two are a pair of point coordinates (x,y). **In clockwise order** they are connected into a **closed region**, at least 3 pairs of points are needed (or 6 integers). The default value is `[]`. Developers need to set the point coordinates manually. If it is a quadrilateral region, the coordinate order is `top left, top right , bottom right, bottom left`. Developers can run [this code](... /... /tools/get_video_info.py) to get the resolution frames of the predicted video. It also supports customizing and adjusting the visualisation of the polygon area.
The code for the visualisation of the customized polygon area runs as follows.
```python
python get_video_info.py --video_file=demo.mp4 --region_polygon 200 200 400 200 300 400 100 400
```
A quick tip for drawing customized area: first take any point to get the picture, open it with the drawing tool, mouse over the area point and the coordinates will be displayed, record it and round it up, use it as a region_polygon parameter for this visualisation code and run the visualisation again, and fine-tune the point coordinates parameter.
【Showcase】
<div width="600" align="center">
<img src="../images/mot_vehicle.gif"/>
</div>
## Solution
【Solution and feature】
- PP-YOLOE is adopted for vehicle detection frame of object detection, multi-object tracking in the picture/video input. For details, please refer to [PP-YOLOE](... /... /... /configs/ppyoloe/README_cn.md) and [PPVehicle](../../../../configs/ppvehicle)
- [OC-SORT](https://arxiv.org/pdf/2203.14360.pdf) is adopted as multi-object tracking model. PP-YOLOE replaced YOLOX as detector, and OCSORTTracker is the tracker. For more details, please refer to [OC-SORT](../../../../configs/mot/ocsort)
## Reference
```
@article{cao2022observation,
title={Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking},
author={Cao, Jinkun and Weng, Xinshuo and Khirodkar, Rawal and Pang, Jiangmiao and Kitani, Kris},
journal={arXiv preprint arXiv:2203.14360},
year={2022}
}
```

88
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_plate.md Normal file
View File

@@ -0,0 +1,88 @@
[English](ppvehicle_plate_en.md) | 简体中文
# PP-Vehicle车牌识别模块
车牌识别，在车辆应用场景中有着非常广泛的应用，起到车辆身份识别的作用，比如车辆出入口自动闸机。PP-Vehicle中提供了车辆的跟踪及其车牌识别的功能，并提供模型下载：
| 任务 | 算法 | 精度 | 预测速度(ms) |预测模型下载链接 |
|:---------------------|:---------:|:------:|:------:| :---------------------------------------------------------------------------------: |
| 车辆检测/跟踪 | PP-YOLOE-l | mAP: 63.9 | - |[下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| 车牌检测模型 | ch_PP-OCRv3_det | hmean: 0.979 | - | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz) |
| 车牌识别模型 | ch_PP-OCRv3_rec | acc: 0.773 | - | [下载链接](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz) |
1. 跟踪模型使用PPVehicle数据集（整合了BDD100K-MOT和UA-DETRAC），是将BDD100K-MOT中的car, truck, bus, van和UA-DETRAC中的car, bus, van都合并为1类vehicle(1)后的数据集。
2. 车牌检测、识别模型使用PP-OCRv3模型在CCPD2019、CCPD2020混合车牌数据集上fine-tune得到。
## 使用方法
1. 从上表链接中下载模型并解压到```PaddleDetection/output_inference```路径下，并修改配置文件中模型路径，也可默认自动下载模型。设置```deploy/pipeline/config/infer_cfg_ppvehicle.yml```中`VEHICLE_PLATE`的enable: True
`infer_cfg_ppvehicle.yml`中配置项说明：
```
VEHICLE_PLATE: #模块名称
det_model_dir: output_inference/ch_PP-OCRv3_det_infer/ #车牌检测模型路径
det_limit_side_len: 480 #检测模型单边输入尺寸
det_limit_type: "max" #检测模型输入尺寸长短边选择，"max"表示长边
rec_model_dir: output_inference/ch_PP-OCRv3_rec_infer/ #车牌识别模型路径
rec_image_shape: [3, 48, 320] #车牌识别模型输入尺寸
rec_batch_num: 6 #车牌识别batchsize
word_dict_path: deploy/pipeline/ppvehicle/rec_word_dict.txt #OCR模型查询字典
basemode: "idbased" #流程类型，'idbased'表示基于跟踪模型
enable: False #功能是否开启
```
2. 图片输入时，启动命令如下(更多命令参数说明，请参考[快速开始-参数说明](./PPVehicle_QUICK_STARTED.md#41-参数说明))。
```python
#单张图片
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_file=test_image.jpg \
--device=gpu \
#图片文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_dir=images/ \
--device=gpu \
```
3. 视频输入时，启动命令如下
```python
#单个视频文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
#视频文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_dir=test_videos/ \
--device=gpu \
```
4. 若修改模型路径，有以下两种方式：
- 方法一：```./deploy/pipeline/config/infer_cfg_ppvehicle.yml```下可以配置不同模型路径，车牌识别模型修改`VEHICLE_PLATE`字段下配置
- 方法二：命令行中--config配置项后面增加`-o VEHICLE_PLATE.det_model_dir=[YOUR_DETMODEL_PATH] VEHICLE_PLATE.rec_model_dir=[YOUR_RECMODEL_PATH]`修改模型路径。
测试效果如下：
<div width="600" align="center">
<img src="../images/ppvehicleplate.jpg"/>
</div>
## 方案说明
1. 目标检测/多目标跟踪获取图片/视频输入中的车辆检测框，模型方案为PP-YOLOE，详细文档参考[PP-YOLOE](../../../configs/ppyoloe/README_cn.md)
2. 通过车辆检测框的坐标在输入图像中截取每个车辆
3. 使用车牌检测模型在每张车辆截图中识别车牌所在位置，同理截取车牌区域，模型方案为PP-OCRv3_det模型，经CCPD数据集在车牌场景fine-tune得到。
4. 使用字符识别模型识别车牌中的字符。模型方案为PP-OCRv3_rec模型，经CCPD数据集在车牌场景fine-tune得到。
**性能优化措施：**
1. 使用跳帧策略，每10帧做一次车牌检测，避免每帧做车牌检测的算力消耗。
2. 车牌结果稳定策略，避免单帧结果的波动，利用同一个id的历史所有车牌识别结果进行投票，得到该id最大可能的正确结果。
## 参考资料
1. PaddeDetection特色检测模型[PP-YOLOE](../../../../configs/ppyoloe)。
2. Paddle字符识别模型库[PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)。

90
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_plate_en.md Normal file
View File

@@ -0,0 +1,90 @@
English | [简体中文](ppvehicle_plate.md)
# PP-Vehicle License Plate Recognition Modules
License plate recognition has a very wide range of applications with vehicle identification functions, such as automatic vehicle entrance/exit gates.
PP-Vehicle supports vehicle tracking and license plate recognition. Models are available for download:
| Task | Algorithm | Accuracy | Inference speed(ms) | Model Download |
|:-------------------------- |:---------------:|:------------:|:-------------------:|:------------------------------------------------------------------------------------------:|
| Vehicle Detection/Tracking | PP-YOLOE-l | mAP: 63.9 | - | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| Vehicle Detection Model | ch_PP-OCRv3_det | hmean: 0.979 | - | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz) |
| Vehicle Detection Model | ch_PP-OCRv3_rec | acc: 0.773 | - | [Link](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz) |
1. The tracking model uses the PPVehicle dataset ( which integrates BDD100K-MOT and UA-DETRAC). The dataset merged car, truck, bus, van from BDD100K-MOT and car, bus, van from UA-DETRAC all into 1 class vehicle(1).
2. License plate detection and recognition models were obtained from fine-tuned PP-OCRv3 model on the CCPD2019 and CCPD2020 mixed license plate datasets.
## How to Use
1. Download models from the above table and unzip it to```PaddleDetection/output_inference```, and modify the model path in the configuration file. Models can also be downloaded automatically by default: Set enable: True of `VEHICLE_PLATE` in `deploy/pipeline/config/infer_cfg_ppvehicle.yml`
Config Description of `infer_cfg_ppvehicle.yml`：
```
VEHICLE_PLATE: #模块名称
det_model_dir: output_inference/ch_PP-OCRv3_det_infer/ #车牌检测模型路径
det_limit_side_len: 480 #检测模型单边输入尺寸
det_limit_type: "max" #检测模型输入尺寸长短边选择，"max"表示长边
rec_model_dir: output_inference/ch_PP-OCRv3_rec_infer/ #车牌识别模型路径
rec_image_shape: [3, 48, 320] #车牌识别模型输入尺寸
rec_batch_num: 6 #车牌识别batchsize
word_dict_path: deploy/pipeline/ppvehicle/rec_word_dict.txt #OCR模型查询字典
basemode: "idbased" #流程类型，'idbased'表示基于跟踪模型
enable: False #功能是否开启
```
2. For picture input, the start command is as follows (for more descriptions of the command parameters, please refer to [Quick Start - Parameter Description](. /PPVehicle_QUICK_STARTED.md#41-parameter description)).
```python
#Single image
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_file=test_image.jpg \
--device=gpu \
#Image folder
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--image_dir=images/ \
--device=gpu \
```
3. For video input, the start command is as follows
```bash
#Single video
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
#Video folder
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_dir=test_videos/ \
--device=gpu \
```
4. There are two ways to modify the model path
- Config different model path in ```./deploy/pipeline/config/infer_cfg_ppvehicle.yml```, and modify`VEHICLE_PLATE`field to config license plate recognition model modification
- **[Recommand]** Add`-o VEHICLE_PLATE.det_model_dir=[YOUR_DETMODEL_PATH] VEHICLE_PLATE.rec_model_dir=[YOUR_RECMODEL_PATH]` to config file in command line.
The test results are as follows:
<div width="600" align="center">
<img src="../images/ppvehicleplate.jpg"/>
</div>
## Solutions
1. PP-YOLOE is adopted for vehicle detection frame of object detection, multi-object tracking in the picture/video input. For details, please refer to [PP-YOLOE](... /... /... /configs/ppyoloe/README_cn.md)
2. By using the coordinates of the vehicle detection frame, each vehicle's image is intercepted in the input image
3. Use the license plate detection model to identify the location of the license plate in each vehicle screenshot as well as the license plate area. The PP-OCRv3_det model is adopted as the solution, obtained from fine-tuned CCPD dataset in terms of number plate.
4. Use a character recognition model to identify characters in a number plate. The PP-OCRv3_det model is adopted as the solution, obtained from fine-tuned CCPD dataset in terms of number plate.
**Performance optimization measures：**
1. Use a frame skipping strategy to detect license plates every 10 frames to reduce the computing workload.
2. Use the license plate result stabilization strategy to avoid the volatility of single frame results; use all historical license plate recognition results of the same id to gain the most likely result for that id.
## Reference
1. PaddeDetection featured detection model PP-YOLOE](../../../../configs/ppyoloe)。
2. Paddle OCR Model Library [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)。

115
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_press.md Normal file
View File

@@ -0,0 +1,115 @@
[English](ppvehicle_press_en.md) | 简体中文
# PP-Vehicle压实线识别模块
车辆压实线识别在智慧城市，智慧交通等方向具有广泛应用。在PP-Vehicle中，集成了车辆压实线识别模块，可识别车辆是否违章压实线。
| 任务 | 算法 | 精度 | 预测速度 | 下载链接|
|-----------|------|-----------|----------|---------------|
| 车辆检测/跟踪 | PP-YOLOE | mAP 63.9 | 38.67ms | [预测部署模型](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| 车道线识别 | PP-liteseg | mIou 32.69 | 47 ms | [预测部署模型](https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip) |
注意：
1. 车辆检测/跟踪模型预测速度是基于NVIDIA T4, 开启TensorRT FP16得到。模型预测速度包含数据预处理、模型预测、后处理部分。
2. 车辆检测/跟踪模型的训练和精度测试均基于[VeRi数据集](https://www.v7labs.com/open-datasets/veri-dataset)。
3. 车道线模型预测速度基于Tesla P40,python端预测，模型预测速度包含数据预处理、模型预测、后处理部分。
4. 车道线模型训练和精度测试均基于[BDD100K-LaneSeg](https://bdd-data.berkeley.edu/portal.html#download)和[Apollo Scape](http://apolloscape.auto/lane_segmentation.html#to_dataset_href),两个数据集车道线分割[标签](https://bj.bcebos.com/v1/paddledet/data/mot/bdd100k/lane_dataset_label.zip)
## 使用方法
### 配置项说明
[配置文件](../../config/infer_cfg_ppvehicle.yml)中与车辆压线相关的参数如下：
```
VEHICLE_PRESSING:
enable: True #是否开启功能
LANE_SEG:
lane_seg_config: deploy/pipeline/config/lane_seg_config.yml #车道线提取配置文件
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip #模型文件路径
```
[车道线配置文件](../../config/lane_seg_config.yml)中与车道线提取相关的参数如下：
```
type: PLSLaneseg #选择分割模型
PLSLaneseg:
batch_size: 1 #图片batch_size
device: gpu #选择gpu还是cpu
filter_flag: True #是否过滤水平方向道路线
horizontal_filtration_degree: 23 #过滤水平方向车道线阈值，当分割出来的车道线最大倾斜角与
#最小倾斜角差值小于阈值时，不进行过滤
horizontal_filtering_threshold: 0.25 #确定竖直方向与水平方向分开阈值
#thr = (min_degree+max_degree)*0.25
#根据车道线倾斜角与thr的大小比较，将车道线分为垂直方向与水平方向
```
### 使用命令
1. 从模型库下载`车辆检测/跟踪`, `车道线识别`两个预测部署模型并解压到`./output_inference`路径下；默认会自动下载模型，如果手动下载，需要修改模型文件夹为模型存放路径。
2. 修改配置文件中`VEHICLE_PRESSING`项的`enable: True`，以启用该功能。
3. 图片输入时，启动命令如下(更多命令参数说明，请参考[快速开始-参数说明](./PPVehicle_QUICK_STARTED.md))：
```bash
# 预测单张图片文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_PRESSING.enable=true
--image_file=test_image.jpg \
--device=gpu
# 预测包含一张或多张图片的文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_PRESSING.enable=true
--image_dir=images/ \
--device=gpu
```
4. 视频输入时，启动命令如下：
```bash
#预测单个视频文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_PRESSING.enable=true
--video_file=test_video.mp4 \
--device=gpu
#预测包含一个或多个视频的文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_dir=test_videos/ \
-o VEHICLE_PRESSING.enable=true
--device=gpu
```
5. 若修改模型路径，有以下两种方式：
- 方法一：`./deploy/pipeline/config/infer_cfg_ppvehicle.yml`下可以配置不同模型路径，车道线识别模型修改`LANE_SEG`字段下配置
- 方法二：直接在命令行中增加`-o`，以覆盖配置文件中的默认模型路径：
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
-o VEHICLE_PRESSING.enable=true
LANE_SEG.model_dir=output_inference
```
测试效果如下：
<div width="1000" align="center">
<img src="https://raw.githubusercontent.com/LokeZhou/PaddleDetection/develop/deploy/pipeline/docs/images/vehicle_press.gif"/>
</div>
## 方案说明
1.车道线识别模型使用了[PaddleSeg](https://github.com/PaddlePaddle/PaddleSeg) 的超轻量分割方案。训练样本[标签](https://bj.bcebos.com/v1/paddledet/data/mot/bdd100k/lane_dataset_label.zip)分为4类：
0 背景
1 双黄线
2 实线
3 虚线
车辆压线分析过滤虚线类；
2.车道线通过对分割结果聚类得到，且默认过滤水平方向车道线，若不过滤可在[车道线配置文件](../../config/lane_seg_config.yml)修改`filter_flag`参数；
3.车辆压线判断条件：车辆的检测框底边线与车道线是否有交点；
**性能优化措施**
1.因摄像头视角原因，可以根据实际情况决定是否过滤水平方向车道线;

115
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_press_en.md Normal file
View File

@@ -0,0 +1,115 @@
English | [简体中文](ppvehicle_press.md)
# PP-Vehicle press line identification module
Vehicle compaction line recognition is widely used in smart cities, smart transportation and other directions.
In PP-Vehicle, a vehicle compaction line identification module is integrated to identify whether the vehicle is in violation of regulations.
| task | algorithm | precision | infer speed | download|
|-----------|------|-----------|----------|---------------|
| Vehicle detection/tracking | PP-YOLOE | mAP 63.9 | 38.67ms | [infer deploy model](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| Lane line segmentation | PP-liteseg | mIou 32.69 | 47 ms | [infer deploy model](https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip) |
Notes：
1. The prediction speed of vehicle detection/tracking model is based on NVIDIA T4 and TensorRT FP16. The model prediction speed includes data preprocessing, model prediction and post-processing.
2. The training and precision test of vehicle detection/tracking model are based on [VeRi](https://www.v7labs.com/open-datasets/veri-dataset).
3. The predicted speed of lane line segmentation model is based on Tesla P40 and python prediction. The predicted speed of the model includes data preprocessing, model prediction and post-processing.
4. Lane line model training and precision testing are based on [BDD100K-LaneSeg](https://bdd-data.berkeley.edu/portal.html#download)and [Apollo Scape](http://apolloscape.auto/lane_segmentation.html#to_dataset_href),The label data of the two data sets is in[Lane_dataset_label](https://bj.bcebos.com/v1/paddledet/data/mot/bdd100k/lane_dataset_label.zip)
## Instructions
### Description of Configuration
The parameters related to vehicle line pressing in [config file](../../config/infer_cfg_ppvehicle.yml) is as follows:
```
VEHICLE_PRESSING:
enable: True #Whether to enable the funcion
LANE_SEG:
lane_seg_config: deploy/pipeline/config/lane_seg_config.yml #lane line seg config file
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip #model path
```
The parameters related to Lane line segmentation in [lane line seg config file](../../config/lane_seg_config.yml)is as follows:
```
type: PLSLaneseg #Select segmentation Model
PLSLaneseg:
batch_size: 1 #image batch_size
device: gpu #device is gpu or cpu
filter_flag: True #Whether to filter the horizontal direction road route
horizontal_filtration_degree: 23 #Filter the threshold value of the lane line in the horizontal direction. When the difference between the maximum inclination angle and the minimum inclination angle of the segmented lane line is less than the threshold value, no filtering is performed
horizontal_filtering_threshold: 0.25 #Determine the threshold value for separating the vertical direction from the horizontal direction thr=(min_degree+max_degree) * 0.25 Divide the lane line into vertical direction and horizontal direction according to the comparison between the gradient angle of the lane line and thr
```
### How to Use
1. Download 'vehicle detection/tracking' and 'lane line recognition' two prediction deployment models from the model base and unzip them to '/ output_ Invitation ` under the path; By default, the model will be downloaded automatically. If you download it manually, you need to modify the model folder as the model storage path.
2. Modify Profile ` VEHICLE_PRESSING ' -'enable: True' item to enable this function.
3. When inputting a picture, the startup command is as follows (for more command parameter descriptions,please refer to [QUICK_STARTED - Parameter_Description](./PPVehicle_QUICK_STARTED.md)
```bash
# For single image
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_PRESSING.enable=true
--image_file=test_image.jpg \
--device=gpu
# For folder contains one or multiple images
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_PRESSING.enable=true
--image_dir=images/ \
--device=gpu
```
4. For video input, please run these commands.
```bash
#For single video
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_PRESSING.enable=true
--video_file=test_video.mp4 \
--device=gpu
#For folder contains one or multiple videos
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_dir=test_videos/ \
-o VEHICLE_PRESSING.enable=true
--device=gpu
```
5. There are two ways to modify the model path:
- 1.Set paths of each model in `./deploy/pipeline/config/infer_cfg_ppvehicle.yml`,For Lane line segmentation, the path should be modified under the `LANE_SEG`
- 2.Directly add `-o` in command line to override the default model path in the configuration file:
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
-o VEHICLE_PRESSING.enable=true
LANE_SEG.model_dir=output_inference
```
The result is shown as follow:
<div width="1000" align="center">
<img src="https://raw.githubusercontent.com/LokeZhou/PaddleDetection/develop/deploy/pipeline/docs/images/vehicle_press.gif"/>
</div>
## Features to the Solution
1.Lane line recognition model uses [PaddleSeg]（ https://github.com/PaddlePaddle/PaddleSeg ）Super lightweight segmentation scheme.Train [lable](https://bj.bcebos.com/v1/paddledet/data/mot/bdd100k/lane_dataset_label.zip)it is divided into four categories:
0 Background
1 Double yellow line
2 Solid line
3 Dashed line
Lane line recognition filtering Dashed lines;
2.Lane lines are obtained by clustering segmentation results, and the horizontal lane lines are filtered by default. If not, you can modify the `filter_flag` in [lane line seg config file](../../config/lane_seg_config.yml);
3.Judgment conditions for vehicle line pressing: whether there is intersection between the bottom edge line of vehicle detection frame and lane line;
**Performance optimization measures：**
1.Due to the camera angle, it can be decided whether to filter the lane line in the horizontal direction according to the actual situation;

126
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_retrograde.md Normal file
View File

@@ -0,0 +1,126 @@
[English](ppvehicle_retrograde_en.md) | 简体中文
# PP-Vehicle车辆逆行识别模块
车辆逆行识别在智慧城市，智慧交通等方向具有广泛应用。在PP-Vehicle中，集成了车辆逆行识别模块，可识别车辆是否逆行。
| 任务 | 算法 | 精度 | 预测速度 | 下载链接|
|-----------|------|-----------|----------|---------------|
| 车辆检测/跟踪 | PP-YOLOE | mAP 63.9 | 38.67ms | [预测部署模型](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| 车道线识别 | PP-liteseg | mIou 32.69 | 47 ms | [预测部署模型](https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip) |
注意：
1. 车辆检测/跟踪模型预测速度是基于NVIDIA T4, 开启TensorRT FP16得到。模型预测速度包含数据预处理、模型预测、后处理部分。
2. 车辆检测/跟踪模型的训练和精度测试均基于[VeRi数据集](https://www.v7labs.com/open-datasets/veri-dataset)。
3. 车道线模型预测速度基于Tesla P40,python端预测，模型预测速度包含数据预处理、模型预测、后处理部分。
4. 车道线模型训练和精度测试均基于[BDD100K-LaneSeg](https://bdd-data.berkeley.edu/portal.html#download.)和[Apollo Scape](http://apolloscape.auto/lane_segmentation.html#to_dataset_href)。两个数据集的标签文件[Lane_dataset_label](https://bj.bcebos.com/v1/paddledet/data/mot/bdd100k/lane_dataset_label.zip)
## 使用方法
### 配置项说明
[配置文件](../../config/infer_cfg_ppvehicle.yml)中与车辆逆行识别模块相关的参数如下：
```
LANE_SEG:
lane_seg_config: deploy/pipeline/config/lane_seg_config.yml #车道线提取配置文件
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip #模型文件路径
VEHICLE_RETROGRADE:
frame_len: 8 #采样帧数
sample_freq: 7 #采样频率
enable: True #开启车辆逆行判断功能
filter_horizontal_flag: False #是否过滤水平方向车辆
keep_right_flag: True #按车辆靠右行驶规则，若车辆靠左行驶，则设为False
deviation: 23 #过滤水平方向车辆的角度阈值，如果大于该角度则过滤
move_scale: 0.01 #过滤静止车辆阈值，若车辆移动像素大于图片对角线*move_scale，则认为车辆移动，反之
#车辆静止
fence_line: [] #车道中间线坐标，格式[x1,y1,x2,y2] 且y2>y1。若为空，由程序根据车流方向自动判断
```
[车道线配置文件](../../config/lane_seg_config.yml)中与车道线提取相关的参数如下：
```
type: PLSLaneseg #选择分割模型
PLSLaneseg:
batch_size: 1 #图片batch_size
device: gpu #选择gpu还是cpu
filter_flag: True #是否过滤水平方向道路线
horizontal_filtration_degree: 23 #过滤水平方向车道线阈值，当分割出来的车道线最大倾斜角与
#最小倾斜角差值小于阈值时，不进行过滤
horizontal_filtering_threshold: 0.25 #确定竖直方向与水平方向分开阈值
#thr = (min_degree+max_degree)*0.25
#根据车道线倾斜角与thr的大小比较，将车道线分为垂直方向与水平方向
```
### 使用命令
1. 从模型库下载`车辆检测/跟踪`, `车道线识别`两个预测部署模型并解压到`./output_inference`路径下；默认会自动下载模型，如果手动下载，需要修改模型文件夹为模型存放路径。
2. 修改配置文件中`VEHICLE_RETROGRADE`项的`enable: True`，以启用该功能。
3. 车辆逆行识别功能需要视频输入时，启动命令如下：
```bash
#预测单个视频文件
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_RETROGRADE.enable=true \
--video_file=test_video.mp4 \
--device=gpu
#预测包含一个或多个视频的文件夹
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_RETROGRADE.enable=true \
--video_dir=test_video \
--device=gpu
```
5. 若修改模型路径，有以下两种方式：
- 方法一：`./deploy/pipeline/config/infer_cfg_ppvehicle.yml`下可以配置不同模型路径，车道线识别模型修改`LANE_SEG`字段下配置
- 方法二：直接在命令行中增加`-o`，以覆盖配置文件中的默认模型路径：
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
-o LANE_SEG.model_dir=output_inference/
VEHICLE_RETROGRADE.enable=true
```
测试效果如下：
<div width="1000" align="center">
<img src="https://raw.githubusercontent.com/LokeZhou/PaddleDetection/develop/deploy/pipeline/docs/images/vehicle_retrograde.gif"/>
</div>
**注意:**
- 车道线中间线自动判断条件：在采样的视频段内同时有两个相反方向的车辆，且判断一次后固定，不再更新；
- 因摄像头角度以及2d视角问题，车道线中间线判断存在不准确情况;
- 可在配置文件手动输入中间线坐标.参考[车辆违章配置文件](../../config/examples/infer_cfg_vehicle_violation.yml)
## 方案说明
1.车辆在采样视频段内，根据车道中间线的位置与车辆轨迹，判断车辆是否逆行，判断流程图：
<div width="1000" align="center">
<img src="https://raw.githubusercontent.com/LokeZhou/PaddleDetection/develop/deploy/pipeline/docs/images/vehicle_retrograde.png"/>
</div>
2.车道线识别模型使用了[PaddleSeg](https://github.com/PaddlePaddle/PaddleSeg) 的超轻量分割方案。训练样本[标签](https://bj.bcebos.com/v1/paddledet/data/mot/bdd100k/lane_dataset_label.zip)分为4类：
0 背景
1 双黄线
2 实线
3 虚线
车辆逆行分析过滤虚线类；
3.车道线通过对分割结果聚类得到，且默认过滤水平方向车道线，若不过滤可在[车道线配置文件](../../config/lane_seg_config.yml)修改`filter_flag`参数;
4.车辆逆行判断默认过滤水平方向车辆，若不过滤可在[配置文件](../../config/infer_cfg_ppvehicle.yml)修改`filter_horizontal_flag`参数;
5.车辆逆行默认按靠右行驶规则判断，若修改，可在[配置文件](../../config/infer_cfg_ppvehicle.yml)修改`keep_right_flag`参数;
**性能优化措施**：
1.因摄像头视角原因，可以根据实际情况决定是否过滤水平方向车道线与水平方向车辆;
2.车道中间线可手动输入；

125
paddle_detection/deploy/pipeline/docs/tutorials/ppvehicle_retrograde_en.md Normal file
View File

@@ -0,0 +1,125 @@
English | [简体中文](ppvehicle_retrograde.md)
# PP-Vehicle vehicle retrograde identification module
Vehicle reverse identification is widely used in smart cities, smart transportation and other directions. In PP-Vehicle, a vehicle retrograde identification module is integrated to identify whether the vehicle is retrograde.
| task | algorithm | precision | infer speed | download|
|-----------|------|-----------|----------|---------------|
| Vehicle detection/tracking | PP-YOLOE | mAP 63.9 | 38.67ms | [infer deploy model](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip) |
| Lane line segmentation | PP-liteseg | mIou 32.69 | 47 ms | [infer deploy model](https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip) |
Notes：
1. The prediction speed of vehicle detection/tracking model is based on NVIDIA T4 and TensorRT FP16. The model prediction speed includes data preprocessing, model prediction and post-processing.
2. The training and precision test of vehicle detection/tracking model are based on [VeRi](https://www.v7labs.com/open-datasets/veri-dataset).
3. The predicted speed of lane line segmentation model is based on Tesla P40 and python prediction. The predicted speed of the model includes data preprocessing, model prediction and post-processing.
4. Lane line model training and precision testing are based on [BDD100K-LaneSeg](https://bdd-data.berkeley.edu/portal.html#download) and [Apollo Scape](http://apolloscape.auto/lane_segmentation.html#to_dataset_href),The label data of the two data sets is in [Lane_dataset_label](https://bj.bcebos.com/v1/paddledet/data/mot/bdd100k/lane_dataset_label.zip)
## Instructions
### Description of Configuration
[The parameters related to vehicle retrograde in [config file](../../config/infer_cfg_ppvehicle.yml) is as follows:
```
LANE_SEG:
lane_seg_config: deploy/pipeline/config/lane_seg_config.yml #lane line seg config file
model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/pp_lite_stdc2_bdd100k.zip #model path
VEHICLE_RETROGRADE:
frame_len: 8 #Number of sampling frames
sample_freq: 7 #sampling frequency
enable: True #Whether to enable the funcion
filter_horizontal_flag: False #Whether to filter vehicles in horizontal direction
keep_right_flag: True #According to the right driving rule, if the vehicle keeps left driving, it is set as False
deviation: 23 #Filter the horizontal angle vehicles threshold. If it is greater than this angle, filter
move_scale: 0.01 #Filter the threshold value of stationary vehicles. If the vehicle moving pixel is greater than the image diagonal * move_scale, the vehicle is considered moving, otherwise, the vehicle is stationary
fence_line: [] #Lane centerline coordinates, format[x1,y1,x2,y2] and y2>y1. If it is empty, the program will automatically judge according to the direction of traffic flow
```
The parameters related to Lane line segmentation in [lane line seg config file](../../config/lane_seg_config.yml)is as follows:
```
type: PLSLaneseg #Select segmentation Model
PLSLaneseg:
batch_size: 1 #image batch_size
device: gpu #device is gpu or cpu
filter_flag: True #Whether to filter the horizontal direction road route
horizontal_filtration_degree: 23 #Filter the threshold value of the lane line in the horizontal direction. When the difference between the maximum inclination angle and the minimum inclination angle of the segmented lane line is less than the threshold value, no filtering is performed
horizontal_filtering_threshold: 0.25 #Determine the threshold value for separating the vertical direction from the horizontal direction thr=(min_degree+max_degree) * 0.25 Divide the lane line into vertical direction and horizontal direction according to the comparison between the gradient angle of the lane line and thr
```
### How to Use
1. Download 'vehicle detection/tracking' and 'lane line recognition' two prediction deployment models from the model base and unzip them to '/ output_ Invitation ` under the path; By default, the model will be downloaded automatically. If you download it manually, you need to modify the model folder as the model storage path.
2. Modify Profile`VEHICLE_RETROGRADE`-`enable: True`， item to enable this function.
3. When video input is required for vehicle retrograde recognition function, the starting command is as follows:
```bash
#For single video
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_RETROGRADE.enable=true \
--video_file=test_video.mp4 \
--device=gpu
#For folder contains one or multiple videos
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
-o VEHICLE_RETROGRADE.enable=true \
--video_dir=test_video \
--device=gpu
```
5. There are two ways to modify the model path:
- 1.Set paths of each model in `./deploy/pipeline/config/infer_cfg_ppvehicle.yml`,For Lane line segmentation, the path should be modified under the `LANE_SEG`
- 2.Directly add `-o` in command line to override the default model path in the configuration file:
```bash
python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml \
--video_file=test_video.mp4 \
--device=gpu \
-o LANE_SEG.model_dir=output_inference/
VEHICLE_RETROGRADE.enable=true
```
The result is shown as follow:
<div width="1000" align="center">
<img src="https://raw.githubusercontent.com/LokeZhou/PaddleDetection/develop/deploy/pipeline/docs/images/vehicle_retrograde.gif"/>
</div>
**Note:**
- Automatic judgment condition of lane line middle line: there are two vehicles in opposite directions in the sampled video segment, and the judgment is fixed after one time and will not be updated;
- Due to camera angle and 2d visual angle problems, the judgment of lane line middle line is inaccurate.
- You can manually enter the middle line coordinates in the configuration file.Example as [infer_cfg_vehicle_violation.yml](../../config/examples/infer_cfg_vehicle_violation.yml)
## Features to the Solution
1.In the sampling video segment, judge whether the vehicle is retrograde according to the location of the lane centerline and the vehicle track, and determine the flow chart:
<div width="1000" align="center">
<img src="https://raw.githubusercontent.com/LokeZhou/PaddleDetection/develop/deploy/pipeline/docs/images/vehicle_retrograde_en.png"/>
</div>
2.Lane line recognition model uses [PaddleSeg]（ https://github.com/PaddlePaddle/PaddleSeg ）Super lightweight segmentation scheme.Train [lable](https://bj.bcebos.com/v1/paddledet/data/mot/bdd100k/lane_dataset_label.zip)it is divided into four categories:
0 Background
1 Double yellow line
2 Solid line
3 Dashed line
Lane line recognition filtering Dashed lines;
3.Lane lines are obtained by clustering segmentation results, and the horizontal lane lines are filtered by default. If not, you can modify the `filter_flag` in [lane line seg config file](../../config/lane_seg_config.yml);
4.The vehicles in the horizontal direction are filtered by default when judging the vehicles in the reverse direction. If not, you can modify the `filter_horizontal_flag` in [config file](../../config/infer_cfg_ppvehicle.yml);
5.The vehicle will be judged according to the right driving rule by default.If not, you can modify the `keep_right_flag` in [config file](../../config/infer_cfg_ppvehicle.yml);
**Performance optimization measures：**
1.Due to the camera's viewing angle, it can be decided whether to filter the lane lines and vehicles in the horizontal direction according to the actual situation;
2.The lane middle line can be manually entered;