Pedestrian-Aware Panoramic Video Stitching Based on a Structured Camera Array

Anqi Zhu1, Lin Zhang*1, Juntao Chen1, Yicong Zhou2

1 School of Software Engineering, Tongji University, Shanghai, China

2 Department of Computer and Information Science, University of Macau, China

Introduction

This is the website for our paper "Pedestrian-Aware Panoramic Video Stitching Based on a Structured Camera Array".

Abstract

A panorama stitching system can help the viewer understand the surrounding environment in surveillance or space exploration. Such a system that aligns the surrounding images on a plane and fuses them naturally is necessarily called for. We solve three key issues in the existing workflow that can affect the efficiency and the quality of the obtained panoramic video and present Pedestrian360, a real-time panoramic video system based on a structured camera array. First, to get a geometrically aligned 360 degree view in the horizontal direction, we build a unified multi-camera coordinate system via a novel refinement approach that jointly optimizes camera poses. Second, to eliminate the brightness and color difference of images taken by different cameras, we design a photometric alignment approach by adding a bias to the traditional linear adjustment model and solving it with two-step least-squares. Third, considering that the HVS(Human Visual System) is more sensitive to high-level semantic objects, such as pedestrians and vehicles, we integrate the results of instance segmentation into the dynamic programming in the seam-cutting step. To our knowledge, we are the first to introduce instance segmentation to the seam-cutting problem. This can ensure the integrity of the salient objects in a panorama. In our surveillance oriented system, we choose the most significant target, pedestrians, as the seam avoidance target, and this is the reason for the name Pedestrian360. We demonstrate that Pedestrian360 performs better and have real-time performance compared to existing methods through various scenes and resolutions tests.

Appearance and Workflow

Our Pedestrian360 panorama system is based on a structured camera array, which consists of four wide-angle, small-sized 1080P cameras. Its appearance and detailed internal structure are shown in the figure below. Four fisheye cameras are mounted on a column in four directions. Each camera has 180 degree field-of-view. Adjacent cameras share some overlapped areas and four fisheye cameras together cover the whole surrounding area of the column.

The workflow of Pedestrian360 consists of three steps: calibration, cylindrical projection with photometric alignment, and seam updating. The pipeline is shown in the figure below.

Source Codes and Data

Get the code and data.

Here are some image stitching examples of Pedestrian360:

Here are video examples of Pedestrian360:

Last update: Nov. 17, 2020