Efficient Image/video Restyling and Collage on GPU.
详细信息
文摘
Image/video restyling as an expressive way for producing usercustomized appearances has received much attention in creative media researches. In interactive design,it would be powerful to re-render the stylized presentation of interested objects virtually using computer-aided design tools for retexturing,especially in the image space with a single image or video as input. The nowaday retexturing methods mostly process texture distortion by inter-pixel distance manipulation in image space,the underlying texture distortion is always destroyed due to limitations like improper distortion caused by human mesh stretching,or unavoidable texture splitting caused by texture synthesis. Image/video collage techniques are invented to allow parallel presenting of multiple objects and events on the display canvas. With the rapid development of digital video capture devices,the related issues are to quickly review and brief such large amount of visual media datasets to find out interested video materials. It will be a tedious task to investigate long boring surveillance videos and grasp the essential information quickly. By applying key information and shortened video forms as vehicles for communication,video abstraction and summary are the means to enhance the browsing efficiency and easy understanding of visual media datasets. In this thesis,we first focused our image/video restyling work on efficient retexturing and stylization. We present an interactive retexturing that preserves similar texture distortion without knowing the underlying geometry and lighting environment. We utilized SIFT corner features to naturally discover the underlying texture distortion. The gradient depth recovery and wrinkle stress optimization are applied to accomplish the distortion process. We facilitate the interactive retexturing via real-time bilateral grids and feature-guided distortion optimization using GPU-CUDA parallelism. Video retexturing is achieved through a keyframe-based texture transferring strategy using accurate TV-L1 optical flow with patch motion tracking techniques in real-time. Further,we work on GPU-based abstract stylization that preserves the fine structure in the original images using gradient optimization. We propose an image structure map to naturally distill the fine structure of the original images. Gradient-based tangent generation and tangent-guided morphology are applied to build the structure map. We facilitate the final stylization via parallel bilateral grids and structure-aware stylizing in real-time on GPU-CUDA. In the experiments,our proposed methods consistently demonstrate high quality performance of image/video abstract restyling in real-time. Currently,in video abstraction,video collages are mostly produced with static keyfame-based collage pictures,which contain limited information of dynamic videos and in uence understanding of visual media datasets greatly. We present dynamic video collage that effectively summarizes condensed dynamic activities in parallel on the canvas for easy browsing. We propose to utilize activity cuboids to reorganize and extract dynamic objects for further collaging,and video stabilization is performed to generate stabilized activity cuboids. Spatial-temporal optimization is carried out to optimize the positions of activity cuboids in the 3D collage space. We facilitate the efficient dynamic collage via event similarity and moving relationship optimization on GPU allowing multi-video inputs. Our video collage approach with kernel reordering CUDA processing enables dynamic summaries for easy browsing of long videos,while saving huge memory space for storing and transmitting them. The experiments and user study have shown the efficiency and usefulness of our dynamic video collage,which can be widely applied for video briefing and summary applications. In the future,we will further extend the interactive retexturing to more complicated general video applications with large motion and occluded scene avoiding textures flicking. We will also work on new approaches to make video retexturing more stable by inspiration from latest video processing techniques. Our future work for video collage includes investigating applications of dynamic collage into the surveillance industry,and working on moving camera and general videos,which may contain large amount of camera motions and different types of video shot transitions.