A Framework for Realtime 3-D Reconstruction by Space Carving Using Graphics Hardware

Diplomarbeit von Christian Nitschke

Introduction:

Reconstruction of real-world scenes from a set of multiple images is a topic in Computer Vision and 3D Computer Graphics with many interesting applications. There is a relation to Augmented and Mixed Reality (AR/MR), Computer-Supported Collaborative Work (CSCW), Computer-Aided industrial/architectural Design (CAD), modeling of the real-world (e.g. computer games, scenes/effects in movies), entertainment (e.g. 3D TV/Video) and recognition/analyzing of real-world characteristics by computer systems and robots.

There exists a powerful algorithm theory for shape reconstruction from arbitrary viewpoints, called shape from photo-consistency. However, it is computationally expensive and hence can not be used with applications in the field of 3D video or CSCW as well as interactive 3D model creation.

Attempts have been made to achieve real-time framerates using PC cluster systems. While these provide enough performance they are also expensive and less flexible. Approaches that use GPU hardware-acceleration on single workstations achieve interactive framerates for novel-view synthesis, but do not provide an explicit volumetric representation of the whole scene.

The proposed approach shows the efforts in developing a GPU hardware-accelerated framework for obtaining the volumetric photo hull of a dynamic 3D scene as seen from multiple calibrated cameras. High performance is achieved by employing a shape from silhouette technique in advance to obtain a tight initial volume for shape from photo-consistency.

Also several speed-up techniques are presented to increase efficiency. Since the entire processing is done on a single PC, the framework can be applied to mobile setups, enabling a wide range of further applications.

The approach is explained using programmable vertex and fragment processors and compared to highly optimized CPU implementations. It is shown that the new approach can outperform the latter by more than one magnitude.

The thesis is organized as follows:

Chapter 1 contains an introduction, giving an overview with classification of related techniques, statement of the main problem, novelty of the proposed approach and its fields of application.

Chapter 2 surveys related work in the area of dynamic scene reconstruction by shape from silhouette and shape from photo-consistency. The focus lies on high performance reconstruction and hardware-acceleration.

Chapter 3 introduces the theoretical basis for the proposed approach within three main parts. (1) Camera geometry is important to relate images captured from multiple viewpoints to an unknown 3D scene. (2) When taking a photograph of a scene, the light emitted towards the camera is captured. Therefore light and color is discussed as necessary for this work. (3) At last, the theory behind shape from silhouette and shape from photo-consistency needs to be explained.

Chapter 4 continues with defining the Basic Algorithm as a hybrid approach to scene reconstruction with introducing features like (1) a robust and fast image-segmentation to account for shadows, (2) a set of nested volumes that are sequentially computed to speed-up computation and (3) an implicit visibility computation.

The Basic Algorithm is extended and mapped onto the GPU in chapter 5. The corresponding Advanced Algorithm enables efficient real-time computation by employing a multipass-rendering strategy.

Chapter 6 explains and discusses several experiments to analyze the proposed system in terms of performance and reconstruction quality.

Chapter 7 concludes with giving a summary and discussing limitations and future works.

Due to their number, diagrams and figures relating to experiments are grouped into additional appendix chapters A, B, C and D.

Table of Contents:

Table of Contents:

Text Sample:

Chapter 2.2 Shape from Photo-Consistency:

Techniques, that reconstruct a 3D scene by shape from silhouette employ information about foreground and background pixels. The 3D shape is known as volume intersection, since it represents the 3D intersection of the backprojected 2D silhouettes. Shape from photo-consistency is a generalization of this approach with using additional greyscale or color information. Applying further constraints, these methods can achieve a tighter approximation of the true shape. The property of photo-consistency relates to the fact, that the radiance from a particular (unknown) scene point should be consistent with the irradiance measured in the corresponding image pixels from where the point is visible. For each scene point can thus be determined, if it is consistent with the set of images. This is the inverse approach of the one taken by image-based stereo algorithms. Instead of evaluating the pixels that are corresponding to a particular scene point, they start from the images and try to find pixels that may correspond to the same scene point.

The coordinates of the point are then obtained by triangulation. Due to this, shape from photo-consistency is also referred to as scene-based stereo. Similar to the term of photo-consistency, LECLERC et al. define self-consistency as a pixel correspondence measure for evaluating image-based stereo algorithms.

When comparing both approaches, image-based stereo shows several drawbacks. In particular related to the reconstruction of a global consistent scene. Also, the recovered points are arbitray located within point clouds and hence, do not correspond to a voxel representation on a regular grid. However, since image-based stereo algorithms (1) are related to the ray-based sampling employed for the proposed approach and (2) achieve real-time framerates for generation of novel views, they are considered for this review. A survey of two-view stereo algorithms is provided by SCHARSTEIN and SZELISKI.