Multidimensional image morphing-fast image-based rendering of open 3D and VR environments
Background: In recent years, the demand for interactive photorealistic three-dimensional (3D) environments has increased in various fields, including architecture, engineering, and entertainment. However, achieving a balance between the quality and efficiency of high-performance 3D applications and...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
KeAi Communications Co., Ltd.
2025-04-01
|
| Series: | Virtual Reality & Intelligent Hardware |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2096579623000384 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850183087504228352 |
|---|---|
| author | Simon Seibt Bastian Kuth Bartosz von Rymon Lipinski Thomas Chang Marc Erich Latoschik |
| author_facet | Simon Seibt Bastian Kuth Bartosz von Rymon Lipinski Thomas Chang Marc Erich Latoschik |
| author_sort | Simon Seibt |
| collection | DOAJ |
| description | Background: In recent years, the demand for interactive photorealistic three-dimensional (3D) environments has increased in various fields, including architecture, engineering, and entertainment. However, achieving a balance between the quality and efficiency of high-performance 3D applications and virtual reality (VR) remains challenging. Methods: This study addresses this issue by revisiting and extending view interpolation for image-based rendering (IBR), which enables the exploration of spacious open environments in 3D and VR. Therefore, we introduce multimorphing, a novel rendering method based on the spatial data structure of 2D image patches, called the image graph. Using this approach, novel views can be rendered with up to six degrees of freedom using only a sparse set of views. The rendering process does not require 3D reconstruction of the geometry or per-pixel depth information, and all relevant data for the output are extracted from the local morphing cells of the image graph. The detection of parallax image regions during preprocessing reduces rendering artifacts by extrapolating image patches from adjacent cells in real-time. In addition, a GPU-based solution was presented to resolve exposure inconsistencies within a dataset, enabling seamless transitions of brightness when moving between areas with varying light intensities. Results: Experiments on multiple real-world and synthetic scenes demonstrate that the presented method achieves high ''VR-compatible'' frame rates, even on mid-range and legacy hardware, respectively. While achieving adequate visual quality even for sparse datasets, it outperforms other IBR and current neural rendering approaches. Conclusions: Using the correspondence-based decomposition of input images into morphing cells of 2D image patches, multidimensional image morphing provides high-performance novel view generation, supporting open 3D and VR environments. Nevertheless, the handling of morphing artifacts in the parallax image regions remains a topic for future research. |
| format | Article |
| id | doaj-art-23ced04292d94b948be78805dd280c5d |
| institution | OA Journals |
| issn | 2096-5796 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | KeAi Communications Co., Ltd. |
| record_format | Article |
| series | Virtual Reality & Intelligent Hardware |
| spelling | doaj-art-23ced04292d94b948be78805dd280c5d2025-08-20T02:17:28ZengKeAi Communications Co., Ltd.Virtual Reality & Intelligent Hardware2096-57962025-04-017215517210.1016/j.vrih.2023.06.007Multidimensional image morphing-fast image-based rendering of open 3D and VR environmentsSimon Seibt0Bastian Kuth1Bartosz von Rymon Lipinski2Thomas Chang3Marc Erich Latoschik4Game Tech Laboratory, Faculty of Computer Science, Nuremberg Institute of Technology, Nuremberg 90489, Germany; Corresponding author.Department of Electrical Engineering and Computer Science, Coburg University of Applied Sciences and Arts, Coburg 96450, GermanyGame Tech Laboratory, Faculty of Computer Science, Nuremberg Institute of Technology, Nuremberg 90489, GermanyGame Tech Laboratory, Faculty of Computer Science, Nuremberg Institute of Technology, Nuremberg 90489, GermanyHuman-Computer Interaction Group, Institute of Computer Science, University of Wuerzburg, Wuerzburg 97074, GermanyBackground: In recent years, the demand for interactive photorealistic three-dimensional (3D) environments has increased in various fields, including architecture, engineering, and entertainment. However, achieving a balance between the quality and efficiency of high-performance 3D applications and virtual reality (VR) remains challenging. Methods: This study addresses this issue by revisiting and extending view interpolation for image-based rendering (IBR), which enables the exploration of spacious open environments in 3D and VR. Therefore, we introduce multimorphing, a novel rendering method based on the spatial data structure of 2D image patches, called the image graph. Using this approach, novel views can be rendered with up to six degrees of freedom using only a sparse set of views. The rendering process does not require 3D reconstruction of the geometry or per-pixel depth information, and all relevant data for the output are extracted from the local morphing cells of the image graph. The detection of parallax image regions during preprocessing reduces rendering artifacts by extrapolating image patches from adjacent cells in real-time. In addition, a GPU-based solution was presented to resolve exposure inconsistencies within a dataset, enabling seamless transitions of brightness when moving between areas with varying light intensities. Results: Experiments on multiple real-world and synthetic scenes demonstrate that the presented method achieves high ''VR-compatible'' frame rates, even on mid-range and legacy hardware, respectively. While achieving adequate visual quality even for sparse datasets, it outperforms other IBR and current neural rendering approaches. Conclusions: Using the correspondence-based decomposition of input images into morphing cells of 2D image patches, multidimensional image morphing provides high-performance novel view generation, supporting open 3D and VR environments. Nevertheless, the handling of morphing artifacts in the parallax image regions remains a topic for future research.http://www.sciencedirect.com/science/article/pii/S2096579623000384Computer graphics3D real-time renderingComputer visionImage morphingVirtual reality |
| spellingShingle | Simon Seibt Bastian Kuth Bartosz von Rymon Lipinski Thomas Chang Marc Erich Latoschik Multidimensional image morphing-fast image-based rendering of open 3D and VR environments Virtual Reality & Intelligent Hardware Computer graphics 3D real-time rendering Computer vision Image morphing Virtual reality |
| title | Multidimensional image morphing-fast image-based rendering of open 3D and VR environments |
| title_full | Multidimensional image morphing-fast image-based rendering of open 3D and VR environments |
| title_fullStr | Multidimensional image morphing-fast image-based rendering of open 3D and VR environments |
| title_full_unstemmed | Multidimensional image morphing-fast image-based rendering of open 3D and VR environments |
| title_short | Multidimensional image morphing-fast image-based rendering of open 3D and VR environments |
| title_sort | multidimensional image morphing fast image based rendering of open 3d and vr environments |
| topic | Computer graphics 3D real-time rendering Computer vision Image morphing Virtual reality |
| url | http://www.sciencedirect.com/science/article/pii/S2096579623000384 |
| work_keys_str_mv | AT simonseibt multidimensionalimagemorphingfastimagebasedrenderingofopen3dandvrenvironments AT bastiankuth multidimensionalimagemorphingfastimagebasedrenderingofopen3dandvrenvironments AT bartoszvonrymonlipinski multidimensionalimagemorphingfastimagebasedrenderingofopen3dandvrenvironments AT thomaschang multidimensionalimagemorphingfastimagebasedrenderingofopen3dandvrenvironments AT marcerichlatoschik multidimensionalimagemorphingfastimagebasedrenderingofopen3dandvrenvironments |