Loading...
Context-aware mixed reality: A learning-based framework for semantic-level interaction
Chen, L. Tang, W. John, N.W. Wan, Tao Ruan Zhang, J.J.
Chen, L.
Tang, W.
John, N.W.
Wan, Tao Ruan
Zhang, J.J.
Citations
Altmetric:
Publication Date
2020-02
End of Embargo
Supervisor
Rights
© 2019 The Authors.This is an Open Access article under the terms of the Creative Commons CC-BY license (https://creativecommons.org/licenses/by/4.0/)
Peer-Reviewed
Yes
Open Access status
openAccess
Accepted for publication
Institution
Department
Awarded
Embargo end date
Files
Loading...
chen_et_al_2020
Adobe PDF, 2.23 MB
Additional title
Abstract
Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic‐based interactive MR framework that is beyond current geometry‐based approaches, offering a step change in generating high‐level context‐aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object‐specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real‐world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material‐aware prototype system for context‐aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real‐time semantic‐level interactions.
Version
Published version
Citation
Chen L, Tang W, John NW et al (2020) Context-aware mixed reality: A learning-based framework for semantic-level interaction. Computer Graphics Forum. 39(1): 484-496.
Link to publisher’s version
Link to published version
Link to Version of Record
Type
Article