Thumbnail Image
Publication

Context-aware mixed reality: A learning-based framework for semantic-level interaction

Chen, L.
; 
Tang, W.
; 
John, N.W.
; 
Wan, Tao Ruan
; 
Zhang, J.J.
Publication Date
2020-02
End of Embargo
Supervisor
Keywords
Interaction techniques, Interaction, Methods and applications‐computer games, Methods and applications, Augmented reality, Virtual environments
Rights
© 2019 The Authors. Computer Graphics Forum published by Eurographics ‐ The European Association for Computer Graphics and John Wiley & Sons Ltd. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
Peer-Reviewed
Yes
Open Access status
Accepted for publication
Institution
Department
Awarded
Embargo end date
Collections
Engineering and Digital Technology Publications
Show all metadata
Files
Thumbnail Image
Wan_et_al-2020-Computer_Graphics_Forum.pdf
Adobe PDF, 2.23 MB
To keep suppressed
Adobe PDF, 20.79 MB
To keep suppressed
Adobe PDF, 2.23 MB
Abstract
Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic‐based interactive MR framework that is beyond current geometry‐based approaches, offering a step change in generating high‐level context‐aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object‐specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real‐world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material‐aware prototype system for context‐aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real‐time semantic‐level interactions.
URI
http://hdl.handle.net/10454/17543
Version
Published version
Citation
Chen L, Tang W, John NW et al (2020) Context-aware mixed reality: A learning-based framework for semantic-level interaction. Computer Graphics Forum. 39(1): 484-496.
Link to publisher’s version
Link to published version
Link to Version of Record
https://doi.org/10.1111/cgf.13887
Type
Article
Qualification name
Notes