• Angiography simulation and planning using a multi-fluid approach

      Huang, D.; Tang, P.; Tang, W.; Wan, Tao Ruan (2019-01-22)
      Angiography is a minimally invasive diagnostic procedure in endovascular interventions. Training interventional procedures is a big challenge, due to the complexity of the procedures with the changes of measurement and visualization in blood flow rate, volume, and image contrast. In this paper, we present a novel virtual reality-based 3D interactive training platform for angiography procedure training. We propose a multi-fluid flow approach with a novel corresponding non-slip boundary condition to simulate the effect of diffusion between the blood and contrast media. A novel syringe device tool is also designed as an add-on hardware to the 3D software simulation system to model haptics through real physical interactions to enhance the realism of the simulation-based training. Experimental results show that the system can simulate realistic blood flow in complex blood vessel structures. The results are validated by visual comparisons between real angiography images and simulations. By combining the proposed software and hardware, our system is applicable and scalable to many interventional radiology procedures. Finally, we have tested the system with clinicians to assess its efficacy for virtual reality-based medical training.
    • A Constrained Inverse Kinematics Technique for Real-Time Motion Capture Animation

      Tang, W.; Cavazza, M.; Mountain, D.; Earnshaw, Rae A. (1999)
      In this paper we present a constrained inverse kinematics algorithm for real-time motion capture in virtual environments, that has its origins in the simulation of multi-body systems. We apply this algorithm to an articulated human skeletal model using an electromagnetic motion tracking system with a small number of sensors to create avatar postures. The method offers efficient inverse kinematics computation and it is also generalised for the configurations of an articulated skeletal model. We investigate the possibility of capturing fast gestures by analysing the convergence patterns of the algorithm with the motion tracking sampling frequency for a range of actions.
    • Constraint-Based Soft Tissue Simulation for Virtual Surgical Training

      Tang, W.; Wan, Tao Ruan (2014)
      Most of surgical simulators employ a linear elastic model to simulate soft tissue material properties due to its computational efficiency and the simplicity. However, soft tissues often have elaborate nonlinearmaterial characteristics. Most prominently, soft tissues are soft and compliant to small strains, but after initial deformations they are very resistant to further deformations even under large forces. Such material characteristic is referred as the nonlinear material incompliant which is computationally expensive and numerically difficult to simulate. This paper presents a constraint-based finite-element algorithm to simulate the nonlinear incompliant tissue materials efficiently for interactive simulation applications such as virtual surgery. Firstly, the proposed algorithm models the material stiffness behavior of soft tissues with a set of 3-D strain limit constraints on deformation strain tensors. By enforcing a large number of geometric constraints to achieve the material stiffness, the algorithm reduces the task of solving stiff equations of motion with a general numerical solver to iteratively resolving a set of constraints with a nonlinear Gauss–Seidel iterative process. Secondly, as a Gauss–Seidel method processes constraints individually, in order to speed up the global convergence of the large constrained system, a multiresolution hierarchy structure is also used to accelerate the computation significantly, making interactive simulations possible at a high level of details . Finally, this paper also presents a simple-to-build data acquisition system to validate simulation results with ex vivo tissue measurements. An interactive virtual reality-based simulation system is also demonstrated.
    • Context-aware mixed reality: A learning-based framework for semantic-level interaction

      Chen, L.; Tang, W.; John, N.W.; Wan, Tao Ruan; Zhang, J.J. (2020-02)
      Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic‐based interactive MR framework that is beyond current geometry‐based approaches, offering a step change in generating high‐level context‐aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object‐specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real‐world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material‐aware prototype system for context‐aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real‐time semantic‐level interactions.
    • Cross-Platform Cloth Simulation API for Games

      Tang, W.; Sagi, A.S.; Green, D.; Wan, Tao Ruan (2018-03-22)
      Physics simulation is an active research topic in games, because without realistic physics, even the most beautiful game feels static and lifeless. Although cloth simulation is not new in computer graphics, high level of details of cloth in video games is rare and mostly coarse due to complex and nonlinear physical properties of cloth, which requires substantial computing power necessary to simulate it in real-time. This paper presents a robust and scalable real-time cloth simulation framework by exploring a variety of modern simulation techniques to produce realistic cloth simulation for games. The framework integrates with OpenCL GPGPU library to leverage parallelism. The final result is an API for games development with enriched interactive environments.
    • De-smokeGCN: Generative Cooperative Networks for joint surgical smoke detection and removal

      Chen, L.; Tang, W.; John, N.W.; Wan, Tao Ruan; Zhang, J.J. (IEEE, 2020-05)
      Surgical smoke removal algorithms can improve the quality of intra-operative imaging and reduce hazards in image-guided surgery, a highly desirable post-process for many clinical applications. These algorithms also enable effective computer vision tasks for future robotic surgery. In this paper, we present a new unsupervised learning framework for high-quality pixel-wise smoke detection and removal. One of the well recognized grand challenges in using convolutional neural networks (CNNs) for medical image processing is to obtain intra-operative medical imaging datasets for network training and validation, but availability and quality of these datasets are scarce. Our novel training framework does not require ground-truth image pairs. Instead, it learns purely from computer-generated simulation images. This approach opens up new avenues and bridges a substantial gap between conventional non-learning based methods and which requiring prior knowledge gained from extensive training datasets. Inspired by the Generative Adversarial Network (GAN), we have developed a novel generative-collaborative learning scheme that decomposes the de-smoke process into two separate tasks: smoke detection and smoke removal. The detection network is used as prior knowledge, and also as a loss function to maximize its support for training of the smoke removal network. Quantitative and qualitative studies show that the proposed training framework outperforms the state-of-the-art de-smoking approaches including the latest GAN framework (such as PIX2PIX). Although trained on synthetic images, experimental results on clinical images have proved the effectiveness of the proposed network for detecting and removing surgical smoke on both simulated and real-world laparoscopic images.
    • General discriminative optimization for point set registration

      Zhao, Y.; Tang, W.; Feng, J.; Wan, Tao Ruan; Xi, L. (2022-02)
      Point set registration has been actively studied in computer vision and graphics. Optimization algorithms are at the core of solving registration problems. Traditional optimization approaches are mainly based on the gradient of objective functions. The derivation of objective functions makes it challenging to find optimal solutions for complex optimization models, especially for those applications where accuracy is critical. Learning-based optimization is a novel approach to address this problem, which learns the gradient direction from datasets. However, many learning-based optimization algorithms learn gradient directions via a single feature extracted from the dataset, which will cause the updating direction to be vulnerable to perturbations around the data, thus falling into a bad stationary point. This paper proposes the General Discriminative Optimization (GDO) method that updates a gradient path automatically through the trade-off among contributions of different features on updating gradients. We illustrate the benefits of GDO with tasks of 3D point set registrations and show that GDO outperforms the state-of-the-art registration methods in terms of accuracy and robustness to perturbations.
    • Interactive thin elastic materials

      Tang, W.; Wan, Tao Ruan; Huang, D. (2016-04)
      Despite great strides in past years are being made to generate motions of elastic materials such as cloth and biological skin in virtual world, unfortunately, the computational cost of realistic high-resolution simulations currently precludes their use in interactive applications. Thin elastic materials such as cloth and biological skin often exhibit complex nonlinear elastic behaviors. However, modeling elastic nonlinearity can be computationally expensive and numerically unstable, imposing significant challenges for their use in interactive applications. This paper presents a novel simulation framework for simulating realistic material behaviors with interactive frame rate. Central to the framework is the use of a constraint-based multi-resolution solver for efficient and robust modeling of the material nonlinearity. We extend a strain-limiting method to work on deformation gradients of triangulated surface models in three-dimensional space with a novel data structure. The simulation framework utilizes an iterative nonlinear Gauss–Seidel procedure and a multilevel hierarchy structure to achieve computational speedups. As material nonlinearity are generated by enforcing strain-limiting constraints at a multilevel hierarchy, our simulation system can rapidly accelerate the convergence of the large constraint system with simultaneous enforcement of boundary conditions. The simplicity and efficiency of the framework makes simulations of highly realistic thin elastic materials substantially fast and is applicable of simulations for interactive applications.
    • New haptic syringe device for virtual angiography training

      Huang, D.; Tang, P.; Wang, X.; Wan, Tao Ruan; Tang, W. (2019-05)
      Angiography is an important minimally invasive diagnostic procedure in endovascular interventions. Effective training for the procedure is expensive, time consuming and resource demanding. Realistic simulation has become a viable solution to addressing such challenges. However, much of previous work has been focused on software issues. In this paper, we present a novel hardware system-an interactive syringe device with haptics as an add-on hardware component to 3D VR angiography training simulator. Connected to a realistic 3D computer simulation environment, the hardware component provides injection haptic feedback effects for medical training. First, we present the design of corresponding novel electronic units consisting of many design modules. Second, we describe a curve fitting method to estimate injection dosage and injection speed of the contrast media based on voltage variation between the potentiometer to increase the realism of the simulated training. A stepper motor control method is developed to imitate the coronary pressure for force feedback of syringe. Experimental results show that the validity and feasibility of the new haptic syringe device for achieving good diffusion effects of contrast media in the simulation system. A user study experiment with medical doctors to assess the efficacy and realism of proposed simulator shows good outcomes.
    • Object registration in semi-cluttered and partial-occluded scenes for augmented reality

      Gao, Q.H.; Wan, Tao Ruan; Tang, W.; Chen, L. (2018-06)
      This paper proposes a stable and accurate object registration pipeline for markerless augmented reality applications. We present two novel algorithms for object recognition and matching to improve the registration accuracy from model to scene transformation via point cloud fusion. Whilst the first algorithm effectively deals with simple scenes with few object occlusions, the second algorithm handles cluttered scenes with partial occlusions for robust real-time object recognition and matching. The computational framework includes a locally supported Gaussian weight function to enable repeatable detection of 3D descriptors. We apply a bilateral filtering and outlier removal to preserve edges of point cloud and remove some interference points in order to increase matching accuracy. Extensive experiments have been carried to compare the proposed algorithms with four most used methods. Results show improved performance of the algorithms in terms of computational speed, camera tracking and object matching errors in semi-cluttered and partial-occluded scenes.
    • Recovering dense 3D point clouds from single endoscopic image

      Xi, L.; Zhao, Y.; Chen, L.; Gao, Q.H.; Tang, W.; Wan, Tao Ruan; Xue, T. (2021-06)
      Recovering high-quality 3D point clouds from monocular endoscopic images is a challenging task. This paper proposes a novel deep learning-based computational framework for 3D point cloud reconstruction from single monocular endoscopic images. An unsupervised mono-depth learning network is used to generate depth information from monocular images. Given a single mono endoscopic image, the network is capable of depicting a depth map. The depth map is then used to recover a dense 3D point cloud. A generative Endo-AE network based on an auto-encoder is trained to repair defects of the dense point cloud by generating the best representation from the incomplete data. The performance of the proposed framework is evaluated against state-of-the-art learning-based methods. The results are also compared with non-learning based stereo 3D reconstruction algorithms. Our proposed methods outperform both the state-of-the-art learning-based and non-learning based methods for 3D point cloud reconstruction. The Endo-AE model for point cloud completion can generate high-quality, dense 3D endoscopic point clouds from incomplete point clouds with holes. Our framework is able to recover complete 3D point clouds with the missing rate of information up to 60%. Five large medical in-vivo databases of 3D point clouds of real endoscopic scenes have been generated and two synthetic 3D medical datasets are created. We have made these datasets publicly available for researchers free of charge. The proposed computational framework can produce high-quality and dense 3D point clouds from single mono-endoscopy images for augmented reality, virtual reality and other computer-mediated medical applications.
    • Reweighted Discriminative Optimization for least-squares problems with point cloud registration

      Zhao, Y.; Tang, W.; Feng, J.; Wan, Tao Ruan; Xi, L. (2021-11)
      Optimization plays a pivotal role in computer graphics and vision. Learning-based optimization algorithms have emerged as a powerful optimization technique for solving problems with robustness and accuracy because it learns gradients from data without calculating the Jacobian and Hessian matrices. The key aspect of the algorithms is the least-squares method, which formulates a general parametrized model of unconstrained optimizations and makes a residual vector approach to zeros to approximate a solution. The method may suffer from undesirable local optima for many applications, especially for point cloud registration, where each element of transformation vectors has a different impact on registration. In this paper, Reweighted Discriminative Optimization (RDO) method is proposed. By assigning different weights to components of the parameter vector, RDO explores the impact of each component and the asymmetrical contributions of the components on fitting results. The weights of parameter vectors are adjusted according to the characteristics of the mean square error of fitting results over the parameter vector space at per iteration. Theoretical analysis for the convergence of RDO is provided, and the benefits of RDO are demonstrated with tasks of 3D point cloud registrations and multi-views stitching. The experimental results show that RDO outperforms state-of-the-art registration methods in terms of accuracy and robustness to perturbations and achieves further improvement than non-weighting learning-based optimization.
    • Self-supervised monocular image depth learning and confidence estimation

      Chen, L.; Tang, W.; Wan, Tao Ruan; John, N.W. (2020-03-14)
      We present a novel self-supervised framework for monocular image depth learning and confidence estimation. Our framework reduces the amount of ground truth annotation data required for training Convolutional Neural Networks (CNNs), which is often a challenging problem for the fast deployment of CNNs in many computer vision tasks. Our DepthNet adopts a novel fully differential patch-based cost function through the Zero-Mean Normalized Cross Correlation (ZNCC) to take multi-scale patches as matching and learning strategies. This approach greatly increases the accuracy and robustness of the depth learning. Whilst the proposed patch-based cost function naturally provides a 0-to-1 confidence, it is then used to self-supervise the training of a parallel network for confidence map learning and estimation by exploiting the fact that ZNCC is a normalized measure of similarity which can be approximated as the confidence of the depth estimation. Therefore, the proposed corresponding confidence map learning and estimation operate in a self-supervised manner and is a parallel network to the DepthNet. Evaluation on the KITTI depth prediction evaluation dataset and Make3D dataset show that our method outperforms the state-of-the-art results.
    • SLAM-based Dense Surface Reconstruction in Monocular Minimally Invasive Surgery and its Application to Augmented Reality

      Chen, L.; Tang, W.; John, N.W.; Wan, Tao Ruan; Zhang, J.J. (2018-05)
      While Minimally Invasive Surgery (MIS) offers considerable benefits to patients, it also imposes big challenges on a surgeon's performance due to well-known issues and restrictions associated with the field of view (FOV), hand-eye misalignment and disorientation, as well as the lack of stereoscopic depth perception in monocular endoscopy. Augmented Reality (AR) technology can help to overcome these limitations by augmenting the real scene with annotations, labels, tumour measurements or even a 3D reconstruction of anatomy structures at the target surgical locations. However, previous research attempts of using AR technology in monocular MIS surgical scenes have been mainly focused on the information overlay without addressing correct spatial calibrations, which could lead to incorrect localization of annotations and labels, and inaccurate depth cues and tumour measurements. In this paper, we present a novel intra-operative dense surface reconstruction framework that is capable of providing geometry information from only monocular MIS videos for geometry-aware AR applications such as site measurements and depth cues. We address a number of compelling issues in augmenting a scene for a monocular MIS environment, such as drifting and inaccurate planar mapping. Methods A state-of-the-art Simultaneous Localization And Mapping (SLAM) algorithm used in robotics has been extended to deal with monocular MIS surgical scenes for reliable endoscopic camera tracking and salient point mapping. A robust global 3D surface reconstruction framework has been developed for building a dense surface using only unorganized sparse point clouds extracted from the SLAM. The 3D surface reconstruction framework employs the Moving Least Squares (MLS) smoothing algorithm and the Poisson surface reconstruction framework for real time processing of the point clouds data set. Finally, the 3D geometric information of the surgical scene allows better understanding and accurate placement AR augmentations based on a robust 3D calibration. Results We demonstrate the clinical relevance of our proposed system through two examples: a) measurement of the surface; b) depth cues in monocular endoscopy. The performance and accuracy evaluations of the proposed framework consist of two steps. First, we have created a computer-generated endoscopy simulation video to quantify the accuracy of the camera tracking by comparing the results of the video camera tracking with the recorded ground-truth camera trajectories. The accuracy of the surface reconstruction is assessed by evaluating the Root Mean Square Distance (RMSD) of surface vertices of the reconstructed mesh with that of the ground truth 3D models. An error of 1.24mm for the camera trajectories has been obtained and the RMSD for surface reconstruction is 2.54mm, which compare favourably with previous approaches. Second, \textit{in vivo} laparoscopic videos are used to examine the quality of accurate AR based annotation and measurement, and the creation of depth cues. These results show the potential promise of our geometry-aware AR technology to be used in MIS surgical scenes. Conclusions The results show that the new framework is robust and accurate in dealing with challenging situations such as the rapid endoscopy camera movements in monocular MIS scenes. Both camera tracking and surface reconstruction based on a sparse point cloud are effective and operated in real-time. This demonstrates the potential of our algorithm for accurate AR localization and depth augmentation with geometric cues and correct surface measurements in MIS with monocular endoscopes.