EndoGaussian: Gaussian Splatting for Deformable Surgical Scene Reconstruction

Yifan Liu*, Chenxin Li*, Chen Yang, Yixuan Yuan
The Chinese University of Hong Kong
arXiv Video Code

EndoGaussian reconstructs surgical scene under real-time rendering efficacy (195 FPS real-time, 100x gain), better rendering quality (35+ PSNR), and less training overhead (within 2 min/scene).

The ground truth and the corresponding reconstruction by our method for the "pulling" and "cutting" clips.

Abstract

Reconstructing deformable tissues from endoscopic stereo videos is essential in many downstream surgical applications. However, existing methods suffer from slow inference speed, which greatly limits their practical use.

In this paper, we introduce EndoGaussian, a real-time surgical scene reconstruction framework that builds on 3D Gaussian Splatting. Our framework represents dynamic surgical scenes as canonical Gaussians and a time-dependent deformation field, which predicts Gaussian deformations at novel timestamps. Due to the efficient Gaussian representation and parallel rendering pipeline, our framework significantly accelerates the rendering speed compared to previous methods. In addition, we design the deformation field as the combination of a lightweight encoding voxel and an extremely tiny MLP, allowing for efficient Gaussian tracking with a minor rendering burden. Furthermore, we design a holistic Gaussian initialization method to fully leverage the surface distribution prior, achieved by searching informative points from across the input image sequence.

Experiments on public endoscope datasets demonstrate that our method can achieve real-time rendering speed (195 FPS real-time, 100x gain) while maintaining the state-of-the-art reconstruction quality (35.925 PSNR) and the fastest training speed (within 2 min/scene), showing significant promise for intraoperative surgery applications.

How EndoGaussian facilitate real-time intraoperative application?

xxxx

Using EndoGaussian, you can perform deformable surgical scene reconstruction with nearly 200 FPS, 35+ PSNR.

Method Overview

xxxx

llustration of the proposed EndoGaussian framework, which consists of a) Holistic Gaussian Initialization, b) Voxel-based Gaussian Tracking, and c) Optimization.

Visual Results

xxxx

Display of rendered results of our EndoGaussian against prior SOTA methods on surgical scene reconstruction. The rendering FPS, training time cost and image quality in PSNR is provided.

BibTeX

@article{liu2024endoG,
  author    = {Liu, Yifan and Li, Chenxin and Yang, Chen and Yuan Yixuan},
  title     = {EndoGaussian: Gaussian Splatting for Deformable Surgical Scene Reconstruction},
  journal   = {Arxiv},
  year      = {2024},
}