ReplaceAnything3D

Text-Guided 3D Scene Editing with Compositional Neural Radiance Fields

NeurIPS 2024

Edward Bartrum
University College London,
Alan Turing Institute
Thu Nguyen-Phuoc
Reality Labs Research,
Meta
Chris Xie
Reality Labs Research,
Meta
Zhengqin Li
Reality Labs Research,
Meta
Numair Khan
Reality Labs Research,
Meta
Armen Avetisyan
Reality Labs Research,
Meta
Douglas Lanman
Reality Labs Research,
Meta
Lei Xiao
Reality Labs Research,
Meta
Paper

Abstract

We introduce ReplaceAnything3D model (RAM3D), a novel text-guided 3D scene editing method that enables the replacement of specific objects within a scene. Given multiview images of a scene, a text prompt describing the object to replace, and text prompt describing the new object, our Erase-and-Replace approach can effectively swap objects in the scene with newly generated content while maintaining 3D consistency across multiple viewpoints. We demonstrate the versatility of ReplaceAnything3D by applying it to various realistic 3D scenes, showcasing results of modified foreground objects that are well-integrated with the rest of the scene without affecting its overall integrity.

Method

We distill a pretrained inpainting Latent Diffusion Model over 3 stages

  • Erase stage: remove the masked objects and fill in the background
  • Replace stage: generate new objects and composite them to the inpainted background scene
  • Finally: train a new NeRF using the edited scene images

Visualise Scene edits

Click to play video, then drag the slider to compare the original and edited scene

Additional Results

Statue scene
Octopus
Buddha
Butterfly
Crab
Fern Scene
Strawberry
Cactus
Mushroom
Fried Chicken
Spin-NeRF red net scene
Bubble Tea
Cthulu
Mushroom
Slippers
Mip-NeRF Garden scene
Mushroom
Pineapple
Popcorn
Chess Piece

Citation

If you find our work useful, please consider citing
        
          @misc{bartrum2024replaceanything3dtextguided,
            title={ReplaceAnything3D:Text-Guided 3D Scene Editing
              with Compositional Neural Radiance Fields}, 
            author={Edward Bartrum and Thu Nguyen-Phuoc and
              Chris Xie and Zhengqin Li and Numair Khan and
              Armen Avetisyan and Douglas Lanman and Lei Xiao},
            year={2024},
            eprint={2401.17895},
            archivePrefix={arXiv},
            primaryClass={cs.CV}
      }