Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior

Cheng Chen1, Xiaofeng Yang1, Fan Yang1, Chengzeng Feng1, Zhoujie Fu 1, Chuan-Sheng Foo2, Guosheng Lin1 and Fayao Liu2
1Nanyang Technological University, 2A*STAR, Singapore


Recent works on text-to-3d generation show that using only 2D diffusion supervision for 3D generation tends to produce results with inconsistent appearances (e.g., faces on the back view) and inaccurate shapes (e.g., animals with extra legs). Existing methods mainly address this issue by retraining diffusion models with images rendered from 3D data to ensure multi-view consistency while struggling to balance 2D generation quality with 3D consistency. In this paper, we present a new framework Sculpt3D that equips the current pipeline with explicit injection of 3D priors from retrieved reference objects without re-training the 2D diffusion model. Specifically, we demonstrate that high-quality and diverse 3D geometry can be guaranteed by keypoints supervision through a sparse ray sampling approach. Moreover, to ensure accurate appearances of different views, we further modulate the output of the 2D diffusion model to the correct patterns of the template views without altering the generated object's style. These two decoupled designs effectively harness 3D information from reference objects to generate 3D objects while preserving the generation quality of the 2D diffusion model. Extensive experiments show our method can largely improve the multi-view consistency while retaining fidelity and diversity.

Performance on T3Bench

Dataset Dreamfusion Magic3D LatentNeRF Fantasia3D ProlificDreamer Ours-Sculpt3D
Quality 24.9 38.7 34.2 29.2 51.1 53.6
Alignment 24.0 35.3 32.0 23.5 47.8 49.3
Cons. Rate 34% 38% 30% 26% 32% 76%

We achieve state-of-the-art performance on T3Bench. We manually identify and count 3D inconsistencies (e.g., multiple faces, legs, and other distorted shapes.) to calculate the consistent rate of each method.

Generation Results

Small saguaro cactus

A car made out of sushi

An iron key

A gold glittery carnival mask

A colorful parrot on a tree branch

An elegant feather-quill ink pen

An imperial state crown of england

A model of a house in Tudor style

Diverse Generation Results

A blooming potted orchid with purple flowers

A blooming potted orchid with purple flowers

A metal wristwatch

A metal wristwatch

Appearance Refinement Results

Generated results with appearance ambiguity

Correct appearance result using our appearance modulation strategy


  title={Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior},
  author={Chen, Cheng and Yang, Xiaofeng and Yang, Fan and Feng, Chengzeng and Fu, Zhoujie and Foo, Chuan-Sheng and Lin, Guosheng and Liu, Fayao},
  journal={arXiv preprint arXiv:2403.09140},