- Haian Jin1* 📧
- Isabella Liu2*
- Peijia Xu3
- Xiaoshuai Zhang2
- Songfang Han2
- Sai Bi4
- Xiaowei Zhou1
- Zexiang Xu4†
- Hao Su2†
- 1Zhejiang University
- 2University of California, San Diego
- 3Kingstar Technology Inc.
- 4Adobe Research
- 📧 Project Contact. * Equal Contribution. † Equal advisory.
Abstract
We propose TensoIR, a novel inverse rendering approach
based on tensor factorization and neural fields. Unlike previous works that use purely MLP-based neural
fields, thus suffering from low capacity and high computation
costs, we extend TensoRF, a state-of-the-art approach
for radiance field modeling, to estimate scene geometry,
surface reflectance, and environment illumination
from multi-view images captured under unknown lighting
conditions. Our approach jointly achieves radiance field reconstruction
and physically-based model estimation, leading
to photo-realistic novel view synthesis and relighting
results. Benefiting from the efficiency and extensibility of
the TensoRF-based representation, our method can accurately
model secondary shading effects (like shadows and
indirect lighting) and generally support input images captured
under a single or multiple unknown lighting conditions.
The low-rank tensor representation allows us to not
only achieve fast and compact reconstruction but also better
exploit shared information under an arbitrary number of
capturing lighting conditions. We demonstrate the superiority
of our method to baseline methods qualitatively and
quantitatively on various challenging synthetic and realworld
scenes.
Results on TensoIR-Synthetic Dataset
* Physically-based rendering denotes rendering using the estimated BRDF and environment illumination
* All albedo results are scaled for each channel separately to aligned with ground-truth as is done by NeRFactor.
The scaled factor is computed with the formula used in PhySG.
Scene
Result
Relighting Under Unseen Lighting Conditions
Here we have a set of multi-view images captured under a single unknown lighting condition as input.
We compare with NeRFactor and InvRender on relighting under unseen lighting conditions.
We compare with NeRFactor and InvRender on relighting under unseen lighting conditions.
Scene
Secondary-shading Modelling
(a) Visibility Modelling
We compare with NeRFactor on visibility modeling, showing the lighting visibility under individual directional lights (one light at a time, or OLAT).
Scene
Input View |
Our Method |
NeRFactor |
(b) Indirect Lighting Modelling
Rendered with only indirect light |
Rendered with only direct light |
Full Rendering |
Ground-truth |