K-Planes: Explicit Radiance Fields in Space, Time, and Appearance

The paper addresses the need for efficient, interpretable representations of high-dimensional (3D/4D) radiance fields, particularly for dynamic scenes where a direct 4D volume is prohibitively large. Existing factorizations for 3D static scenes do not naturally extend to higher dimensions. The proposed solution is a simple, compact, and interpretable planar factorization that represents a d-dimensional space using d choose 2 planes (e.g., tri-planes for 3D, hex-planes for 4D). In 4D, three planes capture spatial structure and three capture spatiotemporal changes, enabling clear separation of static and dynamic components and facilitating the inclusion of dimension-specific priors. The approach aims for white-box modeling comparable in quality to black-box MLP-based methods while achieving substantial compression and faster training and rendering.

Related work spans implicit radiance field models (e.g., NeRF) that are accurate but slow and opaque, explicit models (e.g., Plenoxels) that are fast but scale poorly with dimension, and hybrid approaches (e.g., Instant-NGP, TensoRF) that balance speed and memory via spatial decompositions and small MLP decoders. Dynamic scene methods generally either (1) learn deformations on a canonical field or (2) condition directly on time; both have tradeoffs in handling topology changes and disentanglement. Some methods replicate 3D structures per timestep (e.g., NeRFPlayer), which is inefficient. Closest to k-planes is Tensor4D, which uses more planes and multiple MLPs. For varying appearance, NeRF-W uses per-image appearance embeddings to model illumination changes. The paper positions k-planes as the first explicit, interpretable model that unifies static, dynamic, and varying-appearance scenes, scales to arbitrary dimensions, achieves compactness and speed, and minimizes reliance on MLPs.

Core representation: A d-dimensional scene is factorized into k = d choose 2 planes, each a regularly sampled 2D feature grid. In 3D this yields tri-planes (xy, xz, yz); in 4D hex-planes add space-time planes (xt, yt, zt). For a 4D point q=(x,y,z,t), features are obtained by bilinear interpolation ψ on each plane projection π_c, producing f_c(q). Plane features are combined via elementwise multiplication (Hadamard product) across planes: f(q) = Π_c f_c(q). Multiplication enables spatially localized signals and reduces the burden on decoders compared to addition. Interpretability and priors: Separation of space-only and space-time planes makes static content rely on space planes (time planes near multiplicative identity) and dynamic content appear in space-time planes. Priors include: (1) multiscale planes at multiple spatial resolutions (e.g., 64,128,256,512) whose features are concatenated to capture multi-resolution structure; (2) spatial total variation regularization (L2 TV) over space-only planes and spatial dimensions of space-time planes; (3) temporal smoothness with a 1D Laplacian penalty over time on space-time planes; (4) sparse transients via L1 penalty encouraging space-time planes to remain at 1 where no motion occurs. Feature decoders: Two variants. (A) Explicit linear decoder with learned color basis: a small MLP maps viewing direction d to per-channel basis vectors b_R(d), b_G(d), b_B(d) ∈ R^M; color is c(q,d)= [f(q)·b_R(d), f(q)·b_G(d), f(q)·b_B(d)], and density σ(q)= f(q)·b_σ with b_σ ∈ R^M. Sigmoid and exponential post-activations constrain outputs. This replaces spherical harmonics with a learned, scene-adaptive basis. (B) Hybrid decoder: two small MLPs g_σ and g_RGB decode density and view-dependent color from f(q) and an embedded view direction. Both variants can be extended with a per-image global appearance code affecting only color decoding to handle varying illumination. Optimization details: For forward-facing scenes, normalized device coordinates (NDC) are used; for unbounded scenes, an L∞ contraction (Mip-NeRF 360) is applied. Proposal sampling (two-stage) uses a small k-planes density model with histogram loss to concentrate samples near surfaces. For multiview dynamic scenes, importance sampling based on temporal differences (as in DyNeRF) is used late in training to focus on dynamic regions (not applicable to monocular/moving-camera data). Standard volumetric rendering integrates per-sample color weighted by transmittance and alpha. Implementation: Pure PyTorch without custom CUDA kernels. Models use modest feature dimensions per scale; the explicit variant remains MLP-free for scene structure while using a small learned basis MLP for view-dependent color.

- General performance: k-planes achieves competitive and often state-of-the-art reconstruction fidelity across static 3D, dynamic 4D, and varying-appearance scenes while being explicit and interpretable. Training and rendering are fast with a pure PyTorch implementation. - Compression: Representing a 4D volume requires about 200 MB vs >300 GB for a dense 4D grid at comparable resolution (~1000× compression). - Decoder and plane combination ablation (static Lego): Using Hadamard product substantially improves PSNR for the explicit model: 35.29 dB (multiplication) vs 28.78 dB (addition); the hybrid model also benefits (35.75 vs 34.80 dB). Parameter count ≈33M in these settings. - Multiscale ablation: Including lower-resolution scales improves quality versus single-scale high resolution, with favorable trade-offs in parameters (e.g., multi-scale settings around 33M params achieving ~35.7 dB PSNR for hybrid on Lego). - Temporal smoothness ablation: A temporal smoothness weight of 0.01 yields the best PSNR on D-NeRF’s Jumping Jacks; quality degrades with over/under-regularization. - Static scenes (NeRF synthetic and LLFF real): Both explicit and hybrid variants match or exceed prior state-of-the-art on these benchmarks, with the hybrid variant slightly higher on metrics; qualitative examples show high fidelity. - Dynamic scenes – monocular “teleporting camera” (D-NeRF): Both explicit and hybrid outperform D-NeRF in PSNR/SSIM and training time (~1 hour single GPU), though they do not surpass recent hybrids like TiNeuVox (≈30 minutes) and V4D (≈4.9 hours). Visual quality is competitive. - Dynamic scenes – multiview (DyNeRF): k-planes produces quality metrics comparable to state-of-the-art (e.g., MixVoxels), with the hybrid variant achieving higher metrics; representative PSNRs across scenes lie roughly in the 29–33 dB range with SSIM around 0.95–0.97, while training takes <4 GPU-hours versus DyNeRF’s ~1344 GPU-hours (8 GPUs for a week). - Varying appearance (Phototourism): With a 32-D appearance code, the explicit and hybrid models achieve mean PSNRs of 22.25 and 22.92 dB, respectively, versus NeRF-W’s 27.00 dB, but optimize in ~35 GPU minutes versus NeRF-W’s ~384 GPU-hours; appearance interpolation is possible without altering geometry.

The proposed k-planes factorization directly targets the challenge of scaling radiance field representations to higher dimensions. By representing all pairwise interactions between dimensions with 2D planes and combining them multiplicatively, the method achieves spatial and temporal localization necessary for accurate volumetric rendering without large MLPs. The explicit separation between space-only and space-time planes enables interpretable decomposition into static and dynamic components, allowing incorporation of targeted priors (e.g., spatial TV, temporal smoothness, sparsity of transients). This design reduces memory substantially and accelerates optimization while maintaining high fidelity across diverse scenarios, including static scenes, dynamic videos, and variable-appearance photo collections. The learned color basis preserves an explicit decoding framework with adaptive expressivity, and the global appearance code disentangles illumination from geometry. Overall, the findings show that a simple, white-box factorization can rival or surpass black-box models in quality with orders-of-magnitude improvements in efficiency and memory, and it generalizes naturally to arbitrary-dimensional spaces.

The paper introduces k-planes, a simple, interpretable factorization of d-dimensional radiance fields into d choose 2 planes, enabling scalable, fast, and memory-efficient reconstruction across static 3D, dynamic 4D, and varying-appearance settings. Key contributions include: (1) hex-planes for 4D with clear static/dynamic separation; (2) multiplicative plane fusion enabling localized features and strong performance with a linear learned color basis; (3) multiscale planes and simple priors for space and time; (4) a compact, fast, white-box model achieving up to ~1000× compression and competitive or state-of-the-art fidelity without custom kernels. Potential future directions include exploring even higher-dimensional factorizations, richer or data-driven priors for dynamics and appearance, integration with streaming or interactive systems, and further compression/speed improvements via custom kernels or quantization.

- While competitive, the method does not surpass the very latest hybrid models (e.g., TiNeuVox, V4D) on some dynamic benchmarks (D-NeRF). - On Phototourism, reconstruction quality (PSNR) lags behind NeRF-W, though training is far faster; the explicit model may produce slightly lower resolution results than NeRF-W. - Importance sampling based on temporal difference cannot be applied to monocular videos or moving-camera datasets. - Choice of temporal smoothness weight affects performance and requires tuning; over/under-regularization degrades quality. - The explicit variant still relies on a small MLP to learn the view-dependent color basis; fully MLP-free view dependence is not explored. - Reported affiliations in the text suggest broad institutional associations; per-author precise affiliations are not individually distinguished.