RAD Training Details and Implementation Notes

[HGao-cv]

This issue is intended to centralize the discussion and documentation of the RAD training process, including pipeline details, loss design, and common debugging tricks.

本 Issue 用于集中讨论和整理 RAD训练过程中的实现细节，包括流程设计、损失计算，以及常见的调试技巧。

1. Recommendation to freeze the normalization layers during RL fine-tuning.
   See the reference implementation in rad/utils.py

2. Learning rate is crucial for stable training.
   A well-chosen learning rate plays a key role in ensuring smooth convergence during RL fine-tuning. If you are confident that the algorithm and implementation are correct, it's often effective to tune the learning rate when the model fails to converge or behaves unstably. Both an excessively large or too small learning rate can significantly impact convergence quality.

[wangyuyy]

Thanks for sharing your great work!​
I am curious about the 3DGS simulation mechanism. It seems to be ambiguous in the paper.​
Did you reconstruct the dynamic 3D Gaussian from video clips using methods like StreetGaussian (as in RealEngine)? Or did you just mask out the dynamic objects and reconstruct the static scene?

[HGao-cv]

Thanks for sharing your great work!​I am curious about the 3DGS simulation mechanism. It seems to be ambiguous in the paper.​Did you reconstruct the dynamic 3D Gaussian from video clips using methods like StreetGaussian (as in RealEngine)? Or did you just mask out the dynamic objects and reconstruct the static scene?

Thanks for your interest! In the NeurIPS version of our paper (this updated version has not been publicly released yet), we added a more detailed explanation of the 3DGS simulation mechanism, which I hope clarifies your question:

To support closed-loop training, we extend the StreetGaussian framework, focusing on enhancing rendering realism and geometric accuracy, particularly for high-fidelity rendering in off-trajectory views.

First, we employ mesh modeling to constrain the geometry of the road surface, constraining Gaussian spheres to the mesh surface to ensure precise road geometry from any viewpoint. Additionally, we model the sky separately to prevent confusion with foreground objects, improving rendering realism under complex lighting conditions.

For foreground objects (e.g., vehicles, pedestrians), we optimize their poses during training and incorporate depth and normal consistency as supervision signals to further enhance geometric reconstruction accuracy and surface detail fidelity. These optimizations significantly improve rendering quality in novel viewpoints, especially in dynamic scenes, where object motion trajectories and surface details remain consistent with real-world observations.

These improvements allow the 3DGS environment to more effectively support closed-loop training, providing a foundation of high realism and geometric accuracy for large-scale trial-and-error learning in autonomous driving strategies.

Another approach to reconstructing driving scenarios is proposed in ReconDreamer-RL, which introduces ReconSimulator combining a video diffusion prior for appearance modeling with a kinematic model for physical modeling, reconstructing driving scenarios from real-world data.