CLIPRPN: A New “Rain” Understanding Model for the Generative AI Era#
— Rain Pattern Recognition and Deraining through Multimodal Prompt Learning —
In recent years, with the evolution of multimodal learning represented by diffusion models and CLIP, the integrated understanding of vision and language has been rapidly advancing. This research, “CLIPRPN (CLIP-based Rain Pattern Network),” is a new approach applying this trend to the image restoration field, particularly image deraining.
Background: From Simple Deraining to “Understanding-Based Deraining”#
Conventional deraining models used CNNs and Transformers to extract low-level features (brightness, edges, gradients) with the goal of removing rain streaks. However, these models could not understand the diverse forms of “rain” — misty rain, diagonal streaks, scattering from water droplets — leading to over-correction of textures and structures.
CLIPRPN proposes a new paradigm of “recognizing the physical pattern of rain before removing it” to address this problem.
Model Overview: Rain Pattern Sensing with CLIP#
- Rain Pattern Encoder (RPE): Extracts features from CLIP’s visual encoder and quantifies “raininess.”
- Prompt Generator (PG): Generates latent representations as text prompts such as “misty rain,” “diagonally falling rain,” and “water droplets on lens.”
- Deraining Network (DN): Uses the estimated rain pattern as conditional information to achieve structure-preserving deraining.
In this way, the model understands “the meaning of rain in the image” and performs context-based deraining rather than simple noise removal.
Experiments and Results#
In multiple experiments (Rain100H, Rain200L, SPA-Data, etc.), CLIPRPN showed performance surpassing conventional SOTA models (MPRNet, DGUNet, Restormer, etc.).
- PSNR/SSIM improved by an average of +0.6 to 1.2
- Visually natural detail preservation
- Generalization ability for different rain types
Furthermore, the model is also robust to zero-shot transfer, maintaining stable restoration quality even on untrained datasets.
Significance and Future Prospects#
The greatest significance of CLIPRPN lies in “introducing high-level conceptual understanding to low-level restoration tasks.” In the future, it can be applied not only to rain but also to other weather degradations such as snow, fog, and nighttime reflections, and extension to the video domain (spatiotemporal CLIP guidance) is also expected.