[{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/tags/clip/","section":"Tags","summary":"","title":"CLIP","type":"tags"},{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/tags/computer-vision/","section":"Tags","summary":"","title":"Computer Vision","type":"tags"},{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/tags/deep-learning/","section":"Tags","summary":"","title":"Deep Learning","type":"tags"},{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/tags/deraining/","section":"Tags","summary":"","title":"Deraining","type":"tags"},{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/tags/dual-camera/","section":"Tags","summary":"","title":"Dual Camera","type":"tags"},{"content":"","date":"2025年10月18日","externalUrl":null,"permalink":"/tags/example/","section":"Tags","summary":"","title":"Example","type":"tags"},{"content":" Welcome to my personal website.\nI am a PhD student at Waseda University, Japan. In life, I want to accumulate more experiences,\nnurture memories and emotions, learn, and enjoy,\nwhile living a fulfilling life. Currently, I am working on research in computational photography, image/video generation, super-resolution, and diffusion models.\nI am passionate about creating new expressions and experiences through the fusion of image processing and AI. I am expected to complete my doctoral program in December 2026,\nand I am looking for career opportunities as an algorithm engineer in Japan. ","date":"October 18, 2025","externalUrl":null,"permalink":"/en/","section":"GUAN CONG","summary":"\u003cdiv class=\"text-left max-w-3xl mx-auto mt-8\"\u003e\n  \u003cp\u003e\n    Welcome to my personal website.\u003cbr\u003e\n    I am a PhD student at Waseda University, Japan.\n  \u003c/p\u003e\n  \u003cp\u003e\n    In life, I want to accumulate more experiences,\u003cbr\u003e\n    nurture memories and emotions, learn, and enjoy,\u003cbr\u003e\n    while living a fulfilling life.\n  \u003c/p\u003e\n  \u003cp\u003e\n    Currently, I am working on research in computational photography, image/video generation, super-resolution, and diffusion models.\u003cbr\u003e\n    I am passionate about creating new expressions and experiences through the fusion of image processing and AI.\n  \u003c/p\u003e\n  \u003cp\u003e\n    I am expected to complete my doctoral program in December 2026,\u003cbr\u003e\n    and I am looking for career opportunities as an algorithm engineer in Japan.\n  \u003c/p\u003e","title":"GUAN CONG","type":"page"},{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/tags/image-restoration/","section":"Tags","summary":"","title":"Image Restoration","type":"tags"},{"content":" CLIPRPN: A New \u0026ldquo;Rain\u0026rdquo; Understanding Model for the Generative AI Era # — Rain Pattern Recognition and Deraining through Multimodal Prompt Learning —\nIn recent years, with the evolution of multimodal learning represented by diffusion models and CLIP, the integrated understanding of vision and language has been rapidly advancing. This research, \u0026ldquo;CLIPRPN (CLIP-based Rain Pattern Network),\u0026rdquo; is a new approach applying this trend to the image restoration field, particularly image deraining.\nBackground: From Simple Deraining to \u0026ldquo;Understanding-Based Deraining\u0026rdquo; # Conventional deraining models used CNNs and Transformers to extract low-level features (brightness, edges, gradients) with the goal of removing rain streaks. However, these models could not understand the diverse forms of \u0026ldquo;rain\u0026rdquo; — misty rain, diagonal streaks, scattering from water droplets — leading to over-correction of textures and structures.\nCLIPRPN proposes a new paradigm of \u0026ldquo;recognizing the physical pattern of rain before removing it\u0026rdquo; to address this problem.\nModel Overview: Rain Pattern Sensing with CLIP # This model is based on OpenAI\u0026rsquo;s CLIP (Contrastive Language-Image Pretraining) and uses cross-modal representations of images and text to estimate rain patterns in input images.\nRain Pattern Encoder (RPE): Extracts features from CLIP\u0026rsquo;s visual encoder and quantifies \u0026ldquo;raininess.\u0026rdquo; Prompt Generator (PG): Generates latent representations as text prompts such as \u0026ldquo;misty rain,\u0026rdquo; \u0026ldquo;diagonally falling rain,\u0026rdquo; and \u0026ldquo;water droplets on lens.\u0026rdquo; Deraining Network (DN): Uses the estimated rain pattern as conditional information to achieve structure-preserving deraining. In this way, the model understands \u0026ldquo;the meaning of rain in the image\u0026rdquo; and performs context-based deraining rather than simple noise removal.\nExperiments and Results # In multiple experiments (Rain100H, Rain200L, SPA-Data, etc.), CLIPRPN showed performance surpassing conventional SOTA models (MPRNet, DGUNet, Restormer, etc.).\nPSNR/SSIM improved by an average of +0.6 to 1.2 Visually natural detail preservation Generalization ability for different rain types Furthermore, the model is also robust to zero-shot transfer, maintaining stable restoration quality even on untrained datasets.\nSignificance and Future Prospects # The greatest significance of CLIPRPN lies in \u0026ldquo;introducing high-level conceptual understanding to low-level restoration tasks.\u0026rdquo; In the future, it can be applied not only to rain but also to other weather degradations such as snow, fog, and nighttime reflections, and extension to the video domain (spatiotemporal CLIP guidance) is also expected.\n","date":"October 18, 2025","externalUrl":null,"permalink":"/en/projects/1760784774221-clip%E3%81%AB%E5%9F%BA%E3%81%A5%E3%81%8F%E9%9B%A8%E6%B0%B4%E3%83%91%E3%82%BF%E3%83%BC%E3%83%B3%E8%AA%8D%E8%AD%98%E3%81%AB%E3%82%88%E3%82%8B%E7%94%BB%E5%83%8F%E9%99%A4%E9%9B%A8%E3%83%A2%E3%83%87%E3%83%AB/","section":"Projects","summary":"A deraining model using CLIP for automatic routing.","title":"Pattern Recognition: CLIP-driven rain perception: Adaptive deraining with pattern-aware network routing and mask-guided cross-attention","type":"projects"},{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/projects/","section":"Projects","summary":"","title":"Projects","type":"projects"},{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/tags/super-resolution/","section":"Tags","summary":"","title":"Super Resolution","type":"tags"},{"content":"","date":"2025年10月18日","externalUrl":null,"permalink":"/tags/tag/","section":"Tags","summary":"","title":"Tag","type":"tags"},{"content":"","date":"October 18, 2025","externalUrl":null,"permalink":"/en/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":" A new image restoration approach for the era of multi-camera smartphones Overview # In recent years, multi-camera devices such as smartphones and action cameras can simultaneously capture images with different focal lengths, such as wide-angle and telephoto.\nHowever, these images have field of view (FoV) mismatches and resolution/color domain differences, making high-definition reconstruction difficult through simple image fusion.\nThe proposed DM³Net (Dual-camera SR via Domain Modulation and Multi-scale Matching) addresses this challenge by combining \u0026ldquo;multi-stage feature matching\u0026rdquo; with \u0026ldquo;global domain alignment,\u0026rdquo; achieving super-resolution performance that surpasses conventional methods. Background and Challenges # In smartphone dual cameras:\nWide-angle camera: Covers a wide field of view but has low resolution Telephoto camera: Has a narrow field of view but contains high-resolution details The goal of Dual-Camera Super-Resolution is to use the telephoto image as a \u0026ldquo;reference image\u0026rdquo; to enhance the wide-angle image to high definition. Conventional methods (e.g., DCSR, KeDuSR) transfer high-frequency information through patch-level feature matching, but had problems with reduced detail reproduction due to scale constraints.\nProposed Method: Key Components of DM³Net # DM³Net consists of the following three elements:\n1. Multi-Scale Matching # Calculates feature patch similarity at multiple receptive field scales (1/4, 1/2, 1) and selects the optimal reference patch. This significantly improves structural consistency and detail reproduction compared to conventional single-scale matching.\nFurthermore, a \u0026ldquo;Key Pruning\u0026rdquo; strategy was introduced to remove redundant patches with high similarity. This reduced GPU memory consumption by approximately 17% while cutting inference time to one-third.\n2. Domain Modulation # Wide-angle images (LR) and telephoto images (Ref) have domain differences due to different capture pipelines. Therefore, DM³Net learns global domain embeddings (z, zgt) from each pair (LR-GT, LRc-Ref). By introducing a \u0026ldquo;domain-aware loss (L_domain)\u0026rdquo; that aligns the distributions of both and adaptively incorporating it into the reconstruction process, smooth mapping to the high-resolution domain is achieved.\n3. Integrated Reconstruction # Finally, high-frequency features reconstructed through matching and domain-modulated embedding information are integrated to generate visually natural and high-fidelity HR images.\nExperimental Results # On three real-world datasets (DuSR-Real, RealMCVSR-Real, CameraFusion-Real), DM³Net achieved the following performance: In particular, it maintained high accuracy even in non-overlapping regions (Corner Region), confirming robust operation against partial field of view and color differences.\nAblation Studies and Discussion # Multi-scale: PSNR improves as scales increase (27.75 dB). Domain modulation: Best performance when using both z and zgt. Key Pruning: Optimal balance at threshold 0.7 and sampling interval 16. From these results, it is analyzed that DM³Net\u0026rsquo;s performance improvement comes from the combination of improved local matching accuracy and introduction of global domain alignment.\nApplications and Future Prospects # DM³Net is directly applicable to multi-lens capture environments such as smartphones, drones, and vehicle-mounted cameras. In particular, it shows high potential in Computational Photography technology that integrates different angles of view, resolutions, and ISP processing across hardware.\nIn the future, model lightweighting and real-time processing are targeted, with application to diffusion-based restoration and video extension expected.\n","date":"October 18, 2025","externalUrl":null,"permalink":"/en/projects/1760786004876-%E3%83%87%E3%83%A5%E3%82%A2%E3%83%AB%E3%82%AB%E3%83%A1%E3%83%A9%E7%94%BB%E5%83%8F%E3%81%AB%E5%AF%BE%E3%81%99%E3%82%8B%E8%B6%85%E8%A7%A3%E5%83%8F%E6%89%8B%E6%B3%95/","section":"Projects","summary":"Introduction to DM³Net, a high-definition image generation technology utilizing smartphone dual cameras","title":"WACV 2026: DM3Net: Dual-Camera Super-Resolution via Domain Modulation and Multi-scale Matching","type":"projects"},{"content":" Your browser does not support PDF embedding. Click here to download. ","date":"September 15, 2025","externalUrl":null,"permalink":"/en/resume/","section":"GUAN CONG","summary":"\u003ciframe\n    src=\"/resume/congguan_en.pdf\"\n    width=\"200%\"\n    height=\"800px\"\n\u003e\nYour browser does not support PDF embedding.\n\u003ca href=\"/resume/congguan_en.pdf\"\u003eClick here to download\u003c/a\u003e.\n\u003c/iframe\u003e","title":"Resume","type":"page"},{"content":" Reference Links # PR by me\nThe model released\nIntroduction # In AWS OpenSearch, we provide pre-trained models for users to download. These pre-trained models are managed through a dedicated pipeline, where we conduct various validations, from training to inference and deployment. The models are then uploaded to a designated server, and the relevant documentation is updated accordingly.\nAfter releasing a new semantic model, we found that the original pipeline could not effectively validate and deploy it since it is not a conventional dense model. As a result, we decided to implement a new pipeline to manage and maintain our neural sparse model.\nThe work I did # Automated the upload process for Sparse models: I wrote (or modified) a complete GitHub Actions workflow along with the necessary Python code to perform the following actions under specific conditions (when model_type = Sparse):\nPrepare input parameters (model source, ID, version, tracking format, embedding dimension, pooling mode, description, etc.) Automatic tracing: Convert the original model into TorchScript or ONNX format, or generate both formats simultaneously. Embedding validation: Ensure that the converted model\u0026rsquo;s embedding dimensions and functionalities are correct. Check if the model already exists: If it exists and cannot be overwritten, reject the upload. Manual approval: Officially upload to S3 only after approval from key personnel. Update repository documentation and records: Append the upload action to MODEL_UPLOAD_HISTORY.md, supported_models.json, and update CHANGELOG.md. Trigger downstream Jenkins processes to further complete the model\u0026rsquo;s release or integration in ml-models. Technical Used # Python GitHub Actions Huggingface Jenkins AWS S3 AWS OpenSearch AWS IAM AWS Secrets Manager ","date":"January 15, 2025","externalUrl":null,"permalink":"/en/projects/1736923185327-a-model-release-pipeline/","section":"Projects","summary":"A model release pipeline for our semantic search","title":"A Model Release Pipeline","type":"projects"},{"content":"","date":"January 15, 2025","externalUrl":null,"permalink":"/en/tags/aws/","section":"Tags","summary":"","title":"AWS","type":"tags"},{"content":"","date":"January 15, 2025","externalUrl":null,"permalink":"/en/tags/ci/cd/","section":"Tags","summary":"","title":"CI/CD","type":"tags"},{"content":" Abstract # DSLR cameras can achieve various zoom levels by changing the lens distance or swapping lens types. However, due to space constraints, these techniques are not feasible on mobile devices. Most smartphones adopt a hybrid zoom system: typically using a wide-angle (W) camera for low zoom levels and a telephoto (T) camera for high zoom levels. To simulate zoom levels between W and T, these systems crop and digitally enlarge images from W, resulting in significant detail loss. In this paper, we propose an efficient hybrid zoom super-resolution system for mobile devices. This system captures synchronized W and T image pairs and utilizes a machine learning model to align and transfer details from T to W. We further develop an adaptive blending method that can handle depth of field mismatches, scene occlusions, motion uncertainties, and alignment errors. To minimize domain differences, we design a dual smartphone camera setup to capture real scene inputs and real label data for supervised training. In extensive evaluations on real scenes, our method can generate 12-megapixel images on mobile platforms in 500 milliseconds, achieving TA.\nKey word # hybrid zoom, dual camera fusion, deep neural networks\nKey Takeaways # Problem Addressed # Almost all upsampling methods (such as bilinear interpolation, bicubic interpolation, etc.) lead to varying degrees of image quality degradation. Consequently, numerous methods have emerged that utilize high zoom T images as references to add real details to low zoom W images. Commercial solutions are not publicly available, and academic research is relatively inefficient. These methods often perform poorly on mobile devices, are susceptible to defects in reference images, and may introduce domain differences between training and inference. To address these issues, we propose a Hybrid Zoom Super-Resolution (HZSR) system.\nMain Contributions # A machine learning-based hybrid zoom super-resolution (HZSR) system that operates efficiently on mobile devices and exhibits strong robustness to imperfections in real scene images (see Section 3). A training strategy that minimizes domain differences through a dual smartphone camera platform and avoids learning trivial mappings in reference super-resolution (RefSR) tasks (see Section 4). A high-quality synchronized dataset containing 150 pairs of high-resolution (12MP) W and T images, named the Hzsr dataset, which will be released on our project website for future research (see Section 5). Methodology # Approach # Adapting to Imperfect Reference Images # We propose an efficient defocus detection algorithm that excludes defocused areas based on the correlation between scene depth and optical flow. By combining defocus maps, alignment errors, optical flow uncertainties, and scene occlusion information, we design an adaptive blending mechanism to generate high-quality, artifact-free super-resolution results.\nMinimizing Domain Differences through Real Scene Inputs # In reference super-resolution (RefSR) tasks, collecting perfectly aligned W/T image pairs as training data is very challenging. Therefore, previous research has explored two possible but suboptimal solutions:\nUsing the reference image T as the training target (e.g., Wang et al. 2021 and Zhang et al. 2022a), which may transmit defects from the reference image or lead the network to learn identity mapping. Synthesizing low-resolution inputs from target images using degradation models (e.g., Trinidad et al. 2019 and Zhang et al. 2022a), but this method introduces domain differences between training and inference stages, thereby reducing super-resolution performance on real scene images. To avoid learning identity mappings and minimize domain differences, we adopt a design where, during training, a second smartphone of the same model mounted on the photography platform synchronously captures additional T images as references (see Figure 6). With this design, the fusion model uses real W images as input during both training and inference, eliminating domain differences. Additionally, the reference and target images are captured by T cameras from different devices to prevent the network from learning identity mappings. Compared to existing dual zoom RefSR datasets, our design has significant advantages:\nSome datasets exhibit strong temporal motion between W and T (e.g., Wang et al. 2021); Some datasets are limited to static scenes (e.g., Wei et al. 2020). We have collected a large-scale dataset containing high-quality W/T synchronized data, covering dynamic scenes, including portraits, architecture, landscapes, as well as challenging scenes with moving objects and nighttime scenarios. Experiments show that our method outperforms current state-of-the-art methods on both existing dual zoom RefSR datasets and our new dataset. Model/Algorithm # When the user adjusts the zoom to a medium range (e.g., 3-5 times) and presses the shutter button, the system captures a pair of synchronized images. The processing flow is as follows:\nImage Alignment: # First, we perform global coarse alignment through keypoint matching, followed by local dense alignment using optical flow (see Section 3.1). Coarse Alignment: First, we crop the image from the wide-angle camera (W) to match the field of view (FOV) of the telephoto camera (T). Next, we use bicubic interpolation to adjust the spatial resolution of W to match T (4k×3k). Then, we estimate the global 2D translation vector using the FAST feature point matching algorithm (Rosten and Drummond 2006) and adjust the cropped W to obtain the adjusted image \\(I_{src}\\). Dense Alignment: We use PWC-Net to estimate the dense optical flow between \\(I_{src}\\) and \\(I_{ref}\\). It is important to note that the average offset between W and T at 12MP resolution is about 150 pixels, which is much larger than the motion range in most optical flow training data. Therefore, the optical flow estimated from 12MP images is often very noisy. To improve accuracy, we downsample \\(I_{src}\\) and \\(I_{ref}\\) to a smaller resolution of 384×512 to predict the optical flow, then upsample the flow to the original resolution and deform \\(I_{ref}\\) to obtain \\(\\tilde{I}{ref}\\) using bilinear resampling. This method provides more accurate and robust optical flow estimates at a smaller scale. Optimization To accommodate the computational resource limitations of mobile devices, we removed the DenseNet structure from the original PWC-Net, reducing the model size by 50%, latency by 56%, and peak memory by 63%. Although the endpoint error (EPE) of optical flow increased by 8% on the Sintel dataset, the visual quality of the optical flow remains similar. Additionally, we generate an occlusion map \\(M{occ}\\) using forward-backward consistency checks (Alvarez et al. 2007) to identify areas occluded during alignment.\nImage Fusion: # We use UNet (Ronneberger et al. 2015) to fuse the brightness channel of the cropped image from the wide-angle camera (W) with the deformed reference image from the telephoto camera (T) (see Section 3.2).\nUsing the source image, the deformed reference image, and the occlusion mask as inputs Fusion Process: To maintain color consistency in the W image, we perform fusion only in the luminance space. The specific implementation is as follows:\nInput Preparation: Convert the source image \\(I_{src}\\) to a grayscale image, denoted as \\(Y_{src}\\). Convert the deformed reference image \\(I_{ref}\\) to a grayscale image, denoted as \\(\\tilde{Y}_{ref}\\). Input the occlusion mask \\(M_{occ}\\). Fusion Network: We construct a 5-layer UNet network, using \\(Y_{src}\\), \\(\\tilde{Y}{ref}\\), and \\(M{occ}\\) as inputs to generate the fused grayscale image \\(Y_{fusion}\\). The detailed architecture of this UNet is provided in the appendix. Color Recovery: Combine the fused grayscale image \\(Y_{fusion}\\) with the UV color channels of \\(I_{src}\\). Convert back to RGB space to generate the fused output image \\(I_{fusion}\\). Adaptive Blending: # Although machine learning models have strong capabilities in image alignment and fusion, mismatches between W and T can still introduce noticeable artifacts in the output. These mismatches include:\nDepth of Field differences Occluded pixels Warping artifacts during the alignment stage To address these issues, we propose an adaptive blending strategy that combines alpha masks derived from defocus maps, occlusion maps, optical flow uncertainty maps, and alignment rejection maps to adaptively blend \\(Y_{src}\\) and \\(Y_{fusion}\\). The final output image eliminates significant artifacts and maintains high robustness in cases where pixel-level consistency issues exist between W and T.\nThe Narrow Depth of Field Issue of Telephoto (T) # We observe that the telephoto camera (T) on mobile devices typically has a narrower depth of field (DoF) than the wide-angle camera (W). This is because depth of field is proportional to the square of the f-stop number and focal length. Typically, the focal length ratio between T and W is greater than 3, while the f-stop ratio is less than 2.5. Therefore, the depth of field of T is usually significantly narrower than that of W. Thus, it is necessary to exclude defocused pixel areas using the defocus map to avoid introducing artifacts. Estimating the defocus map from a single image is a pathological problem that usually requires complex and computationally intensive machine learning models. To this end, we propose an efficient algorithm that reuses the optical flow information computed during the alignment stage to generate the defocus map:\nThe optical flow information contains depth and motion features of the image. By analyzing the relationship between optical flow and scene depth, we can efficiently generate the defocus map without additional complex models. Defocus Map Generation Method # To estimate the defocus map, we need to determine two key pieces of information:\nCamera\u0026rsquo;s Focus Position: the center area of the image that is in sharp focus. Relative Depth of Each Pixel with Respect to the Focus Position. Since the FOV of W and T is approximately parallel (fronto-parallel), and the magnitude of optical flow is proportional to the camera disparity, which is related to scene depth, we design an efficient optical flow-based defocus map estimation algorithm, as follows (see Figure 5): Obtain the Focus Region of Interest (ROI): Obtain the focus area from the camera\u0026rsquo;s auto-focus module. This module provides a rectangular area representing the region where most pixels in the T image are sharply focused (i.e., the focus ROI). Estimate the Focus Position \\(x_f\\) based on Optical Flow: Using dual-camera stereo vision, treat optical flow as a proxy for scene depth. Assuming that in a static scene, pixels located in the same focal plane have similar optical flow vectors (Szeliski 2022). Apply the k-means clustering algorithm to the optical flow vectors within the focus ROI to determine the area with the highest clustering density (i.e., the maximum cluster). The cluster center is defined as the focus position \\(x_f\\). Estimate Relative Depth of Pixels and Generate the Defocus Map: Calculate the Euclidean distance (L2 distance) between the optical flow vector of each pixel and the optical flow vector at the focus position \\(x_f\\). Defocus Map Calculation Formula # The calculation formula for the defocus map \\(M_{defocus}(x)\\) is as follows: \\( M_{defocus}(x) = \\text{sigmoid} \\left( \\frac{| F_{fwd}(x) - F_{fwd}(x_f) |_2^2 - \\gamma}{\\sigma_f} \\right) \\) where:\n\\(F_{fwd}(x)\\): the forward optical flow vector of pixel \\(x\\). \\(F_{fwd}(x_f)\\): the forward optical flow vector at the focus position \\(x_f\\). \\(\\gamma\\): a threshold that controls the tolerance range. \\(\\sigma_f\\): a parameter that controls the smoothness of the defocus map. Occlusion Map Calculation Formula # The occlusion map \\(M_{occ}(x)\\) is calculated based on forward-backward optical flow consistency, as follows: $$ M_{occ}(x) = \\min\\left(s \\cdot | W(W(x; F_{fwd}); F_{bwd}) - x |_2, 1 \\right) $$ where:\n\\(W\\): bilinear warping operator, used to map image coordinates \\(x\\) to new positions based on optical flow. \\(F_{fwd}\\): forward optical flow from the source image \\(I_{src}\\) to the reference image \\(I_{ref}\\). \\(F_{bwd}\\): backward optical flow from the reference image \\(I_{ref}\\) to the source image \\(I_{src}\\). \\(s\\): a scaling factor used to adjust the sensitivity of optical flow differences. \\(x\\): 2D image coordinates on the source image. Optical Flow Uncertainty Map # Due to the inherently ill-posed nature of dense correspondence problems, we enhance the functionality of PWC-Net to output an optical flow uncertainty map, which describes the uncertainty of each pixel\u0026rsquo;s optical flow prediction. The method is as follows:\nUncertainty Modeling: The enhanced PWC-Net predicts a multivariate Laplacian distribution for the optical flow vector of each pixel, rather than a single point estimate. It predicts two additional channels representing the log-variance of optical flow in the x and y directions, denoted as \\({Var}_x\\) and \\({Var}_y\\).\nConvert to Pixel Units: The log-variance is converted to pixel units using the following formula—here, \\(S(x)\\) is the optical flow uncertainty value for each pixel. \\( S(x) = \\sqrt{\\exp(\\log(\\text{Var}_x(x))) + \\exp(\\log(\\text{Var}_y(x)))} \\)\nGenerate Optical Flow Uncertainty Map: \\( M_{\\text{flow}}(x) = \\frac{\\min(S(x), s_{\\text{max}})}{s_{\\text{max}}} \\) The optical flow uncertainty map typically has higher values in object boundaries or texture-less regions.\nAlignment Rejection Map # To exclude artifacts introduced by alignment errors, we generate an alignment rejection map by comparing the local similarity between the source image and the aligned reference image. The method is as follows:\nMatch Optical Resolution: Use bilinear interpolation to adjust the resolution of the aligned reference frame \\(\\tilde{Y}{\\text{ref}}\\) to match the optical resolution of W, resulting in a downsampled image \\(\\tilde{Y}{\\text{ref}}^\\downarrow\\). Calculate Local Differences: For local patches \\(P_{\\text{src}}\\) from the source image and \\(P_{\\tilde{\\text{ref}}}\\) from the aligned reference image: Subtract the mean of the patches \\(u_{\\text{src}}\\) and \\(u_{\\text{ref}}\\). Calculate the normalized difference: $$ P_\\delta = (P_{\\text{src}} - \\mu_{\\text{src}}) - (P_{\\tilde{\\text{ref}}} - \\mu_{\\text{ref}}) $$ Generate the alignment rejection map $$ M_{\\text{reject}}(x) = 1 - \\exp\\left(-\\frac{|P_\\delta(x)|2^2}{\\sigma{\\text{src}}^2(x) + \\epsilon_0}\\right) $$ Final Blending # $$ M_{\\text{blend}} = \\max(1 - M_{\\text{occ}} - M_{\\text{defocus}} - M_{\\text{flow}} - M_{\\text{reject}}, 0) $$ The final output image is generated through alpha blending and cropped back to the full W image: $$ I_{\\text{final}} = \\text{uncrop}\\left(M_{\\text{blend}} \\odot I_{\\text{fusion}} + (1 - M_{\\text{blend}}) \\odot I_{\\text{src}}, W\\right) $$ By combining multiple masks (such as occlusion maps, defocus maps, optical flow uncertainty maps, and alignment rejection maps), we ensure high quality and robustness of the fusion results, avoiding the introduction of artifacts, blurriness, or alignment errors.\nImplementation Details # Results and Evaluation # Experiments: # Results # Comparisons # Critical Analysis # Strengths # Weaknesses # Insights # Broader Impact and Future Work # Broader Implications # Open Questions # Future Work # Personal Reflection # Relevance to My Work # Practical Applications # Key Learnings # Related Work and Context # Prior Work # Learning-based Single Image Super-Resolution (SISR) Over the past decade, various methods (e.g., Christian Ledig 2017; Dong et al. 2014; Kim et al. 2016; Lai et al. 2017; Wang et al. 2018; Xu et al. 2023; Zhang et al. 2019a, 2022b, 2018) have demonstrated outstanding results in the field of single image super-resolution. However, due to the severely ill-posed nature of this task, these methods tend to generate blurry details under large upsampling factors (e.g., 2-5 times, common in smartphone hybrid zoom). Additionally, some methods are only applicable to specific domains, such as face super-resolution (Chan et al. 2021; Gu et al. 2020; He et al. 2022; Menon et al. 2020).\nReference Super-Resolution (RefSR) Based on Internet Images RefSR generates high-resolution results from low-resolution inputs and one or more high-resolution reference images (Pesavento et al. 2021). Traditional RefSR methods assume that reference images come from the internet (Sun and Hays 2012) or are captured at different times, locations, or camera models during the same event (Wang et al. 2016; Zhang et al. 2019b), focusing on improving dense alignment between source and reference images (Huang et al. 2022; Jiang et al. 2021; Xia et al. 2022; Zheng et al. 2018) or enhancing robustness to unrelated reference images (Lu et al. 2021; Shim et al. 2020; Xie et al. 2020; Yang et al. 2020; Zhang et al. 2019b). In contrast, we alleviate the alignment challenge by synchronously capturing W and T images, avoiding alignment issues caused by object motion.\nReference Super-Resolution (RefSR) Based on Auxiliary Cameras Recent RefSR studies (Trinidad et al. 2019; Wang et al. 2021; Zhang et al. 2022a) have captured reference images of the same scene using auxiliary cameras. However, due to the lack of pixel-aligned inputs and real label image pairs, PixelFusionNet (Trinidad et al. 2019) synthesizes low-resolution inputs from high-resolution reference images using degradation models and trains using pixel-level losses (e.g., l1 and VGG losses). However, this model performs poorly when faced with real scene inputs, primarily due to domain differences between training and inference stage images.\nOn the other hand, SelfDZSR (Zhang et al. 2022a), DCSR (Wang et al. 2021), and RefVSR (Lee et al. 2022) treat reference images as targets for training or fine-tuning. We observe that this training setup is prone to getting stuck in degraded local minima: the model often learns identity mappings, merely copying the content of T images to the output. This leads to severe alignment errors, color shifts, and depth of field mismatches, which are unacceptable in practical photography.\nTo address these issues, we additionally capture a T image during training to alleviate the aforementioned problems and improve training robustness.\nEfficient Mobile Device Reference Super-Resolution (RefSR) Existing methods often consume significant memory due to the use of attention mechanisms/Transformers (e.g., Wang et al. 2021; Yang et al. 2020) or deep network architectures (e.g., Zhang et al. 2022a). These methods may encounter out-of-memory (OOM) issues even on a desktop GPU with 40GB RAM when processing 12MP input resolutions, making them impractical for mobile devices. In contrast, our system processes 12MP inputs on mobile GPUs in just 500 milliseconds, using only 300MB of memory.\nOur system design is inspired by reference image deblurring methods for faces [Lai et al. 2022], but the problems we address are more challenging:\nSuper-Resolution for General Scenes: We apply super-resolution to general images rather than focusing solely on faces. Therefore, our system needs to be more robust to diverse scenes and capable of handling various imperfections and mismatches from both cameras. Domain Differences in Training Data: Unlike face deblurring models that can learn from synthetic data, image super-resolution models are more sensitive to domain differences in training data. Additionally, collecting real training data for reference-based super-resolution tasks is more challenging. Thus, our proposed adaptive blending method and dual smartphone platform design become key distinctions from the method in [Lai et al. 2022]. Historical Context # References # ","date":"January 15, 2025","externalUrl":null,"permalink":"/en/posts/ehz/","section":"Posts","summary":"\u003ch2 class=\"relative group\"\u003eAbstract\n    \u003cdiv id=\"abstract\" class=\"anchor\"\u003e\u003c/div\u003e\n    \n    \u003cspan\n        class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none\"\u003e\n        \u003ca class=\"text-primary-300 dark:text-neutral-700 !no-underline\" href=\"#abstract\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\n    \u003c/span\u003e\n    \n\u003c/h2\u003e\n\u003cp\u003eDSLR cameras can achieve various zoom levels by changing the lens distance or swapping lens types. However, due to space constraints, these techniques are not feasible on mobile devices. Most smartphones adopt a \u003cstrong\u003ehybrid zoom system\u003c/strong\u003e: typically using a wide-angle (W) camera for low zoom levels and a telephoto (T) camera for high zoom levels. To simulate zoom levels between W and T, these systems crop and digitally enlarge images from W, resulting in significant detail loss. In this paper, we propose an efficient hybrid zoom super-resolution system for mobile devices. This system captures \u003cstrong\u003esynchronized W and T image pairs\u003c/strong\u003e and \u003cstrong\u003eutilizes a machine learning model to align and transfer details from T to W\u003c/strong\u003e. We further develop an \u003cstrong\u003eadaptive blending method\u003c/strong\u003e that can handle \u003cstrong\u003edepth of field mismatches, scene occlusions, motion uncertainties, and alignment errors\u003c/strong\u003e. To minimize domain differences, we design a dual smartphone camera setup to capture real scene inputs and real label data for supervised training. In extensive evaluations on real scenes, our method can generate 12-megapixel images on mobile platforms in 500 milliseconds, achieving TA.\u003c/p\u003e","title":"Efficient Hybrid Zoom using Camera Fusion on Mobile Phones Reading Notes","type":"posts"},{"content":"","date":"January 15, 2025","externalUrl":null,"permalink":"/en/tags/machine-learning/","section":"Tags","summary":"","title":"Machine Learning","type":"tags"},{"content":"","date":"January 15, 2025","externalUrl":null,"permalink":"/en/tags/mlops/","section":"Tags","summary":"","title":"MLOps","type":"tags"},{"content":"","date":"January 15, 2025","externalUrl":null,"permalink":"/en/tags/opensearch/","section":"Tags","summary":"","title":"OpenSearch","type":"tags"},{"content":"","date":"January 15, 2025","externalUrl":null,"permalink":"/en/tags/paper/","section":"Tags","summary":"","title":"Paper","type":"tags"},{"content":"","date":"January 15, 2025","externalUrl":null,"permalink":"/en/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"January 15, 2025","externalUrl":null,"permalink":"/en/tags/search/","section":"Tags","summary":"","title":"Search","type":"tags"},{"content":" Some link for your reference # RFC by me\nPR by me\nBlog post by me\nIntroduction # Neural sparse search is an efficient method for semantic retrieval introduced in OpenSearch 2.11. It uses semantic techniques to interpret queries, handling terms that traditional search may miss. While dense models find similar results, they can overlook exact matches. Neural sparse search uses sparse representations to capture both semantic similarities and specific terms, improving result explanation and presentation by providing a more comprehensive retrieval solution.\nBackground # Neural sparse search first expands text (either a query or a document) into a larger set of terms, each weighted by its semantic relevance. It then uses Lucene\u0026rsquo;s efficient term vector computation to identify the highest-scoring results. This approach leads to reduced index and memory costs as well as lower computational expenses. For example, while dense encoding using k-NN retrieval increases RAM costs by 7.9% at search time, neural sparse search uses a native Lucene index, avoiding any increase in RAM cost at search time. Moreover, neural sparse search leads to a much smaller index size compared to dense encoding. A document-only model generates an index that is only 10.4% the size of a dense encoding index, and for a bi-encoder, the index size is 7.2% of a dense encoding index.\nGiven these advantages, we\u0026rsquo;ve continued to refine neural sparse retrieval to make it even more efficient. OpenSearch 2.15 introduced a new feature: the two-phase search pipeline. This pipeline splits the neural sparse query terms into two categories: high-scoring tokens that are more relevant to the search and low-scoring tokens that are less relevant. Initially, the algorithm selects documents using the high-scoring tokens and then recalculates the score for those documents by including both high- and low-scoring tokens. This process significantly reduces computational load while maintaining the quality of the final ranking.\nThe two-phase algorithm # The two-phase search algorithm operates in two stages:\nInitial Phase # The algorithm uses model inference to quickly select a set of candidate documents using high-scoring tokens from the query. These high-scoring tokens, which constitute a small portion of the total number of tokens, have significant weight—or relevance—allowing for a rapid identification of potentially relevant documents. This process significantly reduces the number of documents that need to be processed, thereby lowering computational costs.\nRecalculation Phase # The algorithm then recalculates the scores for the candidate documents selected in the first phase, this time including both high-scoring and low-scoring tokens from the query. Although low-scoring tokens carry less weight individually, they provide valuable information as part of a comprehensive evaluation, particularly when long-tail terms contribute significantly to the overall score. This allows the algorithm to determine final document scores with greater accuracy.\nBy processing documents in stages, this approach reduces computational overhead while mainitaining accuracy. The rapid selection in the first phase enhances efficiency, while the more detailed scoring in the second phase ensures accuracy. Even when handling a large number of long-tail terms, the results remain of high quality, with a notable improvement in computational efficiency.\nPerformance metrics # Depending on the data distribution, the two-phase processor achieved an increase in speed ranging from 1.22x to 1.78x in doc only model. And achieved an increase in speed ranging from 4.15x to 6.87x in bi-encoder model. ","date":"January 15, 2025","externalUrl":null,"permalink":"/en/projects/1736922318122-neural-sparse-search-on-aws-opensearch/","section":"Projects","summary":"A speed up algorithm for AWS OpenSearch Neural Sparse Search","title":"Two Phase Neural Sparse Search On AWS OpenSearch","type":"projects"},{"content":"","date":"January 4, 2025","externalUrl":null,"permalink":"/en/tags/3d-rendering/","section":"Tags","summary":"","title":"3D Rendering","type":"tags"},{"content":"","date":"January 4, 2025","externalUrl":null,"permalink":"/en/tags/c++/","section":"Tags","summary":"","title":"C++","type":"tags"},{"content":" D5 Render # D5 Render is a cutting-edge, real-time 3D rendering software designed to provide architects, designers, and visualization professionals with a seamless and efficient tool for creating photorealistic visuals. Its intuitive interface, advanced rendering engine, and versatile features make it an excellent choice for professionals seeking high-quality results with minimal learning curve.\nUser Case # Most users typically work with various 3D modeling software to create and save their 3D model data, which they then import into D5 Render for real-time rendering to observe the realistic effects of lighting. In the architecture industry, frequent modifications to objects are often required, such as removing a landscape element or resizing certain features. This creates a significant pain point for users, as they need to constantly switch between the rendering software and the 3D modeling software.\nSolution # To address this issue, we developed a plugin for D5 Render that allows users to directly edit 3D model data in Rhino. This plugin enables users to make changes to their 3D models in Rhino and immediately see the results in D5 Render, without the need to switch between the two software.\nVideo # My Role # I was responsible for the overall design and implementation of the plugin, including the development of the plugin\u0026rsquo;s core functionality, the integration of the plugin with D5 Render, and the testing and validation of the plugin\u0026rsquo;s performance.\nTechnology # C++ RhinoSDK D5 Render Result # From zero, I built and released the beta version of this plugin in 2 months. And now it is used by many users.\n","date":"January 4, 2025","externalUrl":null,"permalink":"/en/projects/1735963425218-d5-render-rhino-sync-plugin/","section":"Projects","summary":"A rapid plugin for D5 Render-Rhino.","title":"D5 Render-Rhino Sync Plugin","type":"projects"},{"content":"","date":"January 4, 2025","externalUrl":null,"permalink":"/en/tags/plugin/","section":"Tags","summary":"","title":"Plugin","type":"tags"},{"content":"","date":"2024年1月15日","externalUrl":null,"permalink":"/tags/%E3%82%BF%E3%82%B0/","section":"Tags","summary":"","title":"タグ","type":"tags"},{"content":"","date":"2024年1月15日","externalUrl":null,"permalink":"/tags/%E4%BE%8B/","section":"Tags","summary":"","title":"例","type":"tags"},{"content":"","date":"2023年1月15日","externalUrl":null,"permalink":"/tags/neural-search/","section":"Tags","summary":"","title":"Neural Search","type":"tags"},{"content":"","date":"2023年1月15日","externalUrl":null,"permalink":"/tags/two-phase-search/","section":"Tags","summary":"","title":"Two Phase Search","type":"tags"},{"content":" About Me # I was born in Shandong Province and grew up in Tianjin, China. After spending seven years studying in Nanjing, I came to Japan to pursue my PhD degree.\nI\u0026rsquo;m a big fan of Japanese curry and Sukiyaki. In my free time, I enjoy playing soccer and listening to Japanese rock music, especially Blue Hearts and X Japan - two of my all-time favorite bands.\nTimeline # SouthEast University 2016-2020 China Software Engineering I was a student at Southeast University, where I received my Bachelor's degree in Software Engineering. I like the library of this university which have a very big lake in the front. I studied the basic courses of computer science, such as data structure, computer network, operating system, etc. Nanjing University 2020-2023 China Software Engineering In this period, I was a student at Nanjing University, where I received my Master's degree in Software Engineering. Mainly focused on the application of machine learning in software engineering. AWS 2023-2024 China Software Engineer I worked as a software engineer at Amazon Web Services, where I was responsible for the development of the AWS OpenSearch service. I love the teammate and the working environment in Shanghai. Waseda University 2023-2025 Japan Computer Science I'm a student at Waseda University, where I'm pursuing my PhD degree in Computer Science. I'm working with Prof. Yoshie Osamu. ","externalUrl":null,"permalink":"/en/about/","section":"","summary":"\u003ch2 class=\"relative group\"\u003eAbout Me\n    \u003cdiv id=\"about-me\" class=\"anchor\"\u003e\u003c/div\u003e\n    \n    \u003cspan\n        class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none\"\u003e\n        \u003ca class=\"text-primary-300 dark:text-neutral-700 !no-underline\" href=\"#about-me\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\n    \u003c/span\u003e\n    \n\u003c/h2\u003e\n\u003cp\u003eI was born in Shandong Province and grew up in Tianjin, China. After spending seven years studying in Nanjing, I came to Japan to pursue my PhD degree.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m a big fan of Japanese curry and Sukiyaki. In my free time, I enjoy playing soccer and listening to Japanese rock music, especially Blue Hearts and X Japan - two of my all-time favorite bands.\u003c/p\u003e","title":"","type":"about"},{"content":"","externalUrl":null,"permalink":"/en/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"","externalUrl":null,"permalink":"/en/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","externalUrl":null,"permalink":"/en/series/","section":"Series","summary":"","title":"Series","type":"series"}]