Notice

Title: Empathetic Response in Audio-Visual Conversations Using Emotion Preference Optimization and MambaCompressor

Authors: Yeonju Kim, Se Jin Park, Yong Man Ro

Chatbot research is advancing with the growing importance of chatbots in fields that require human interactions, such as customer support and mental health care. Despite these advancements, chatbots still face significant challenges in understanding subtle nuances and managing long conversation histories. To address these issues, our study introduces a dual approach: firstly, we employ Emotional Preference Optimization (EPO) to train chatbots not only with correct responses but also with counter-emotional responses-those that are contextually similar but emotionally divergent. This training enables the model to discern fine nuance distinctions between correct and counter-emotional responses, thereby enhancing the quality of its responses. Secondly, we introduce MambaCompressor to effectively compress and manage extensive conversation histories, significantly reducing time and memory complexities while improving the chatbot's contextual understanding. Our comprehensive experiments across multiple datasets demonstrate that our model significantly outperforms existing models in generating empathetic responses and efficiently managing lengthy dialogues.

Title: STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding 

Authors: Junho Kim, Hosu Lee, James Matthew Rehg, Minsu Kim, Yong Man Ro

Recent progress in video large language models (Video-LLMs) has enabled strong offline reasoning over long and complex videos. However, real-world deployments increasingly require streaming perception and proactive interaction, where video frames arrive online and the system must decide not only what to respond, but also when to respond. In this work, we revisit proactive activation in streaming video as a structured sequence modeling problem, motivated by the observation that temporal transitions in streaming video naturally form span-structured activation patterns. To capture this span-level structure, we model activation signals jointly over a sliding temporal window and update them iteratively as new frames arrive. We propose STRIDE (Structured Temporal Refinement with Iterative DEnoising), which employs a lightweight masked diffusion module at the activation interface to jointly predict and progressively refine activation signals across the window. Extensive experiments on diverse streaming benchmarks and downstream models demonstrate that STRIDE shows more reliable and temporally coherent proactive responses, significantly improving "when-to-speak" decision quality in online streaming scenarios.

Title: GenRecal: Generation after Recalibration from Large to Small Vision-Language Models 

Authors: Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Yu-Chiang Frank Wang, Yueh-Hua Wu

Recent advancements in vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V. However, deploying these models in real-world scenarios, particularly on resource-constrained devices, remains challenging due to their substantial computational demands. This has spurred interest in distilling knowledge from large VLMs into smaller, more efficient counterparts. A key challenge arises here from the diversity of VLM architectures, which are built on different LLMs and employ varying token types—differing in vocabulary size, token splits, and token index ordering. To address this challenge of limitation to a specific VLM type, we present Generation after Recalibration (GenRecal), a general-purpose distillation framework for VLMs. GenRecal incorporates a Recalibrator that aligns and adapts feature representations between heterogeneous VLMs, enabling effective knowledge transfer across different types of VLMs. Through extensive experiments on multiple challenging benchmarks, we demonstrate that GenRecal significantly improves baseline performances, eventually outperforming large-scale open- and closed-source VLMs.

Title: Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking

Authors: Sangyun Chung*, Youngjoon Yu*, Se Yeon Kim, Youngchae Chee, Yong Man Ro (*equal contribution)

Large-scale Vision-Language Models (VLMs) have achieved notable progress in aligning visual inputs with text. However, their ability to deeply understand the unique physical properties of non-RGB vision sensor images remains limited. In this paper, we revisit and analyze these limitations and introduce a novel, cost-efficient paradigm that significantly advances sensor image understanding-without requiring extensive training data or any modifications to the existing VLM architectures. Specifically, we propose Sensor-Aware Attributes Fine-Tuning (SAFT) with the Diverse Negative Attributes (DNA) optimization, which leverages minimal sensor-specific data to enable robust learning of non-RGB characteristics and overcome RGB-centric biases inherent in current VLMs. In addition, we present VS-TDX-the first comprehensive, public benchmark designed to rigorously evaluate VLMs' sensor-specific understanding across diverse and realistic scenarios. Through extensive experiments on VLMs and various sensor modalities, we validate that our method consistently delivers superior performance and generalization under resource-constrained and architecture-invariant settings. Our approach provides a practical advance towards scalable deployment of VLMs in increasingly sensor-diverse real-world environments.

Hyun Jun Kim, a PhD candidate in Prof. Yong Man Ro’s lab, has secured a research internship at Google in Japan. Google is globally recognized as a leading institution in cutting‑edge AI and Multimodal Large Language Model (MLLM) research. This internship will provide Hyun Jun with an opportunity to further advance his ongoing PhD research in MLLMs and Computer Vision.

He has previously published notable papers at top‑tier AI conferences—including CVPR, NeurIPS, AAAI, and ECCV—focusing on enhancing the robustness of MLLMs, long‑form video understanding (e.g., SALOVA at CVPR 2025), and robust object detection in challenging environments.

As Hyun Jun approaches the completion of his PhD, this internship will build upon his prior research experiences and further deepen his doctoral work. It is expected to strengthen his global competitiveness for future roles in leading AI and technology institutions after completing his PhD.

Prof. Yong Man Ro has been appointed as a member of the Presidential Committee on Future Defense Strategy. As an expert in Multimodal AI, he will provide policy advice on national defense utilizing advanced artificial intelligence technologies. The photograph captures the inaugural meeting of the Presidential Committee on Future Defense Strategy, which was released to the media.

Title: GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memory 

Jeong Hun Yeo*, Sangyun Chung*, Sungjune Park, Dae Hoe Kim, Jinyoung Moon, and Yong Man Ro (*equal contribution)

Long-video understanding remains a significant challenge for Multimodal Large Language Models (MLLMs) due to inherent token limitations and the complexity of capturing long-term temporal dependencies. Existing methods often fail to capture the global context and complex event relationships necessary for deep video reasoning. To address this, we introduce GCAgent, a novel Global-Context-Aware Agent framework that achieves comprehensive long-video understanding. Our core innovation is the Schematic and Narrative Episodic Memory. This memory structurally models events and their causal and temporal relations into a concise, organized context, fundamentally resolving the long-term dependency problem. Operating in a multi-stage Perception-Action-Reflection cycle, our GCAgent utilizes a Memory Manager to retrieve relevant episodic context for robust, context-aware inference. Extensive experiments confirm that GCAgent significantly enhances long-video understanding, achieving up to 23.5% accuracy improvement on the Video-MME Long split over a strong MLLM baseline. Furthermore, our framework establishes state-of-the-art performance among comparable 7B-scale MLLMs, achieving 73.4% accuracy on the Long split and the highest overall average (71.9%) on the Video-MME benchmark, validating our agent-based reasoning paradigm and structured memory for cognitively-inspired long-video understanding.

We are proud to announce that Jeong Hun Yeo has been selected for the prestigious 2026 Jang Young Sil Fellow Program (Excellence Track).

Supported by KAIST, this fellowship is an exclusive honor awarded only to a highly select group of top-tier postdoctoral researchers. This recognition highlights Jeong Hun’s exceptional potential and academic brilliance. Through this program, he will receive dedicated research support and funding to further his excellence within our laboratory.

Professor Ro and all members of the IVL laboratory join together in congratulating Jeong Hun on this achievement. 

We are proud of his dedication and look forward to his continued success and groundbreaking contributions to the field. Congratulations, Jeong Hun.

Title: MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Sangyun Chung, Se Yeon Kim, Youngchae Chee, and Yong Man Ro

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a trainingfree method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model’s inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8% and 2.0% improvements for VideoLLaMA2-AV, 8.7% and 4.7% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at https://github.com/top-yun/MAD

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), CVPR 2026

Title: Recursive Think-Answer Process for LLMs and VLMs

Byung-Kwan Lee,  Youngchae Chee, Yong Man Ro

Think–Answer reasoners such as DeepSeek-R1 have made notable progress by leveraging interpretable internal reasoning. However, despite the frequent presence of self reflective cues like “Oops!”, they remain vulnerable to output errors during single-pass inference. To address this limitation, we propose an efficient Recursive Think–Answer Process (R-TAP) that enables models to engage in iterative reasoning cycles and generate more accurate answers, going beyond conventional single-pass approaches. Central to this approach is a confidence generator that evaluates the certainty of model responses and guides subsequent improvements. By incorporating two complementary rewards—Recursively Confidence Increase Reward and Final Answer Confidence Reward—we show that R-TAP-enhanced models consistently outperform conventional single-pass methods for both large language models (LLMs) and vision-language models (VLMs). Moreover, by analyzing the frequency of “Oops”-like xpressions in model responses, we find that R-TAP–applied models exhibit significantly fewer self-reflective patterns, resulting in more stable and faster inference-time reasoning. We hope R-TAP pave the way evolving into efficient and elaborated methods to refine the reasoning processes of future AI.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), CVPR 2026

Title: ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

Hosu Lee, Junho Kim, Hyunjun Kim, Yong Man Ro

Recent progress in Large Multi-modal Models (LMMs) has enabled effective vision-language reasoning, yet the ability to video understanding remains constrained by suboptimal frame selection strategies, albeit with the rapid development of video-specialized LMMs. Prior works attempted to solve this with static heuristics or external retrieval modules to feed frame-level information, but these approaches often fail to capture visual cues grounded to the given user queries conflating raw visual dynamics with true semantic relevance. In this paper, we introduce ReFoCUS (Reinforcement-guided Frame Optimization for Contextual UnderStanding), the first framework to integrate online policy-gradient reinforcement learning into frame-level optimization for video-LLMs. ReFoCUS aims to learn a frame selection policy, leveraging reward signals derived from reference models to capture their underlying scoring behavior over frame combinations that best support temporally  grounded responses. To efficiently explore the large combinatorial frame space, we employ an autoregressive and  query-conditional selection architecture that ensures con textual consistency while reducing complexity. Our policy  learning removes the need for explicit frame-level super vision, as it implicitly discovers optimal and semantically  consistent frame compositions. ReFoCUS consistently improves reasoning accuracy across multiple video QA benchmarks, demonstrating the advantage of aligning frame selection with model-internal utility.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), CVPR 2026

Se Jin Park, a doctoral researcher from Professor Yong Man Ro’s laboratory, received his Ph.D. in February 2026 and has been appointed as an Assistant Professor in the Department of Electronic Engineering at Kyung Hee University as of March 2026. Throughout her doctoral studies, Park has been conducting research on multimodal artificial intelligence that integrates speech, vision, and language, with the goal of enabling natural and seamless interaction between humans and AI.

Park has been developing methods for visual–acoustic representation learning, modeling long- and short-term conversational context, and leveraging both linguistic and nonverbal cues from human interaction for dialogue understanding and generation. Her research achievements have been recognized internationally. She has presented a total of 13 papers at top-tier conferences such as ICML, ACL, CVPR, AAAI, and ICASSP, and her work has been selected for several prestigious distinctions, including the ACL Outstanding Paper Award, ICML Oral, CVPR Highlight, ACL Oral, and AAAI Oral. Through these accomplishments, Park has established herself as a competitive researcher in the fields of multimodal AI and conversational intelligence.

Park has expressed her intention to continue pursuing research on conversational intelligence that enables AI systems to collaborate and communicate effectively with real users in complex interaction environments that combine speech, vision, and language. Our school sincerely congratulates her on this new beginning and looks forward to her future contributions in education, research, and industry collaboration at Kyung Hee University.

Title: Dr. Hong Joo Lee (Advisor: Prof. Yong Man Ro) appointed as Assistant Professor at Seoul National University of Science and Technology.

Dr. Hong Joo Lee, an alumnus of the School of Electrical Engineering at KAIST (Advisor: Prof. Yong Man Ro), has been appointed as an Assistant Professor in the Department of Applied Artificial Intelligence at Seoul National University of Science and Technology, effective March 1, 2026.

Dr. Lee earned his Ph.D. with a dissertation titled "Investigating Adversarial Robustness via Booster Signal." During his doctoral studies, he participated in the Center for Applied Research in Artificial Intelligence (CARAI) for National Defense Research. His research has been widely recognized through numerous publications in top-tier conferences and journals, including CVPR, IEEE TIP, and IEEE TNNLS.

After receiving his doctorate in 2023, Dr. Lee served as a postdoctoral researcher at the Technical University of Munich (TUM) in Germany. His postdoctoral work focused on the reliability of AI models in the medical field, leading to further high-impact publications in ECCV, MICCAI, and AAAI.

In his new role as a professor, Dr. Lee plans to deepen his research on Reliable Intelligence Systems, focusing on the vulnerability, safety, and fairness of AI models.

Title: Robust Grounding with MLLMs against Occlusion and Small Objects via Language-Guided Semantic Cues

Beomchan Park*, Seongho Kim*, Hyunjun Kim, Sungjune Park, Yong Man Ro (*equal contribution)

While Multimodal Large Language Models (MLLMs) have enhanced grounding capabilities in general scenes, their robustness in crowded scenes remains underexplored. Crowded scenes entail visual challenges (i.e., occlusion and small objects), which impair object semantics and degrade grounding performance. In contrast, language expressions are immune to such degradation and preserve object semantics. In light of these observations, we propose a novel method that overcomes such constraints by leveraging Language-Guided Semantic Cues (LGSCs). Specifically, our approach introduces a Semantic Cue Extractor (SCE) to derive semantic cues of objects from the visual pipeline of an MLLM. We then guide these cues using corresponding text embeddings to produce LGSCs as linguistic semantic priors. Subsequently, they are reintegrated into the original visual pipeline to refine object semantics. Extensive experiments and analyses demonstrate that incorporating LGSCs into an MLLM effectively improves grounding accuracy in crowded scenes.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICASSP 2026

Title: A Causal Lens on Non-RGB Vision Sensor Understanding in Vision Language Models

Youngjoon Yu, Yong Man Ro

Language Models (VLMs) have achieved remarkable success in tasks involving natural RGB images, their capability to understand non-RGB sensor data, including thermal, depth, hyperspectral, and X-ray imagery, remains severely limited. This limitation stems from an entrenched RGB-centric bias, leading current VLMs to treat these distinct modalities as ordinary photographs, thus failing to account for their unique physical properties. To systematically evaluate and address this pervasive issue, we present CausalSense, a novel benchmark suite designed to expose RGB-centric bias within large-scale VLMs using non-RGB sensor data. Concurrently, we devise a causal learning framework specifically engineered to alleviate this RGB-bounded bias. Our approach effectively employs confounder dictionaries and backdoor adjustments from causal inference to integrate essential sensor-specific knowledge into VLMs, circumventing the need for extensive retraining on massive datasets. Our comprehensive evaluations using CausalSense underscore a significant performance deficiency in state-of-the-art VLMs concerning non-RGB vision sensor comprehension. Crucially, we demonstrate that our proposed causal deconfounded cross-modal encoder substantially improves VLMs’ ability to reason about the physical attributes captured by these modalities, thereby achieving a measurable reduction in the observed performance gap. This combined benchmark and framework pave the way for developing more resilient and sensor-aware vision–language models, capable of robustly interpreting diverse real-world phenomena beyond the visible spectrum.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), IEEE TIP

Title: Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier

Hyeongseop Rha, Jeong Hun Yeo, KIM YEONJU, Yong Man Ro 

The recent advancement of Multimodal Large Language Models (MLLMs) is transforming human-computer interaction (HCI) from surface-level exchanges into more nuanced and emotionally intelligent communication. To realize this shift, emotion understanding becomes essential allowing systems to capture subtle cues underlying user intent. Furthermore, providing faithful explanations for predicted emotions is crucial to ensure interpretability and build user trust. However, current MLLM-based methods often generate emotion explanations that diverge from the ground-truth (GT) labels and sometimes even contradict their own predicted emotions. This inconsistency poses a critical risk for misunderstanding and erodes reliability in interactive settings. To address this, we propose a novel approach: the Emotional Rationale Verifier (ERV) and an Explanation Reward. Our method guides the model to produce reasoning that is explicitly consistent with the GT emotion during multimodal emotion recognition without modifying the model architecture or requiring paired video–description annotations. Our method significantly improves faithful explanation–prediction consistency and explanation emotion accuracy on the MAFW and DFEW datasets. Through extensive experiments and human evaluations, we show that our approach not only enhances alignment between explanation and prediction but also empowers MLLMs to deliver emotionally coherent, trustworthy interactions, marking a key step toward truly human-like HCI systems.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), AAAI 2026

전기및전자공학부 합격생중에서  국비 석사 2명, KAIST 장학생 (석사, 박사),  KAIST 프로그램(KEPSI, EPSS, LGenius, EPSD) 을 모집합니다.

김재철 AI 대학원 합격생중에서 KAIST 장학생 (석사) 을 모집합니다.

학생 초청 연구분야는 MLLM 을 기반으로 비전, 멀티모달분야 입니다.

MLLM (Multimodal large language model) +  (Vision, Audio, Language)

Integration Vision, Language and Audio

많은 학생들의 관심과 문의 감사드립니다.    

IVL 연구실에서 MLLM+ 를 연구하고자 하는 학생은 합격 후 교수님(ymro@kaist.ac.kr)에게 면담신청 하시기 바랍니다.

We are pleased to announce that Professor Yong Man Ro's project, Inclusive Multimodal LLM for Vocal and Non-Vocal Human Communication, has been selected for the NVIDIA Academic Grant Program.

As part of this award, KAIST has received 32,000 A100 GPU-hours on Brev to support the research. This grant will accelerate our efforts in developing lightweight, inclusive multimodal language models.

We thank NVIDIA for their support and look forward to sharing impactful results from this collaboration.

Title: Unified Reinforcement and Imitation Learning for Vision-Language Models

Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Frank Wang, Yueh-Hua Wu

Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for resource-constrained environments. This paper introduces Unified Reinforcement and Imitation Learning (RIL), a novel and efficient training algorithm designed to create powerful, lightweight VLMs. RIL distinctively combines the strengths of reinforcement learning with adversarial imitation learning. This enables smaller student VLMs not only to mimic the sophisticated text generation of large teacher models but also to systematically improve their generative capabilities through reinforcement signals. Key to our imitation framework is a LLM-based discriminator that adeptly distinguishes between student and teacher outputs, complemented by guidance from multiple large teacher VLMs to ensure diverse learning. This unified learning strategy, leveraging both reinforcement and imitation, empowers student models to achieve significant performance gains, making them competitive with leading closed-source VLMs. Extensive experiments on diverse vision-language benchmarks demonstrate that RIL significantly narrows the performance gap with state-of-the-art open- and closed-source VLMs and, in several instances, surpasses them.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), NeurIPS 2025

Title: Adaptive Integration of Textual Context and Visual Embeddings for Underrepresented Vision Classification

Seongyeop Kim, Hyung-Il Kim, Yong Man Ro

The challenge of long-tail distribution in image classification, particularly where data is scarce, is significant. Traditional research on long-tail visual classification has primarily focused on visual data, frequently overlooking how textual information can enhance performance by providing rich contextual details. This is particularly significant for rare classes, where visual data alone may be insufficient to support effective inference during training. This research introduces a novel approach to long-tail visual classification by integrating textual data with visual inputs using advanced language models and the newly designed Adaptive Integration Block for Vision-Text Synergy (AIB-VTS). We explore the latest advancements in vision-language models that combine both image and textual data. AIB-VTS, integrated within Vision Transformer architectures, adaptively adjusts the balance between visual representation and textual context representation, improving accuracy for underrepresented classes. By utilizing large language models (LLMs) for generating contextual descriptions, we enrich the dataset for tail classes with limited visual data. Our results show notable enhancements in classification robustness and accuracy across the long-tail distribution. The carefully designed AIB-VTS module enhances the vision classification model by adaptively leveraging textual context information learned during training. By querying image representation to AIB-VTS, it actively pulls relevant textual context, effectively addressing data scarcity and minimizing interference from textual information in well-represented head classes.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), Pattern Recognition

Title: Causal Unsupervised Semantic Segmentation

Junho Kim*, Byung-Kwan Lee*, Yong Man Ro (*equal contributor)

Unsupervised semantic segmentation aims to achieve high-quality semantic grouping without human-labeled annotations. With the advent of self-supervised pre-training, various frameworks utilize the pre-trained features to train prediction heads for unsupervised dense prediction. However, a significant challenge in this unsupervised setup is determining the appropriate level of clustering required for segmenting concepts. To address it, we propose a novel framework, CAusal Unsupervised Semantic sEgmentation (CAUSE), which leverages insights from causal inference. Specifically, we bridge intervention-oriented approach (i.e., frontdoor adjustment) to define suitable two-step tasks for unsupervised prediction. The first step involves constructing a concept clusterbook as a mediator, which represents possible concept prototypes at different levels of granularity in a discretized form. Then, the mediator establishes an explicit link to the subsequent concept-wise self-supervised learning for pixel-level grouping. Through extensive experiments and analyses on various datasets, we corroborate the effectiveness of CAUSE and achieve state-of-the-art performance in unsupervised semantic segmentation.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), Pattern Recognition

Title: Closing the Modality Gap: Integrating LLMs with LIDAR for 3D Object Detection and Object-level Understanding

Youngchae Chee, Taeheon Kim, Youngjoon Yu, HyunWook Park, and Yong Man Ro

Multimodal Large Language Models (MLLMs), which integrate multiple modalities such as vision and language by leveraging Large Language Models (LLMs), have demonstrated remarkable progress in multimodal understanding. However, the incorporation of LiDAR—a key modality for spatial and geometric comprehension—into multimodal LLMs (MLLMs) remains underdeveloped. In this work, we devise a novel framework that enhances MLLMs with object-level LiDAR comprehension by jointly aligning both scene-level and region-of-interest (ROI) LiDAR features with natural language. Our method includes three components: scene-level LiDAR compression, ROI-based extraction of object-level features, and multimodal alignment with language. To support accurate object-level understanding, we propose two complementary tasks: LiDAR-to-Class (predicting object classes from LiDAR features) and Class-to-LiDAR (identifying LiDAR objects from natural language descriptions). Leveraging the large-scale NuScenes dataset with over one million annotated objects, our model achieves strong performance, including 95% accuracy on the Class-to-LiDAR task. Leveraging LiDAR data, this work advances precise object-level semantic understanding in 3D environments and underscores the potential of MLLMs in safety-critical and interactive scenarios such as autonomous driving and robotics.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICIP 2025

Title: Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations

Minho Park, Young Joo Jo, Jae-Hyeok Lee, Ji Yong Lee, Dong-oh Kang, Yong Man Ro

Instruction-based image editing enables intuitive modifications of images through natural language descriptions.  However, existing models often struggle to accurately identify the target region, which refers to the area that should be modified. As a result, unintended changes may occur in non-target areas, where the original image should remain unchanged. To address this issue, we propose FoRE, an MLLM-guided framework that identifies the target region based on the given edit instruction and performs image editing using region-aware embeddings. Within FoRE, the Region-guided Edit Adapter projects these embeddings from the MLLM domain to the diffusion condition space.

Subsequently, the Region-guided Refinement Module refines the projected features to enhance spatial accuracy prior to guiding the diffusion process. Through comprehensive evaluations, we demonstrate that FoRE significantly improves localization accuracy and instruction fidelity compared to existing approaches. By explicitly incorporating region-aware conditioning, our framework effectively bridges the gap between instruction comprehension and spatially precise image modifications, advancing the capabilities of instruction-based image editing.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ACM MM 2025

Title: Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations

Jeong Hun Yeo, Minsu Kim, Chae Won Kim, Stavros Petridis, Yong Man Ro

We explore a novel zero-shot Audio-Visual Speech Recognition (AVSR) framework, dubbed Zero-AVSR, which enables speech recognition in target languages without requiring any audio-visual speech data in those languages. Specifically, we introduce the Audio-Visual Speech Romanizer (AV-Romanizer), which learns language-agnostic speech representations by predicting Roman text. Then, by leveraging the strong multilingual modeling capabilities of Large Language Models (LLMs), we propose converting the predicted Roman text into language-specific graphemes, forming the proposed Cascaded Zero-AVSR. Taking it a step further, we explore a unified Zero-AVSR approach by directly integrating the audio-visual speech representations encoded by the AV-Romanizer into the LLM. This is achieved through finetuning the adapter and the LLM using our proposed multi-task learning scheme. To capture the wide spectrum of phonetic and linguistic diversity, we also introduce a Multilingual Audio-Visual Romanized Corpus (MARC) consisting of 2,916 hours of audio-visual speech data across 82 languages, along with transcriptions in both language-specific graphemes and Roman text. Extensive analysis and experiments confirm that the proposed Zero-AVSR framework has the potential to expand language support beyond the languages seen during the training of the AV-Romanizer.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICCV 2025

Junho Kim has successfully defended his Ph.D. and is set to join the Siebel School of Computing and Data Science at the University of Illinois Urbana-Champaign (UIUC) as a postdoctoral researcher. His move marks a forward step in his burgeoning career in artificial intelligence research.

During his doctoral studies, Junho's research focused on enhancing the interpretability and robustness of AI models. He developed novel perturbation-based methods for explaining the behavior of blackbox AI models. A key contribution of his work involved a comprehensive analysis of AI models' vulnerabilities to adversarial perturbations, leading to the successful distinction between robust and non-robust features. These significant findings were published in AI Top-tier venues such as IEEE Transactions on Image Processing (TIP) and the Conference on Neural Information Processing Systems (NeurIPS).

More recently, Junho's research addressed the pressing issue of hallucinations in Multimodal Large Language Models (MLLMs). He proposed innovative mitigation strategies to alleviate these issues in large-scale models, introducing techniques based on counterfactual approaches and decoding-time interventions. This work was presented at the Conference on Computer Vision and Pattern Recognition (CVPR).

Junho's transition to UIUC is anticipated to further deepen his contributions to AI research, reinforcing his position as an emerging international scholar in the field.

Se Jin Park, a PhD student in Human Multimodal LLMs from the IVL lab, has secured a research internship at Meta in the USA. Meta is recognized as a leading institution in the field of LLM research.

This internship will provide Se Jin with the opportunity to enhance her ongoing PhD research in human multimodal LLMs. She has previously published notable papers on human multimodal AI, focusing on processing and understanding human-relevant modalities such as spoken language and human speech.

As Se Jin approaches the completion of her PhD, this internship is intended to build upon her prior research experiences. The aim is to expand and deepen her doctoral work, which is expected to strengthen her global competitiveness for future roles in leading AI institutions post-PhD.

Title: TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages

Minsu Kim*, Jee-weon Jung*, Hyeongseop Rha, Soumi Maiti, Siddhant Arora, Xuankai Chang, Shinji Watanabe, Yong Man Ro (*equal contribution)

The capability to jointly process multi-modal information is becoming an essential task. However, the limited number of paired multi-modal data and the large computational requirements in multi-modal learning hinder the development. We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text. We introduce a novel viewpoint, where we interpret different modalities as different languages, and treat multi-modal translation as a well-established machine translation problem. To this end, we tokenize speech and image data into discrete tokens, which provide a unified interface across modalities and significantly decrease the computational cost. In the proposed TMT, a multi-modal encoder-decoder conducts the core translation, whereas modality-specific processing is conducted only within the tokenization and detokenization stages. We evaluate the proposed TMT on all six modality translation tasks. TMT outperforms single model counterparts consistently, demonstrating that unifying tasks is beneficial not only for practicality but also for performance.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), IEEE Transactions on Multimedia

Title: MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens

Jeong Hun Yeo*, Hyeongseop Rha*, Se Jin Park, Yong Man Ro (* equal contribution)

Audio-Visual Speech Recognition (AVSR) achieves robust speech recognition in noisy environments by combining auditory and visual information. However, recent Large Language Model (LLM) based AVSR systems incur high computational costs due to the high temporal resolution of audio-visual speech processed by LLMs. In this work, we introduce an efficient multimodal speech LLM framework that minimizes token length while preserving essential linguistic content. Our approach employs an early av-fusion module for streamlined feature integration, an audio-visual speech Q-Former that dynamically allocates tokens based on input duration, and a refined query allocation strategy with a speech rate predictor to adjust token allocation according to speaking speed of each audio sample. Extensive experiments on the LRS3 dataset show that our method achieves state-of-the-art performance with a WER of 0.74% while using only 3.5 tokens per second. Moreover, our approach not only reduces token usage by 86% compared to the previous multimodal speech LLM framework, but also improves computational efficiency by reducing FLOPs by 35.7%.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ACL 2025

Title: Long-Form Speech Generation with Spoken Language Models

Se Jin Park, Julian Salazar, Aren Jansen, Keisuke Kinoshita, Yong Man Ro, RJ Skerry-Ryan

We consider the generative modeling of speech over multiple minutes, a requirement for long-form multimedia generation and audio-native voice assistants. However, current spoken language models struggle to generate plausible speech past tens of seconds, from high temporal resolution of speech tokens causing loss of coherence, to architectural issues with long-sequence training or extrapolation, to memory costs at inference time. With these considerations we propose SpeechSSM, the first speech language model to learn from and sample long-form spoken audio (e.g., 16 minutes of read or extemporaneous speech) in a single decoding session without text intermediates, based on recent advances in linear-time sequence modeling. Furthermore, to address growing challenges in spoken language evaluation, especially in this new long-form setting, we propose: new embedding-based and LLM-judged metrics; quality measurements over length and time; and a new benchmark for long-form speech processing and generation, LibriSpeech-Long. Speech samples and the dataset are released at this https URL.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICML 2025

국비 석사 1명, KAIST 장학생 (석사, 박사),  KAIST 프로그램(KEPSI, EPSS, LGenius, EPSD) 등 학생을 모집합니다.

학생 초청 연구분야는 MLLM 을 기반으로 비전, 멀티모달분야 입니다.

MLLM (Multimodal large language model) +  (Vision, Audio, Language)

Integration Vision, Language and Audio

많은 학생들의 관심과 문의 감사드립니다.    

IVL 연구실에서 MLLM+ 를 연구하고자 하는 학생은 합격후 교수님(ymro@kaist.ac.kr)에게 면담신청 하시기 바랍니다.

IVL Lab proudly announces that two of PhD graduates, Dr. Minsu Kim and Dr. Joanna Hong, have been recruited by DeepMind, a world leader in artificial intelligence research. Both Dr. Kim and Dr. Hong were instrumental in pioneering Human Multimodal research during their time at IVL Lab, making significant strides in the field.

We are incredibly proud of Dr. Kim and Dr. Hong's achievements, said Prof. YM Ro, director of IVL Lab. Their dedication and innovative research have left a lasting impact on IVL lab, and we are excited to see the contributions they will make at DeepMind."

Dr. Kim and Dr. Hong's work at IVL Lab focused on Human multimodality transformation, e.g., developing advanced multimodal learning models, exploring human-computer interaction through AI. Their research has been published in top-tier conferences and journals, showcasing the cutting-edge work produced at  IVL Lab. You can explore their research further:

Dr. Kim's Research: https://sites.google.com/view/ms-dot-k 

Dr. Hong's Research: https://joannahong.github.io/ 

This achievement highlights IVL Lab's commitment to fostering top talent and conducting impactful research in the field of artificial intelligence. We wish Dr. Kim and Dr. Hong all the best in their new endeavors at DeepMind and look forward to their future contributions to the AI community.

Title: SALOVA: Segment-Augmented Long Video Assistance for Targeted Retrieval and Routing in Long-Form Video Analysis

Junho Kim*, Hyunjun Kim*, Hosu Lee, Yong Man Ro (* equal contributor)

Despite advances in Large Multi-modal Models, applying them to long and untrimmed video content remains challenging due to limitations in context length and substantial memory overhead. These constraints often lead to significant information loss and reduced relevance in the model responses. With the exponential growth of video data across web platforms, understanding long-form video is crucial for advancing generalized intelligence. In this paper, we introduce SALOVA: Segment-Augmented LOng Video Assistant, a novel video-LLM framework designed to enhance the comprehension of lengthy video content through targeted retrieval process. We address two main challenges to achieve it: (i) We present the SceneWalk dataset, a high-quality collection of 87.8K long videos, each densely captioned at the segment level to enable models to capture scene continuity and maintain rich descriptive context. (ii) We develop robust architectural designs integrating dynamic routing mechanism and spatio-temporal projector to efficiently retrieve and process relevant video segments based on user queries. Our framework mitigates the limitations of current video-LMMs by allowing for precise identification and retrieval of relevant video segments in response to queries, thereby improving the contextual relevance of the generated responses. Through extensive experiments, SALOVA demonstrates enhanced capability in processing complex long-form videos, showing significant capability to maintain contextual integrity across extended sequences.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), CVPR 2025

Title: VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models

Byung-Kwan Lee, Ryo Hachiuma, Yu-Chiang Frank Wang, Yong Man Ro, Yueh-Hua Wu

The recent surge in high-quality visual instruction tuning samples from closed-source vision-language models (VLMs) such as GPT-4V has accelerated the release of open-source VLMs across various model sizes. However, scaling VLMs to improve performance using larger models brings significant computational challenges, especially for deployment on resource-constrained devices like mobile platforms and robots. To address this, we propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which prioritizes efficiency without compromising accuracy. VLsI leverages a unique, layer-wise distillation process, introducing intermediate "verbalizers" that map features from each layer to natural language space, allowing smaller VLMs to flexibly align with the reasoning processes of larger VLMs. This approach mitigates the training instability often encountered in output imitation and goes beyond typical final-layer tuning by aligning the small VLMs’ layer-wise progression with that of the large ones. We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling, merging, or architectural changes.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), CVPR 2025

Title: Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language

Jeong Hun Yeo, Chae Won Kim, Hyunjun Kim, Hyeongseop Rha, Seunghee Han, Wen-Huang Cheng, Yong Man Ro

Lip reading aims to predict spoken language by analyzing lip movements. Despite advancements in lip reading technologies, performance degrades when models are applied to unseen speakers due to their sensitivity to variations in visual information such as lip appearances. To address this challenge, speaker adaptive lip reading technologies have advanced by focusing on effectively adapting a lip reading model to target speakers in the visual modality. The effectiveness of adapting language information, such as vocabulary choice, of the target speaker has not been explored in the previous works. Moreover, existing datasets for speaker adaptation have limited vocabulary size and pose variations, limiting the validation of previous speaker-adaptive methods in real-world scenarios. To address these issues, we propose a novel speaker-adaptive lip reading method that adapts a pre-trained model to target speakers at both vision and language levels. Specifically, we integrate prompt tuning and the LoRA approach, applying them to a pre-trained lip reading model to effectively adapt the model to target speakers. In addition, to validate its effectiveness in real-world scenarios, we introduce a new dataset, VoxLRS-SA, derived from VoxCeleb2 and LRS3. It contains a vocabulary of approximately 100K words, offers diverse pose variations, and enables the validation of adaptation methods in wild, sentence-level lip reading for the first time. Through various experiments, we demonstrate that the existing speaker-adaptive method also improves performance in the wild at the sentence level. Moreover, with the proposed adaptation method, we show that the proposed method achieves larger improvements when applied to the target speaker, compared to the previous works.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), AAAI 2025

Title: Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition 

Minsu Kim, Hyeong-Il Kim, Yong Man Ro

Visual Speech Recognition (VSR) aims to infer speech into text depending on lip movements alone. As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements, and this makes the VSR models show degraded performance when they are applied to unseen speakers. In this paper, to remedy the performance degradation of the VSR model on unseen speakers, we propose prompt tuning methods of Deep Neural Networks (DNNs) for speaker-adaptive VSR. Specifically, motivated by recent advances in Natural Language Processing (NLP), we finetune prompts on adaptation data of target speakers instead of modifying the pre-trained model parameters. Different from the previous prompt tuning methods mainly limited to Transformer variant architecture, we explore different types of prompts, the addition, the padding, and the concatenation form prompts that can be applied to the VSR model which is composed of CNN and Transformer in general. With the proposed prompt tuning, we show that the performance of the pre-trained VSR model on unseen speakers can be largely improved by using a small amount of adaptation data (e.g., less than 5 minutes), even if the pre-trained model is already developed with large speaker variations. Moreover, by analyzing the performance and parameters of different types of prompts, we investigate when the prompt tuning is preferred over the finetuning methods. The effectiveness of the proposed method is evaluated on both word- and sentence-level VSR databases, LRW-ID and GRID.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), IEEE TPAMI 2024

Byung Kwan Lee will join NVIDIA to enhance his ongoing research on Vision LLM for his Ph.D. studies. Byung Kwan has recently published several top-tier papers on Vision LLMs, focusing on integrating vision and language as well as efficient VLLM. He expects to complete a paper as an outcome of his internship. This research internship experience will enable him to expand and deepen his Ph.D. research, as well as IVL lab research, thereby building global competitiveness.

Title: Advancing Causal Intervention in Image Captioning with Causal Prompt

Youngjoon Yu, Yeonju Kim, Yong Man Ro

This paper introduces a novel approach, called Causal Prompting Network (CPNet), to enhance the causal intervention in the context of image captioning. By leveraging visual prompt engineering in the feature space, this method aims to achieve superior performance in causal intervention tasks. Since CPNet is highly flexible and adaptable, it can be incorporated into any existing causal intervention-based image captioning framework. Specifically, two types of visual prompts — Causal RoI Prompt (CRP) and Causal Matching Prompt (CMP) — are employed to refine the feature representations effectively. CRP is utilized on the Region of Interest (RoI) feature of the object feature to enhance RoI features with deconfounded causal features. Meanwhile, CMP is used to strengthen the contextual representation of confounders linked to image captioning tasks. To evaluate the proposed CPNet’s effectiveness, an extensive range of experiments are conducted on the popular MS-COCO and Flickr30k dataset, and the results are validated using the Karpathy split. Experimental results demonstrate that the proposed CPNet surpasses the performance of other state-of-the-art image captioning methods.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), IEEE TNNLS 2024

국비 석사 2명,  산학장학생 등 모집합니다.

학생 초청 연구분야

• Vision + LLM (large language model) / LVLM (large vision language model)

• AI 모델: XAI, 역량, 메모리, Human-Machine Interaction, Robust, Multimodal AI

• Integration Vision, Language and Audio/Speech

관심있는 학생은 ymro@kaist.ac.kr 로 메일하기 바랍니다.

Title: CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models

Junho Kim*, Hyunjun Kim*, Yeonju Kim, Yong Man Ro (* equal contribution)

Large Multi-modal Models (LMMs) have recently demonstrated remarkable abilities in visual context understanding and coherent response generation. However, alongside these advancements, the issue of hallucinations has emerged as a significant challenge, producing erroneous responses that are unrelated to the visual contents. In this paper, we introduce a novel contrastive-based decoding method, COuntering DEscription Contrastive Decoding (CODE), which leverages self-generated descriptions as contrasting references during the decoding phase of LMMs to address hallucination issues. CODE utilizes the comprehensive descriptions from model itself as visual counterpart to correct and improve response alignment with actual visual content. By dynamically adjusting the information flow and distribution of next-token predictions in the LMM's vocabulary, CODE enhances the coherence and informativeness of generated responses. Extensive experiments demonstrate that our method significantly reduces hallucinations and improves cross-modal consistency across various benchmarks and cutting-edge LMMs. Our method provides a simple yet effective decoding strategy that can be integrated to existing LMM frameworks without additional training.

Title: Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models

Byung-Kwan Lee, Chae Won Kim, Beomchan Park, Yong Man Ro

The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the performance gap with powerful closed-source LLVMs. These advancements are attributed to multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present a new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce a new concept of traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained to generate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models.

Title: TroL: Traversal of Layers for Large Language and Vision Models

Byung-Kwan Lee, Sangyun Chung, Chae Won Kim, Beomchan Park, Yong Man Ro

Large language and vision models (LLVMs) have been driven by the generalization power of large language models (LLMs) and the advent of visual instruction tuning. Along with scaling them up directly, these models enable LLVMs to showcase powerful vision language (VL) performances by covering diverse tasks via natural language instructions. However, existing open-source LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often considered too large (e.g., 26B, 34B, and 110B parameters), having a larger number of layers. These large models demand costly, high-end resources for both training and inference. To address this issue, we present a new efficient LLVM family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL), which enables the reuse of layers in a token-wise manner. This layer traversing technique simulates the effect of looking back and retracing the answering stream while increasing the number of forward propagation layers without physically adding more layers. We demonstrate that TroL employs a simple layer traversing approach yet efficiently outperforms the open-source LLVMs with larger model sizes and rivals the performances of the closed-source LLVMs with substantial sizes.

Title: Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing

Authors: Jeong Hun Yeo*, Seunghee Han*, Minsu Kim, Yong Man Ro (* equal contributor)

In visual speech processing, context modeling capability is one of the most important requirements due to the ambiguous nature of lip movements. For example, homophenes, words that share identical lip movements but produce different sounds, can be distinguished by considering the context. In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs. Specifically, VSP-LLM is designed to perform multi-tasks of visual speech recognition and translation, where the given instructions control the type of task. The input video is mapped to the input latent space of an LLM by employing a self-supervised visual speech model. Focused on the fact that there is redundant information in input frames, we propose a novel deduplication method that reduces the embedded visual features by employing visual speech units. Through the proposed deduplication and Low Rank Adaptation (LoRA), VSP-LLM can be trained in a computationally efficient manner. In the translation dataset, the MuAViC benchmark, we demonstrate that VSP-LLM trained on just 30 hours of labeled data can more effectively translate lip movements compared to the recent model trained with 433 hours of data.

Title: What if...?: Thinking Counterfactual Keywords Helps to Mitigate Hallucination in Large Multi-modal Models

Authors: Junho Kim, Yeon Ju Kim, Yong Man Ro

This paper presents a way of enhancing the reliability of Large Multi-modal Models (LMMs) in addressing hallucination, where the models generate cross-modal inconsistent responses. Without additional training, we propose Counterfactual Inception, a novel method that implants counterfactual thinking into LMMs using self-generated counterfactual keywords. Our method is grounded in the concept of counterfactual thinking, a cognitive process where human considers alternative realities, enabling more extensive context exploration. Bridging the human cognition mechanism into LMMs, we aim for the models to engage with and generate responses that span a wider contextual scene understanding, mitigating hallucinatory outputs. We further introduce Plausibility Verification Process (PVP), a simple yet robust keyword constraint that effectively filters out sub-optimal keywords to enable the consistent triggering of counterfactual thinking in the model responses. Comprehensive analyses across various LMMs, including both open-source and proprietary models, corroborate that counterfactual thinking significantly reduces hallucination and helps to broaden contextual understanding based on true visual clues.

The paper was also presented as an oral presentation at the ACL 2024 conference in Bangkok, where it garnered significant attention.

Professor Yong Man Ro stated, " This research marks a significant advancement in human-AI interaction, and we hope this technology will be widely applied in various real-world applications. This award is yet another example of the international recognition of the excellence of AI research at KAIST’s School of Electrical Engineering."

Title: Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation

Authors: Minsu Kim, Jeongsoo Choi, Dahun Kim, Yong Man Ro

In this paper, we propose a pre-training method to learn unified representations of multilingual speech and text without using text, especially focusing on the purpose of multimodal-to-speech machine translation. To this end, we represent multilingual speech with speech units that are the discretized representations of speech features derived from a self-supervised speech model. By treating the speech units as pseudo-text, we can focus on the linguistic content of the speech, which can be easily associated with both speech and text modalities at the phonetic level information. By setting both the inputs and outputs of our learning problem as speech units, we propose to pre-train an encoder-decoder model in a many-to-many spoken language translation setting, namely Unit-to-Unit Translation (UTUT). Specifically, the encoder is conditioned on the source language token to correctly understand the input spoken language, while the decoder is conditioned on the target language token to generate the translated speech in the target language. Therefore, during the pre-training, the model can build the knowledge of how languages are comprehended and how to relate them to different languages. Since speech units can be easily associated from both audio and text by quantization and phonemization respectively, the UTUT pre-trained model can easily transferred to text-related tasks even if it is trained with textless. We show that a single UTUT pre-trained model can be employed for diverse multilingual speech- and text-related tasks, Speech-toSpeech Translation (STS), multilingual Text-to-Speech Synthesis (TTS), and Text-to-Speech Translation (TTST). By conducting comprehensive experiments encompassing various languages, we validate the efficacy of the proposed method across diverse multilingual tasks. Moreover, we show UTUT pre-trained model can also perform language translations for novel language pairs that are not present during training as pairs, which has not well been explored in the previous literature. Samples can be found on https://choijeongsoo.github.io/utut.

Title: Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation

Authors: Minsu Kim*, Jeonghun Yeo*, Se Jin Park, Hyeongseop Rha, Yong Man Ro (* equal contributor)

This paper explores sentence-level multilingual Visual Speech Recognition (VSR) that can recognize different languages with a single trained model. As the massive multilingual modeling of visual data requires huge computational costs, we propose a novel training strategy, processing with visual speech units. Motivated by the recent success of the audio speech unit, we propose to use a visual speech unit that can be obtained by discretizing the visual speech features extracted from the self-supervised visual speech model. Through analysis, we verify that the visual speech units mainly contain viseme information while suppressing non-linguistic information. By using the visual speech units as the inputs of our system, we propose to pre-train a VSR model to predict corresponding text outputs on multilingual data constructed by merging several VSR databases. As both the inputs (i.e., visual speech units) and outputs (i.e., text) are discrete, we can greatly improve the training efficiency compared to the standard VSR training. Specifically, the input data size is reduced to 0.016% of the original video inputs. In order to complement the insufficient visual information in speech recognition, we apply curriculum learning where the inputs of the system begin with audio-visual speech units and gradually change to visual speech units. After pre-training, the model is finetuned on continuous features. We set new state-of-the-art multilingual VSR performances by achieving comparable performances to the previous language-specific VSR models, with a single trained model.

Title; MoAI: Mixture of All Intelligence for Large Language and Vision Models

Authors: Byung-Kwan Lee, Beomchan Park, Chae Won Kim, Yong Man Ro

The rise of large language models (LLMs) and instruction tuning has led to the current trend of instruction-tuned large language and vision models (LLVMs). This trend involves either meticulously curating numerous instruction tuning datasets tailored to specific objectives or enlarging LLVMs to manage vast amounts of vision language (VL) data. However, current LLVMs have disregarded the detailed and comprehensive real-world scene understanding available from specialized computer vision (CV) models in visual perception tasks such as segmentation, detection, scene graph generation (SGG), and optical character recognition (OCR). Instead, the existing LLVMs rely mainly on the large capacity and emergent capabilities of their LLM backbones. Therefore, we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models. MoAI operates through two newly introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the outputs of the external CV models, the MoAI-Compressor aligns and condenses them to efficiently use relevant auxiliary visual information for VL tasks. MoAI-Mixer then blends three types of intelligence (1) visual features, (2) auxiliary features from the external CV models, and (3) language features by utilizing the concept of Mixture of Experts. Through this integration, MoAI significantly outperforms both open-source and closed-source LLVMs in numerous zero-shot VL tasks, particularly those related to real-world scene understanding such as object existence, positions, relations, and OCR without enlarging the model size or curating extra visual instruction tuning datasets.

Title; Text-Guided Distillation Learning to Diversify Video Embeddings for Text-Video Retrieval

Authors: Sangmin Lee, Hyung-Il Kim, Yong Man Ro

Conventional text-video retrieval methods typically match a video with a text on a one-to-one manner. However, a single video can contain diverse semantics, and text descriptions can vary significantly. Therefore, such methods fail to match a video with multiple texts simultaneously. In this paper, we propose a novel approach to tackle this one-to-many correspondence problem in text-video retrieval. We devise diverse temporal aggregation and a multikey memory to consider temporal and semantic diversity, consequently constructing multiple video embedding paths from a single video. Additionally, we introduce text-guided distillation learning that enables each video path to acquire meaningful distinct competencies in representing varied semantics. Our video embedding approach is text-agnostic, allowing the prepared video embeddings to be used continuously for any new text query. Experiments show our method outperforms other methods on four datasets. We further validate the effectiveness of our designs with ablation studies and analyses on multiple video embeddings.

Title; Weather-aware Drone-view Object Detection via Environmental Context Understanding

Authors: Hyunjun Kim, Dahye Lee, Sungjune Park, Yong Man Ro

Drone-view object detection has shown noticeable performances and has been adopted by various real-world applications. However, there exist still several problems to be handled for its safe usage. While most existing methods have tried to manage a variety of object scales, there are very few works to deal with diverse weather conditions. Therefore, in this paper, we propose a novel approach to build a drone-view object detector robust against the adverse effects of diverse environmental factors, such as foggy, rainy, and low illumination. To this end, we generated a weather content feature set using a multimodal large language model (MLLM), to describe diverse weather, illumination, and visibility conditions. These features are then adaptively selected based on the input image and applied to the detection framework to recognize the environmental semantics in the given visual images. Hereby, a detection framework can have environmental context understanding capability in drone-view images. With the comprehensive experiments and analysis, we corroborate the effectiveness of the proposed method showing the robustness against adverse weather conditions.

Title; Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-driven Approach for Cross-modal Alignment Fusion

Authors: Taeheon Kim*, Sangyun Chung*, Youngjoon Yu*, Yong Man Ro (*equal contributor)

Multispectral pedestrian detection is a crucial component in various critical applications. However, a significant challenge arises due to the misalignment between these modalities, particularly under real-world conditions where data often appear heavily misaligned. Conventional methods developed on well-aligned or minimally misaligned datasets fail to address these discrepancies adequately. This paper introduces a new framework for multispectral pedestrian detection designed specifically to handle heavily misaligned datasets without the need for costly and complex traditional pre-processing calibration. By leveraging Large-scale Vision-Language Models (LVLM) for cross-modal semantic alignment, our approach seeks to enhance detection accuracy by aligning semantic information across the RGB and thermal domains. This method not only simplifies the operational requirements but also extends the practical usability of multispectral detection technologies in practical applications.

국비 석사 2명,  KAIST 석사 1 명,  산학장학생 등 모집합니다.

모집 연구분야

• Vision + LLM (large language model) / LVLM (large vision language model)

• Multimodal + LLM / LVLM

• Integration Vision, Language and Speech/Sound

관심있는 학생은 ymro@kaist.ac.kr 로 메일하기 바랍니다.

Dr. Minsu Kim, who received his Ph.D. in February 2024, has joined the AI research group at META in London as a postdoctoral researcher. We extend our congratulations to him and hope that he will achieve outstanding results in AI research. By combining the research skills he developed during his Ph.D. at the IVY and LVL labs, particularly in human multimodal AI, with the cutting-edge research he will undertake at META, we believe Dr. Kim will make significant contributions to the field of AI.

Two PhD students from the IVY lab have secured research internships at Amazon and Google in USA, both  leading institutions in the field of AI. Sungjune Park will join Amazon, and Se Jin Park will join Google to enhance their ongoing research during their PhD studies. Sungjune Park has published several top-tier papers on multimodal AI, focusing on integrating vision and language, while Se Jin Park has published several top-tier papers on human multimodal AI, specifically on the ability to process and understand human-relevant modalities such as spoken language and facial-audio expressions. They expect to complete a paper as an outcome of their internships. This research internship experience will enable them to expand and deepen their PhD research, thereby building global competitiveness.

Title: CoLLaVO: Crayon Large Language and Vision mOdel

Authors: Byung-Kwan Lee, Beomchan Park, Chae Won Kim, Yong Man Ro

The remarkable success of Large Language Models (LLMs) and instruction tuning drives the evolution of Vision Language Models (VLMs) towards a versatile general-purpose model. Yet, it remains unexplored whether current VLMs genuinely possess quality object-level image understanding capabilities determined from 'what objects are in the image?' or 'which object corresponds to a specified bounding box?'. Our findings reveal that the image understanding capabilities of current VLMs are strongly correlated with their zero-shot performance on vision language (VL) tasks. This suggests that prioritizing basic image understanding is crucial for VLMs to excel at VL tasks. To enhance object-level image understanding, we propose Crayon Large Language and Vision mOdel (CoLLaVO), which incorporates instruction tuning with Crayon Prompt as a new visual prompt tuning scheme based on panoptic color maps. Furthermore, we present a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ACL 2024

Title: Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation

Authors: Se Jin Park*, Chae Won Kim*, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeonghun Yeo, and Yong Man Ro 

In this paper, we introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response, marking the initial step towards creating an avatar chatbot system without relying on intermediate text. To this end, we newly introduce MultiDialog, the first large-scale multimodal (\ie, audio and visual) spoken dialogue corpus containing 387 hours of approximately 10,000 dialogues, recorded based on the open domain dialogue dataset, TopicalChat. The MultiDialog contains parallel audio-visual recordings of conversation partners acting according to the given script with emotion annotations, which we expect to open up research opportunities in multimodal synthesis. Our Face-to-Face spoken dialogue model incorporates a textually pretrained large language model and adapts it into the audio-visual spoken dialogue domain by incorporating speech-text joint pretraining. Through extensive experiments, we validate the effectiveness of our model in facilitating a face-to-face conversation. All the data will be open-sourced.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ACL 2024

Title: Robust Pedestrian Detection via Constructing Versatile Pedestrian Knowledge Bank 

Authors: {Sungjune Park, Hyunjun Kim: equal first authors}, and Yong Man Ro

Pedestrian detection is a crucial field of computer vision research which can be adopted in various real-world applications (e.g., self-driving systems). However, despite the noticeable evolution of pedestrian detection, the pedestrian representations learned within a detection framework are usually limited to the particular scene data in which they were trained. Therefore, in this paper, we propose a novel approach to construct versatile pedestrian knowledge bank containing representative pedestrian knowledge which can be applicable to various detection frameworks and adopted in diverse scenes. We extract generalized pedestrian knowledge from a large-scale pretrained model, and we curate them by quantizing most representative features and guiding them to be more distinguishable from various background scenes. After they are stored in the versatile pedestrian knowledge bank, we leverage them to complement and enhance pedestrian features within a detection framework. Through comprehensive experiments, we validate the effectiveness of our method, demonstrating its versatility and outperforming state-of-the-art  detection performances.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), Pattern Recognition

Title: Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detection 

Authors: {Sungjune Park, Hyunjun Kim: equal first authors}, and Yong Man Ro

Large language models (LLMs) have shown their capability in understanding contextual and semantic information regarding appearance knowledge of instances. In this paper, we introduce a novel approach to utilize the strength of an LLM in understanding contextual appearance variations and to leverage its knowledge into a vision model (here, pedestrian detection). While pedestrian detection is considered one of crucial tasks directly related with our safety (e.g., intelligent driving system), it is challenging because of varying appearances and poses in diverse scenes. Therefore, we propose to formulate language-derived appearance elements and incorporate them with visual cues in pedestrian detection. To this end, we establish description corpus which includes numerous narratives describing various appearances of pedestrians and others. By feeding them through an LLM, we extract appearance knowledge sets that contain the representations of appearance variations. After that, we perform a task-prompting process to obtain appearance elements which are representative appearance knowledge guided to be relevant to a downstream pedestrian detection task. The obtained knowledge elements are adaptable to various detection frameworks, so that we can provide plentiful appearance information by integrating the language-derived appearance elements with visual cues within a detector. Through comprehensive experiments with various pedestrian detectors, we verify the adaptability and effectiveness of our method showing noticeable performance gains and achieving state-of-the-art detection performance on two public pedestrian detection benchmarks (i.e., CrowdHuman and WiderPedestrian).

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), IEEE TCSVT

국비 석사 2명,  KAIST박사 1명,  산학장학생 등 모집합니다.

모집 연구분야

• Vision + LLM (large language model) / LVLM (large vision language model)

• Multimodal + LLM / LVLM

• Integration Vision, Language and Speech/Sound

관심있는 학생은 ymro@kaist.ac.kr 로 메일하기 바랍니다.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST),

Title: Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection

Authors: {Taeheon Kim, Sebin Shin: equal first authors}, Youngjoon Yu, Hak Gu Kim, and Yong Man Ro

RGBT multispectral pedestrian detection has emerged as a promising solution for safety-critical applications that require day/night operations. However, the modality bias problem remains unsolved as multispectral pedestrian detectors learn the statistical bias in datasets. Specifically, datasets in multispectral pedestrian detection mainly distribute between ROTO (day) and RXTO (night) data; the majority of the pedestrian labels statistically co-occur with their thermal features. As a result, multispectral pedestrian detectors show poor generalization ability on examples beyond this statistical correlation, such as ROTX data. To address this problem, we propose a novel Causal Mode Multiplexer (CMM) framework that effectively learns the causalities between multispectral inputs and predictions. Moreover, we construct a new dataset (ROTX-MP) to evaluate modality bias in multispectral pedestrian detection. ROTX-MP mainly includes ROTX examples not presented in previous datasets. Extensive experiments demonstrate that our proposed CMM framework generalizes well on existing datasets (KAIST, CVC-14, FLIR) and the new ROTX-MP. We will release our new dataset to the public for future research.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), CVPR 2024

Title: AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation

Authors:  {Jeongsoo Choi, Se Jin Park, Minsu Kim: equal first authors}, and Yong Man Ro

This paper proposes a novel direct Audio-Visual Speech to Audio-Visual Speech Translation (AV2AV) framework, where the input and output of the system are multimodal (i.e., audio and visual speech). With the proposed AV2AV, two key advantages can be brought: 1) We can perform real-like conversations with individuals worldwide in a virtual meeting by utilizing our own primary languages. In contrast to Speech-to-Speech Translation (A2A), which solely translates between audio modalities, the proposed AV2AV directly translates between audio-visual speech. This capability enhances the dialogue experience by presenting synchronized lip movements along with the translated speech. 2) We can improve the robustness of the spoken language translation system. By employing the complementary information of audio-visual speech, the system can effectively translate spoken language even in the presence of acoustic noise, showcasing robust performance. To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A. This is done by learning unified audio-visual speech representations through self-supervised learning in advance to train the translation system. Moreover, we propose an AV-Renderer that can generate raw audio and video in parallel. It is designed with zero-shot speaker modeling, thus the speaker in source audio-visual speech can be maintained at the target translated audio-visual speech. The effectiveness of AV2AV is evaluated with extensive experiments in a many-to-many language translation setting.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), CVPR 2024

Title: AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio  Knowledge of a Pretrained Model

Authors: Jeong Hun Yeo, Minsu Kim, Jeongsoo Choi, Dae Hoe Kim, and Yong Man Ro

Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements. VSR is regarded as a challenging task because of the insufficient information on lip movements. In this paper, we propose an Audio Knowledge empowered Visual Speech Recognition framework (AKVSR) to complement the insufficient speech information of visual modality by using audio modality. Different from the previous methods, the proposed AKVSR 1) utilizes rich audio knowledge encoded by a large-scale pretrained audio model, 2) saves the linguistic information of audio knowledge in compact audio memory by discarding the non-linguistic information from the audio through quantization, and 3) includes Audio Bridging Module which can find the best-matched audio features from the compact audio memory, which makes our training possible without audio inputs, once after the compact audio memory is composed. We validate the effectiveness of the proposed method through extensive experiments, and achieve new state-of-the-art performances on the widely-used LRS3 dataset.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), IEEE Transactions on Multimedia

Title: Recruitment for PhD and MS Students

The IVY Laboratory is promoting international exchanges. For students applying to join the lab after September 2024, we prefer PhD candidates who are interested in international growth after completing their doctoral program and aim to pursue international opportunities upon graduation. Additionally, for master's degree candidates interested in joining our laboratory, we invite even those who aspire to pursue a PhD abroad or seek international career paths. We particularly welcome students who already have a lot of interest and experience in studying and researching deep learning-based approaches. Interested students are encouraged to contact us via email at ymro@kaist.ac.kr.

We look forward to hearing from you.

Title: Prof. Yong Man Ro Named ICT Endowed Chair Professor at KAIST

Prof. Yong Man Ro has been appointed as the ICT Endowed Chair Professor at KAIST. Since establishing the IVY Lab in 1997, Prof. Ro has been instrumental in advancing research in image processing, computer vision, artificial intelligence (AI), and multimedia.

Under his guidance, the IVY Lab has achieved remarkable milestones, including the graduation of 25 PhD and 70 Master's students, who have gone on to make significant contributions in IT area in the world. The laboratory's research output is highly competitive and excellent, including more than 520 peer-reviewed journal articles and top conference papers.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICASSP 2024

Title: Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens

Authors: Minsu Kim,  Jeongsoo Choi, Soumi Maiti, Jeong Hun Yeo,  Shinji Watanabe, and Yong Man Ro

In this paper, we propose methods to build a powerful and efficient Image-to-Speech captioning (Im2Sp) model. To this end, we start with importing the rich knowledge related to image comprehension and language modeling from a large-scale pre-trained visionlanguage model into Im2Sp. We set the output of the proposed Im2Sp as discretized speech units, i.e., the quantized speech features of a self-supervised speech model. The speech units mainly contain linguistic information while suppressing other characteristics of speech. This allows us to incorporate the language modeling capability of the pre-trained vision-language model into the spoken language modeling of Im2Sp. With the vision-language pre-training strategy, we set new state-of-the-art Im2Sp performances on two widely used benchmark databases, COCO and Flickr8k. Then, we further improve the efficiency of the Im2Sp model. Similar to the speech unit case, we convert the original image into image units, which are derived through vector quantization of the raw image. With these image units, we can drastically reduce the required data storage for saving image data to just 0.8% when compared to the original image data in terms of bits. Demo page: bit.ly/3Z9T6LJ

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICASSP 2024

Title: Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper

Authors: Jeong Hun Two, Minsu Kim, Shinji Watanabe, and Yong Man Ro

This paper proposes a powerful Visual Speech Recognition (VSR) method for multiple languages, especially for low-resource languages that have a limited number of labeled data. Different from previous methods that tried to improve the VSR performance for the target language by using knowledge learned from other languages, we explore whether we can increase the amount of training data itself for the different languages without human intervention. To this end, we employ a Whisper model which can conduct both language identification and audio-based speech recognition. It serves to filter data of the desired languages and transcribe labels from the unannotated, multilingual audio-visual data pool. By comparing the performances of VSR models trained on automatic labels and the human-annotated labels, we show that we can achieve similar VSR performance to that of human-annotated labels even without utilizing human annotations. Through the automated labeling process, we label large-scale unlabeled multilingual databases, VoxCeleb2 and AVSpeech, producing 1,002 hours of data for four low VSR resource languages, French, Italian, Spanish, and Portuguese. With the automatic labels, we achieve new state-of-the-art performance on mTEDx in four languages, significantly surpassing the previous methods. The automatic labels are available online: bit.ly/3Lajr6w

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICASSP 2024

Title: Persona Extraction through Semantic Similarity for Emotional Support Conversation Generation

Authors: Seunghee Han,  Se Jin Park,  Chae Won Kim, and Yong Man Ro

Providing emotional support through dialogue systems is becoming increasingly important in today’s world, as it can support both mental health and social interactions in many conversation scenarios. Previous works have shown that using persona is effective for generating empathetic and supportive responses. They have often relied on pre-provided persona rather than inferring them during conversations. However, it is not always possible to obtain a user persona before the conversation begins. To address this challenge, we propose PESS (Persona Extraction through Semantic Similarity), a novel framework that can automatically infer informative and consistent persona from dialogues. We devise completeness loss and consistency loss based on semantic similarity scores. The completeness loss encourages the model to generate missing persona information, and the consistency loss guides the model to distinguish between consistent and inconsistent persona. Our experimental results demonstrate that high-quality persona information inferred by PESS is effective in generating emotionally supportive responses.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICASSP 2024

Title: Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models 

Authors: Jeongsoo Choi,  Minsu Kim,  Se Jin Park,  and Yong Man Ro

In this paper, we present a method for reprogramming pre-trained audio-driven talking face synthesis models to operate in a text-driven manner. Consequently, we can easily generate face videos that articulate the provided textual sentences, eliminating the necessity of recording speech for each inference, as required in the audio-driven model. To this end, we propose to embed the input text into the learned audio latent space of the pre-trained audio-driven model, while preserving the face synthesis capability of the original pretrained model. Specifically, we devise a Text-to-Audio Embedding Module (TAEM) which maps a given text input into the audio latent space by modeling pronunciation and duration characteristics. Furthermore, to consider the speaker characteristics in audio while using text inputs, TAEM is designed to accept a visual speaker embedding. The visual speaker embedding is derived from a single target face image and enables improved mapping of input text to the learned audio latent space by incorporating the speaker characteristics inherent in the audio. The main advantages of the proposed framework are that 1) it can be applied to diverse audio-driven talking face synthesis models and 2) we can generate talking face videos with either text inputs or audio inputs with high flexibility.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICASSP 2024

Title: Exploring Phonetic Context-aware Lip-Sync for Talking Face Generation

Authors: Se Jin Park, Minsu Kim, Jeongsoo Choi, and Yong Man Ro

Talking face generation is the challenging task of synthesizing a natural and realistic face that requires accurate synchronization with a given audio. Due to co-articulation, where an isolated phone is influenced by the preceding or following phones, the articulation of a phone varies upon the phonetic context. Therefore, modeling lip motion with the phonetic context can generate more spatio-temporally aligned lip movement. In this respect, we investigate the phonetic context in generating lip motion for talking face generation. We propose Context-Aware Lip-Sync framework (CALS), which explicitly leverages phonetic context to generate lip movement of the target face. CALS is comprised of an Audio-to-Lip module and a Lip-toFace module. The former is pretrained based on masked learning to map each phone to a contextualized lip motion unit. The contextualized lip motion unit then guides the latter in synthesizing a target identity with context-aware lip motion. From extensive experiments, we verify that simply exploiting the phonetic context in the proposed CALS framework effectively enhances spatio-temporal alignment. We also demonstrate the extent to which the phonetic context assists in lip synchronization and find the effective window size for lip generation to be approximately 1.2 seconds.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), ICASSP 2024

Title: Improving Open Set Recognition via Visual Prompts Distilled from Common-Sense Knowledge

Authors: Seongyeop Kim, Hyung-Il Kim, and Yong Man Ro

Open Set Recognition (OSR) poses significant challenges in distinguishing known from unknown classes. In OSR, the overconfidence problem has become a persistent obstacle, where visual recognition models often misclassify unknown objects as known objects with high confidence. This issue stems from the fact that visual recognition models often lack the integration of common-sense knowledge, a feature that is naturally present in language-based models but lacking in visual recognition systems. In this paper, we propose a novel approach to enhance OSR performance by distilling common-sense knowledge into visual prompts. Utilizing text prompts that embody common-sense knowledge about known classes, the proposed visual prompt is learned by extracting semantic common-sense features and aligning them with image features from visual recognition models. The unique aspect of this work is the training of individual visual prompts for each class to encapsulate this common-sense knowledge. Our methodology is model-agnostic, capable of enhancing OSR across various visual recognition models, and computationally light as it focuses solely on training the visual prompts. This research introduces a method for addressing OSR, aiming at a more systematic integration of visual recognition systems with common-sense knowledge. The obtained results indicate an enhancement in recognition accuracy, suggesting the applicability of this approach in practical settings.

IMAGE VIDEO SYSTEM (IVY.) KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY (KAIST), AAAI 2024

Title: Defending Video Recognition Model against Adversarial Perturbations via Defense Patterns

Authors: Hong Joo Lee and Yong Man Ro

Deep Neural Networks (DNNs) have been widely successful in various domains, but they are vulnerable to adversarial attacks. Recent studies have also demonstrated that video recognition models are also susceptible to adversarial perturbations, but the existing defense strategies in the image domain do not transfer well to the video domain due to the lack of considering temporal development and require a high computational cost for training video recognition models. This paper, first, investigates the temporal vulnerability of video recognition models by quantifying the effect of temporal perturbations on the model’s performance. Based on these investigations, we propose Defense Patterns (DPs) that can effectively protect video recognition models by adding them to the input video frames. The DPs are generated on top of a pre-trained model, eliminating the need for retraining or fine-tuning, which significantly reduces the computational cost. Experimental results on two benchmark datasets and various action recognition models demonstrate the effectiveness of the proposed method in enhancing the robustness of video recognition models.

“Note: This work was done when Dr. Lee was a PhD student at KAIST. He is now a Postdoctoral Researcher at Technical University of Munich (TUM) after completing his PhD.”

Title:  Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

Authors: Joanna Hong, Se Jin Park, and Yong Man Ro

We present a novel approach to multilingual audio-visual speech recognition tasks by in troducing a single model on a multilingual  dataset. Motivated by the human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, the proposed model can capture which language is given as an input speech by distinguishing the inherent similarities and  differences between languages. To do so, we  design prompt fine-tuning into the largely pretrained audio-visual  representation model in  order to provide language information, both  label and nuance. Thus, the network can predict the correct speech with the correct language. To verify the effectiveness of the pro posed model, we conduct experiments on a multilingual audio-visual corpus, namely MuAViC, containing 9 languages. Our work contributes  to developing more robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.

Title: Enabling Visual Object Detection with Object Sounds via Visual Modality Recalling Memory 

Authors: Jung Uk Kim and Yong Man Ro

When humans hear sound of an object, they recall associated visual information and integrate the sounds and the recalled visual information to detect the objects. In this paper, we present a novel sound-based object detector that mimics these processes of humans. We design a Visual Modality Recalling (VMR) memory that recalls information of a visual modality, given an audio modal input (i.e., sound). To achieve this goal, we propose visual modality recalling loss and audiovisual association loss to guide VMR memory to memorize the visual modal information by establishing associations between the audio and visual modalities. With the recalled visual modal information through the VMR memory and the original audio modal input, audio-visual integration is conducted. In this step, we introduce integrated  feature contrastive loss which allows the integrated feature to be embedded as if it were encoded using both the audio and visual modalities. This guidance enables our sound-based object detector to perform robust object detection even when the only sound is provided. We believe that our work is a cornerstone study that can provide a new perspective to the conventional object detection studies that rely only on the visual modality. Comprehensive experimental results demonstrate the effectiveness of the proposed method with VMR memory.

“Note: This work was done when Dr. Jung was a PhD student at KAIST. He is now a professor at KyungHee University after completing his PhD.”

2024년 전기 합격한 학생들 축하합니다. 연구실에서 2024년도 전기에 국비 박사과정 2명(완료), KAIST 박사과정 2명(완료), KAIST 석사과정 1명 및 산학장학생 (KEPSI, EPSS, LGenius) 등을 모집합니다.

모집 연구분야

- Multi-modal (vision-sound-language) learning

- Deep learning AI (XAI, Competency, Robustness)

- Vision (object detection, classification etc.) + Large Scale Model

최근 연구실 석박사과정 딥러닝 관련 해외 학회 발표실적 - link

최근 연구실 석박사과정 해외 저널 실적 - link

연구실 입학관련 면담요청은 노용만 교수님 (ymro@kaist.ac.kr)에게 이메일 하기 바랍니다.

Title: Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge

Authors: Minsu Kim*, Jeong Hun Yeo*, Jeongsoo Choi, and Yong Man Ro (* equally contributed)

This paper proposes a novel lip reading framework, especially for low-resource languages, which has not been well addressed in the previous literature. Since low-resource languages do not have enough video-text paired data to train the model to have sufficient power to model lip movements and language, it is regarded as challenging to develop lip reading models for low-resource languages. In order to mitigate the challenge, we try to learn general speech knowledge, the ability to model lip movements, from a high-resource language through the prediction of speech units. It is known that different languages partially share common phonemes, thus general speech knowledge learned from one language can be extended to other languages. Then, we try to learn language-specific knowledge, the ability to model language, by proposing Language-specific Memory-augmented Decoder (LMDecoder). LMDecoder saves language-specific audio features into memory banks and can be trained on audio-text paired data which is more easily accessible than video-text paired data. Therefore, with LMDecoder, we can transform the input speech units into language-specific audio features and translate them into texts by utilizing the learned rich language knowledge. Finally, by combining general speech knowledge and language-specific knowledge, we can efficiently develop lip reading models even for low-resource languages. Through extensive experiments using five languages, English, Spanish, French, Italian, and Portuguese, the effectiveness of the proposed method is evaluated.

Title: Mitigating Adversarial Vulnerability through Causal Parameter Estimation by Adversarial Double Machine Learning

Authors: Byung-Kwan Lee*, Junho Kim*, and Yong Man Ro (* equally contributed)

Adversarial examples derived from deliberately crafted perturbations on visual inputs can easily harm decision process of deep neural networks. To prevent potential threats, various adversarial training-based defense methods have grown rapidly and become a de facto standard approach for robustness. Despite recent competitive achievements, we observe that adversarial vulnerability varies across targets and certain vulnerabilities remain prevalent. Intriguingly, such peculiar phenomenon cannot be relieved even with deeper architectures and advanced defense methods. To address this issue, in this paper, we introduce a causal approach called Adversarial Double Machine Learning (ADML), which allows us to quantify the degree of adversarial vulnerability for network predictions and capture the effect of treatments on outcome of interests. ADML can directly estimate causal parameter of adversarial perturbations per se and mitigate negative effects that can potentially damage robustness, bridging a causal perspective into the adversarial vulnerability. Through extensive experiments on various CNN and Transformer architectures, we corroborate that ADML improves adversarial robustness with large margins and relieve the empirical observation.

Title: DiffV2S: Diffusion-based Video-to-Speech Synthesiswith Vision-guided Speaker Embedding

Authors:  Jeongsoo Choi*, Joanna Hong*, and Yong Man Ro (* equally contributed)

Recent research has demonstrated impressive results in video-to-speech synthesis which involves reconstructing speech solely from visual input. However, previous works have struggled to accurately synthesize speech due to a lack of sufficient guidance for the model to infer the correct content with the appropriate sound. To resolve the issue, they have adopted an extra speaker embedding as a speaking style guidance from a reference auditory information. Nevertheless, it is not always possible to obtain the audio information from the corresponding video input, especially during the inference time. In this paper, we present a novel vision-guided speaker embedding extractor using a self-supervised pre-trained model and P-tuning technique. In doing so, the rich speaker embedding information can be produced solely from input visual information, and the extra audio information is not necessary during the inference time. Using the extracted vision-guided speaker embedding representations, we further develop a diffusion-based video-to-speech synthesis model, so called DiffV2S, conditioned on those speaker embeddings and the visual representation extracted from the input video. The proposed DiffV2S not only maintains phoneme details contained in the input video frames, but also creates a highly intelligible mel-spectrogram in which the speaker identities of the multiple speakers are all preserved. Our experimental results show that DiffV2S achieves the state-of-the-art performance compared to the previous video-to-speech synthesis technique.

1. Title: Robust multispectral pedestrian detection via spectral position-free feature mapping

Authors: Sungjune Park, Jung Uk Kim, Jin Mo Song, and Yong Man Ro

Abstract: Recently, although multispectral pedestrian detection has achieved remarkable performances, there is still a problem to be handled, position shift problem. Due to the problem, a pedestrian looks like existing in different positions between each modal image. Then, a single bounding box usually fails to capture an entire pedestrian properly in both modal images at the same time, which means it would not contain some parts of a pedestrian and includes noisy backgrounds instead. In this paper, we propose a novel approach, that is, a pedestrian feature mapping from mis-captured pedestrian features to well-captured pedestrian features which encode an entire pedestrian properly in both modal images. To this end, we utilize a memory architecture which stores well-captured pedestrian features, and then, the well-captured features can enhance the quality of pedestrian representation by providing the distinctive information of a pedestrian. We validate the effectiveness of our approach with comprehensive experiments on two multispectral pedestrian detection datasets, achieving state-of-the-art performances.

2. Title: Mitigating Dataset Bias in Image Captioning through CLIP Confounder-free Captioning Network

Authors: YeonJu Kim, Junho Kim, Byung-Kwan Lee, Sebin Shin, and Yong Man Ro

Abstract: The dataset bias has been identified as a major challenge in image captioning. When the image captioning model predicts a word, it should consider the visual evidence associated with the word, but the model tends to use contextual evidence from the dataset bias and results in biased captions, especially when the dataset is biased toward some specific situations. To solve this problem, we approach from the causal inference perspective and design a causal graph. Based on the causal graph, we propose a novel method named C 2Cap which is CLIP  confounder-free captioning network. We use the global visual confounder to control the confounding factors in the image and train the model  to produce debiased captions. We validate our proposed method on MSCOCO benchmark and demonstrate the effectiveness of our method.

Title: Adversarial anchor-guided feature refinement for adversarial defense

Authors: Hakmin Lee and Yong Man Ro 

Abstract: Adversarial training (AT), which is known as a robust training method for defending against adversarial examples, usually loses the performance of models for clean examples due to the feature distribution discrepancy between clean and adversarial. In this paper, we propose a novel Adversarial Anchor-guided Feature Refinement (AAFR) defense method aimed at reducing the discrepancy and delivering reliable performances for both clean and adversarial examples. We devise adversarial anchor that detects whether the feature comes from clean or adversarial example. Then, we use adversarial anchor to refine the feature to reduce the discrepancy. As a result, the proposed method substantially achieves adversarial robustness while preserving the performance for clean examples. The effectiveness of the proposed method is verified with comprehensive experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets.

2023년 후기 학생들 축하해요. 연구실에서 2023년도 후기에 국비 석사과정 1명 및 산학장학생 (KEPSI, EPSS, LGenius), 국비 박사(완료) 등을 모집합니다. 

모집 연구분야

- Multi-modal (vision-sound-language) learning

- Deep learning AI (XAI, Competency, Robustness)

- Large Scale Model + Alpha

최근 연구실 석박사과정 딥러닝 관련 해외 학회 발표실적 - link 

최근 연구실 석박사과정 해외 저널 실적 - link 

연구실 입학관련 면담요청은 노용만 교수님 (ymro@kaist.ac.kr)에게 이메일 하기 바랍니다.

Title: Robust Proxy: Improving Adversarial Robustness by Robust Proxy Learning

Authors: Hong Joo Lee and Yong Man Ro

Recently, it has been widely known that deep neural networks are highly vulnerable and easily broken by adversarial attacks. To mitigate the adversarial vulnerability, many defense algorithms have been proposed. Recently, to improve adversarial robustness, many works try to enhance feature representation by imposing more direct supervision on the discriminative feature. However, existing approaches lack an understanding of learning adversarially robust feature representation. In this paper, we propose a novel training frameworks called Robust Proxy Learning. In the proposed method, the model explicitly learns robust feature representations with robust proxies. To this end, firstly, we demonstrate that we can generate class representative robust features by adding class-wise robust perturbations. Then, we use the class representative features as robust proxies. With the classwise robust features, the model explicitly learns adversarially robust feature through the proposed robust proxy learning framework. Through extensive experiments, we verify that we can manually generate robust features, and our proposed learning framework could increase the robustness of the DNNs.

[Recent PhD graduates: postdocs] Ph.D graduates of 2023 have joined postdocs in AI research at UIUC and TUM.

Dr. Sangmin Lee and Dr. Hong Joo Lee, who received their Ph.Ds in 2023, joined the AI research group at the University of Illinois at Urbana-Champaign (UIUC) and the Technical University of Munich (TUM) as postdocs, respectively. We congratulate them and hope that they will establish world competitiveness in AI research by combining their research skills built in their PhDs at IVY lab and the research they will build at new institutions.

On the other hand, Dr. Jung Uk Kim, who received his PhD in 2022, was appointed as a professor in the school of computing at Kyung Hee University last year.

Also, recent PhD graduates from IVY lab had excellent AI research achievements and were selected as postdocs at the AI top institutes and as professors in AI research. Prof. Hak Gu Kim and Prof. Sung Tae Kim, who were postdocs at EPFL and TUM, respectively, returned to Korea a few years ago and are continuing their AI research as professors at Chung-Ang University and Kyung Hee University, respectively.

[META, CMU Internships] Joanna and Minsu will join META and CMU for research internships, respectively.

Two PhD students from the human multimodal AI research group in IVY lab have secured research internships at META and CMU, the leading institutes in the AI field. Joanna Hong will join META (https://about.meta.com/realitylabs/ ) and Minsu Kim will join CMU (https://lti.cs.cmu.edu/work ) for a few months, respectively. They have both published several top-tier papers on human multimodal AI that deal with the ability to process and understand human-related modalities, such as facial __expression__, speech, and language. They expect to collaborate with their mentors and colleagues at the two institutes to publish a top-tier paper during their internships. This experience will enable them to expand and deepen their PhD research and establish their world competitiveness.

TItle: Intelligible Lip-to-speech Synthesis with Speech Units 

Authors: Jeongsoo Choi and Minsu Kim and Yong Man Ro

In this paper, we propose a novel Lip-to-Speech synthesis (L2S) framework, for synthesizing intelligible speech from a silent lip movement video. Specifically, to complement the insufficient supervisory signal of the previous L2S model, we propose to use quantized self-supervised speech representations, named speech units, as an additional prediction target of the proposed L2S model. Therefore, the proposed L2S model is trained to generate multi-target, mel-spectrogram and speech units. As the speech units are discrete representations while mel-spectrogram is continuous, the proposed multi-target L2S model can be trained with strong content supervision, even without using text-labeled data. Moreover, to accurately convert the synthesized mel-spectrogram into a waveform, we introduce a multi-input vocoder that can generate a clear waveform even from blurry and noisy mel-spectrogram by referring to the speech units. Evaluation results confirm the effectiveness of the proposed method.

Title: Deep learning-based classification system of bacterial keratitis and fungal keratitis using anterior segment images

Authors:  Yeo Kyoung Won*, Hyebin Lee*, Youngjun Kim, Gyule Han, Tae-Young Chung, Yong Man Ro and Dong Hui Lim

(* equal contributor)

Introduction: Infectious keratitis is a vision threatening disease. Bacterial and fungal keratitis are often confused in the early stages, so right diagnosis and optimized treatment for causative organisms is crucial. Antibacterial and antifungal medications are completely different, and the prognosis for fungal keratitis is even much worse. Since the identification of microorganisms takes a long time, empirical treatment must be started according to the appearance of the lesion before an accurate diagnosis. Thus, we developed an automated deep learning (DL) based diagnostic system of bacterial and fungal keratitis based on the anterior segment photographs using two proposed modules, Lesion Guiding Module (LGM) and Mask Adjusting Module (MAM).

Title: Stereoscopic Vision Recalling Memory for Monocular 3D Object Detection

Authors: Jung Uk Kim, Hyung-Il Kim, and Yong Man Ro

Monocular 3D object detection has drawn increasing attention in various human-related applications, such as autonomous vehicles, due to its cost-effective property. On the other hand, a monocular image alone inherently contains insufficient information to infer the 3D information. In this paper, we propose a new monocular 3D object detector that can recall the stereoscopic visual information about an object, given a monocular each object by being aware of its location. Next, given the object appearance of the monocular image, we devise Monocular-to-tereoscopic (M2S) memory that can recall the object appearance of the counterpart view and corresponding depth information. For this purpose, we introduce a stereoscopic vision memorizing loss that guides M2S memory to store the stereoscopic visual information. Further, we propose a binocular vision association loss to guide M2S memory that can associate information of the left-right view about the object when estimating the depth. As a result, our monocular 3D object detector with M2S memory can effectively exploit the recalled stereoscopic visual information in the inference phase. The comprehensive experimental results on the two public datasets, KITTI 3D Object Detection Benchmark and Waymo Open Dataset, demonstrate the effectiveness of the proposed method. We claim that our method is a step forward method that follows the behaviors of humans that can recall the stereoscopic visual information even when one eye is closed.

"Note: Jung Uk Kim is a professor at KyungHee University after completing his PhD."

Title: Advancing Adversarial Training by Injecting Booster Signal

Authors: Hong Joo Lee and Youngjoon Yu, and Yong Man Ro

Recent works have demonstrated that deep neural networks (DNNs) are highly vulnerable to adversarial attacks. To defend against adversarial attacks, many defense strategies have been proposed, among which adversarial training has  been demonstrated to be the most effective strategy. However, it has been known that adversarial training sometimes hurts natural accuracy. Then, many works focus on optimizing model parameters to handle the problem. Different from the previous approaches, in this paper, we propose a new approach to improvethe adversarial robustness by using an external signal rather than model parameters. In the proposed method, a well-optimized universal external signal called a booster signal is injected to theoutside of the image which does not overlap with the original content. Then, it boosts both adversarial robustness and natural accuracy. The booster signal is optimized in parallel to modelparameters step by step collaboratively. Experimental results show that the booster signal can improve both the natural and robust accuracies over the recent state-of-the-art adversarial training methods. Also, optimizing the booster signal is generaland flexible enough to be adopted on any existing adversarial training methods.

Title: Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring 

Authors: Joanna Hong*, Minsu Kim*, Jeongsoo Choi, and Yong Man Ro (* equally contributed)

Visual Speech Recognition (AVSR) under multimodal input corruption situation where audio inputs and visual inputs are both corrupted, which is not well addressed in previous research directions. Previous studies have focused on how to complement the corrupted audio inputs with the clean visual inputs with the assumption of the availability of clean visual inputs. However, in real life, the clean visual inputs are not always accessible and can even be corrupted by occluded lip region or with noises. Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models. Then, we design multimodal input corruption modeling to develop robust AVSR models. Lastly, we propose a novel AVSR framework, namely Audio-Visual Reliability Scoring module (AV-RelScore), that is robust to the corrupted multimodal inputs. The AV-RelScore can determine which input modal stream is reliable or not for the prediction and also can exploit the more reliable streams in prediction. The effectiveness of the proposed method is evaluated with comprehensive experiments on popular benchmark databases, LRS2 and LRS3. We also show that the reliability scores obtained by AV-RelScore well reflect the degree of corruption and make the proposed model focus on the reliable multimodal representations.

Title: Demystifying Causal Features on Adversarial Examples and Causal Inoculation for Robust Network by Adversarial Instrumental Variable Regression 

Authors: Junho Kim*, Byung-Kwan Lee*, and Yong Man Ro (* equally contributed)

The origin of adversarial examples is still inexplicable in research fields, and it arouses arguments from various viewpoints, albeit comprehensive investigations. In this paper, we propose a way of delving into the unexpected vulnerability in adversarially trained networks from a causal perspective, namely adversarial instrumental variable (IV) regression. By deploying it, we estimate the causal relation of adversarial prediction under an unbiased environment dissociated from unknown confounders. Our approach aims to demystify inherent causal features on adversarial examples by leveraging a zero-sum optimization game between a casual feature estimator (i.e., hypothesis model) and worst-case counterfactuals (i.e., test function) disturbing to find causal features. Through extensive analyses, we demonstrate that the estimated causal features are highly related to the correct prediction for adversarial robustness, and the counterfactuals exhibit extreme features significantly deviating from the correct prediction. In addition, we present how to effectively inoculate CAusal FEatures (CAFE) into defense networks for improving adversarial robustness.

Title: Lip-to-speech Synthesis in the Wild with Multi-task Learning

Authors: Minsu Kim∗, Joanna Hong∗, and Yong Man Ro (* equally contributed)

Recent studies have shown impressive performance in Lip-to-speech synthesis that aims to reconstruct speech from visual information alone. However, they have been suffering from synthesizing accurate speech in the wild, due to insufficient supervision for guiding the model to infer the correct content. Distinct from the previous methods, in this paper, we develop a powerful Lip2Speech method that can reconstruct speech with correct contents from the input lip movements, even in a wild environment. To this end, we design multi-task learning that guides the model using multimodal supervision, i.e. text and audio, to complement the insufficient word representations of acoustic feature reconstruction loss. Thus, the proposed framework brings the advantage of synthesizing speech containing the right content of multiple speakers with unconstrained sentences. We verify the effectiveness of the proposed method using LRS2, LRS3, and LRW datasets.

Title: Similarity Relation Preserving Cross-Modal Learning For Multispectral Pedestrian Detection Against Adversarial Attacks

Authors: Jung Uk Kim and Yong Man Ro (* equally contributed)

Although multispectral pedestrian detection studies have shown remarkable detection performances, they are still vulnerable to adversarial attacks. We see the similarity relations between object candidates were not maintained because of the adversarial attacks, resulting in performance degradation. In this paper, we introduce a new method that can preserve the similarity relation between candidates against adversarial attacks using multispectral knowledge. First, we propose Similarity Relation Generation (SRG) module to generate the optimal similarity relation between clean candidates by referring to the two modalities (color and thermal). Second, we propose Adversarial Similarity Relation Preserving (ASRP) module to guide the similarity relation between adversarial candidates to be similar to that of the clean candidates. By maintaining the relationship between candidates, our multispectral detector can distinguish between pedestrian/background classes even in adversarial attacks. Comprehensive experimental results show that our method conspicuously improves the adversarial robustness.

Title: Multi-Temporal Lip-Audio Memory for Visual Speech Recognition

Authors: Jeong Hun Yeo, Minsu Kim, and Yong Man Ro

Visual Speech Recognition (VSR) is a task to predict a sentence or word from lip movements. Some works have been recently presented which use audio signals to supplement visual information. However, existing methods utilize only limited information such as phoneme-level features and soft labels of Automatic Speech Recognition (ASR) networks. In this paper, we present a Multi-Temporal Lip-Audio Memory (MTLAM) that makes the best use of audio signals to complement insufficient information of lip movements. The proposed method is mainly composed of two parts: 1) MTLAM saves multi-temporal audio features produced from short- and long-term audio signals, and the MTLAM memorizes a visual-to-audio mapping to load stored multi-temporal audio features from visual features at the inference phase. 2) We design an audio temporal model to produce multi-temporal audio features capturing the context of neighboring words. In addition, to construct effective visual-to-audio mapping, the audio temporal models can generate audio features time-aligned with visual features. Through extensive experiments, we validate the effectiveness of the MTLAM achieving state-of-the-art performances on two public VSR datasets.

https://www.youtube.com/watch?v=WiILFSZcBUw

Title: Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video

Authors: Minsu Kim, Chae Won Kim, and Yong Man Ro 

Forced alignment refers to a technology that time-aligns a given transcription with a corresponding speech. However, as the forced alignment technologies have developed using speech audio, they might fail in alignment when the input speech audio is noise-corrupted or is not accessible. We focus on that there is another component that the speech can be inferred from, the speech video (i.e., talking face video). Since the drawbacks of audio-based forced alignment can be complemented using the visual information when the audio signal is under poor condition, we try to develop a novel video-based forced alignment method. However, different from audio forced alignment, it is challenging to develop a reliable visual forced alignment technology for the following two reasons: 1) Visual Speech Recognition (VSR) has a much lower performance compared to audio-based Automatic Speech Recognition (ASR), and 2) the translation from text to video is not reliable, so the method typically used for building audio forced alignment cannot be utilized in developing visual forced alignment. In order to alleviate these challenges, in this paper, we propose a new method that is appropriate for visual forced alignment, namely Deep Visual Forced Alignment (DVFA). The proposed DVFA can align the input transcription (i.e., sentence) with the talking face video without accessing the speech audio. Moreover, by augmenting the alignment task with anomaly case detection, DVFA can detect mismatches between the input transcription and the input video while performing the alignment. Therefore, we can robustly align the text with the talking face video even if there exist error words in the text. Through extensive experiments, we show the effectiveness of the proposed DVFA not only in the alignment task but also in interpreting the outputs of VSR models.

Title: Multispectral Invisible Coating: Laminated Visible-Thermal Physical Attack against Multispectral Object Detectors using Transparent Low-e films

Authors: Taeheon Kim,  Youngjoon Yu, and Yong Man Ro

Multispectral object detection plays a vital role in safety-critical vision systems that require an around-the-clock operation and encounter dynamic real-world situations(e.g., self-driving cars and autonomous surveillance systems). Despite its crucial competence in safety-related applications, its security against physical attacks is severely understudied. We investigate the vulnerability of multispectral detectors against physical attacks by proposing a new physical method: Multispectral Invisible Coating. Utilizing transparent Low-e films, we realize a laminated visible-thermal physical attack by attaching Low-e films over a visible attack printing. Moreover, we apply our physical method to manufacture a Multispectral Invisible Suit that hides persons from the multiple view angles of Multispectral detectors. To simulate our attack under various surveillance scenes, we constructed a large-scale multispectral pedestrian dataset which we will release in public. Extensive experiments show that our proposed method effectively attacks the state-of-the-art multispectral detector both in the digital space and the physical world.

Title: Defending Person Detection Against Adversarial Patch Attack by using Universal Defensive Frame

Authors: Youngjoon Yu*, Hong Joo Lee*, Hakmin Lee, and Yong Man Ro (*: equally contributed)

Person detection has attracted great attention in the computer vision area and is an imperative element in human-centric computer vision. Although the predictive performances of person detection networks have been improved dramatically, they are vulnerable to adversarial patch attacks. Changing the pixels in a restricted region can easily fool the person detection network in safety-critical applications such as autonomous driving and security systems. Despite the necessity of countering adversarial patch attacks, very few efforts have been dedicated to defending person detection against adversarial patch attack. In this paper, we propose a novel defense strategy that defends against an adversarial patch attack by optimizing a defensive frame for person detection. The defensive frame alleviates the effect of the adversarial patch while maintaining person detection performance with clean person. The proposed defensive frame in the person detection is generated with a competitive learning algorithm which makes an iterative competition between detection threatening module and detection shielding module in person detection. Comprehensive experimental results demonstrate that the proposed method effectively defends person detection against adversarial patch attacks.

Title: Face Shape-Guided Deep Feature Alignment for Face Recognition Robust to Face Misalignment

Authors: Hyung-Il Kim, Kimin Yun, and Yong Man Ro

For the past decades, face recognition (FR) has been actively studied in computer vision and pattern recognition society. Recently, due to the advances in deep learning, the FR technology shows high performance for most of the benchmark datasets. However, when the FR algorithm is applied to a real-world scenario, the performance has been known to be still unsatisfactory. This is mainly attributed to the mismatch between training and testing sets. Among such mismatches, face misalignment between training and testing faces is one of the factors that hinder successful FR. To address this limitation, we propose a face shape-guided deep feature alignment framework for FR robust to the face misalignment. Based on a face shape prior (e.g., face keypoints), we train the proposed deep network by introducing alignment processes, i.e., pixel and feature alignments, between well-aligned and misaligned face images. Through the pixel alignment process that decodes the aggregated feature extracted from a face image and face shape prior, we add the auxiliary task to reconstruct the well-aligned face image. Since the aggregated features are linked to the face feature extraction network as a guide via the feature alignment process, we train the robust face feature to the face misalignment. Even if the face shape estimation is required in the training stage, the additional face alignment process, which is usually incorporated in the conventional FR pipeline, is not necessarily needed in the testing phase. Through the comparative experiments, we validate the effectiveness of the proposed method for the face misalignment with the FR datasets.

2023년 전기 합격생들 축하해요. 연구실에서 2023년도 전기에 국비 석사과정 2명, KAIST 석사과정, 및 산학장학생 (KEPSI, EPSS, LGenius) 등을 모집합니다.

모집 연구분야:

 - 딥러닝 기반 인공지능 (설명가능인공지능, 역량인지 인공지능, 강인한 인공지능)

 - Machine learning with multi-modal data

 - Computer vision

 - Multi-modal (vision-sound- language) co-learning

최근 연구실 석박사과정 딥러닝 관련 해외 학회 발표실적 - 링크 (LINK)

최근 연구실 석박사과정 해외 저널 실적 - 링크 (LINK)

연구실 입학관련 면담요청은 노용만 교수님(ymro@kaist.ac.kr)에게 이메일 하기 바랍니다.

Title: Defending Physical Adversarial Attack on Object Detection via Adversarial Patch-Feature Energy

Authors: Taeheon Kim, Youngjoon Yu, and Yong Man Ro

Object detection plays an important role in security-critical systems such as autonomous vehicles but has shown to be vulnerable to adversarial patch attacks. Existing defense methods against adversarial patches are restricted to localized noise attacks by removing noisy regions in the input image. However, adversarial patches have developed into natural-looking patterns which evade existing defenses. To address this issue, we propose a defense method based on a novel concept “Adversarial Patch-Feature Energy” (APE) which exploits common deep feature characteristics of an adversarial patch. Our proposed defense consists of APE-masking and APE-refinement which can be employed to defend against any adversarial patch on literature. Extensive experiments demonstrate that APE-based defense achieves impressive robustness against adversarial patches both in the digital space and the physical world.

Title: Speaker-adaptive Lip Reading with User-dependent Padding

Authors: Minsu Kim, Hyunjun Kim, and Yong Man Ro 

Lip reading aims to predict speech based on lip movements alone. As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements. This makes the lip reading models show degraded performance when they are applied to unseen speakers due to the mismatch between training and testing conditions. Speaker adaptation technique aims to reduce this mismatch between train and test speakers, thus guiding a trained model to focus on modeling the speech content without being intervened by the speaker variations. In contrast to the efforts made in audio-based speech recognition for decades, the speaker adaptation methods have not well been studied in lip reading. In this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding. The user-dependent padding is a speaker-specific input that can participate in the visual feature extraction stage of a pre-trained lip reading model. Therefore, the lip appearances and movements information of different speakers can be considered during the visual feature encoding, adaptively for individual speakers. Moreover, the proposed method does not need 1) any additional layers, 2) to modify the learned weights of the pre-trained model, and 3) the speaker label of train data used during pre-train. It can directly adapt to unseen speakers by learning the user-dependent padding only, in a supervised or unsupervised manner. Finally, to alleviate the speaker information insufficiency in public lip reading databases, we label the speaker of a well-known audio-visual database, LRW, and design an unseen-speaker lip reading scenario named LRW-ID. The effectiveness of the proposed method is verified on sentence- and word-level lip reading, and we show it can further improve the performance of a well-trained model with large speaker variations.

Title: VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection

Authors: Joanna Hong, Minsu Kim, and Yong Man Ro 

The goal of this work is to reconstruct speech from a silent talking face video. Recent studies have shown impressive performance on synthesizing speech from silent talking face videos. However, they have not explicitly considered on varying identity characteristics of different speakers, which place a challenge in the video-to-speech synthesis, and this becomes more critical in unseen-speaker settings. Distinct from the previous methods, our approach is to separate the speech content and the visage-style from a given silent talking face video. By guiding the model to independently focus on modeling the two representations, we can obtain the speech of high intelligibility from the model even when the input video of an unseen subject is given. To this end, we introduce speech-visage selection module that separates the speech content and the speaker identity from the visual features of the input video. The disentangled representations are jointly incorporated to synthesize speech through visage-style based synthesizer which generates speech by coating the visage-styles while maintaining the speech content. Thus, the proposed framework brings the advantage of synthesizing the speech containing the right content even when the silent talking face video of an unseen subject is given. We validate the effectiveness of the proposed framework on the GRID, TCD-TIMIT volunteer, and LRW datasets.

Title: Audio-Visual Mismatch-Aware Video Retrieval via Association and Adjustment

Authors: Sangmin Lee, Sungjune Park, and Yong Man Ro 

Retrieving desired videos using natural language queries has attracted increasing attention in research and industry fields as a huge number of videos appear on the internet. Natural language queries made by humans vary greatly and often include details related to audio cues. Some existing methods attempted to address this video retrieval problem by exploiting multi-modal information, especially audio-visual data of videos. However, many videos often have mismatched visual and audio cues for several reasons including background music, noise, and even missing sound. Therefore, the naive fusion of such mismatched visual and audio cues can negatively affect the semantic embedding of video scenes when retrieving video from text queries. Mismatch condition can be categorized into two cases: (i) Audio itself does not exist, (ii) Audio exists but does not match with visual. To deal with (i), we introduce audio-visual associative memory (AVA-Memory) to associate audio cues even from videos without audio data. The associated audio cues from visual data can guide the video embedding feature to be aware of audio information even in the missing audio condition. To address (ii), we propose audio embedding adjustment by considering the degree of matching between visual and audio data. In this procedure, constructed AVA-Memory enables to figure out how well the visual and audio in the video are matched and to adjust the weighting between actual audio and associated audio. Experimental results show that the proposed method outperforms other state-of-the-art video retrieval methods. Further, we validate the effectiveness of the proposed network designs with analyses.

Title: Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition

Authors: Joanna Hong*, Minsu Kim*, Daehun Yoo, and Yong Man Ro (* equally contributed) 

This paper focuses on designing a noise-robust end-to-end Audio-Visual Speech Recognition (AVSR) system. To this end, we propose Visual Context-driven Audio Feature Enhancement module (V-CAFE) to enhance the input noisy audio speech with a help of audio-visual correspondence. The proposed V-CAFE is designed to capture the transition of lip movements, namely visual context and to generate a noise reduction mask by considering the obtained visual context. Through context-dependent modeling, the ambiguity in viseme-to-phoneme mapping can be refined for mask generation. The noisy representations are masked out with the noise reduction mask resulting in enhanced audio features. The enhanced audio features are fused with the visual features and taken to an encoder-decoder model composed of Conformer and Transformer for speech recognition. We show the proposed end-to-end AVSR with the V-CAFE can further improve the noise-robustness of AVSR. The effectiveness of the proposed method is evaluated in noisy speech recognition and overlapped speech recognition experiments using the two largest audio-visual datasets, LRS2 and LRS3.

Title: Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Network

Authors: Byung-Kwan Lee*, Junho Kim*, Yong Man Ro (*: equally contributed) 

Adversarial examples provoke weak reliability and potential security issues in deep neural networks. Although adversarial training has been widely studied to improve adversarial robustness, it works in an over-parameterized regime and requires high computations and large memory budgets. To bridge adversarial robustness and model compression, we propose a novel adversarial pruning method, Masking Adversarial Damage (MAD) that employs second-order information of adversarial loss function. By using it, we can accurately estimate adversarial saliency for model parameters and determine which parameters can be pruned without weakening adversarial robustness. Furthermore, we reveal that model parameters of initial layer are highly sensitive to the adversarial examples and show that compressed feature representation retains semantic information for the target objects. Through extensive experiments on three public datasets, we demonstrate that MAD effectively prunes adversarially trained networks without loosing adversarial robustness and shows better performance than previous adversarial pruning methods.

Title: Weakly Paired Associative Learning for Sound and Image Representations via Bimodal Associative Memory

Authors: Sangmin Lee, Hyung-Il Kim, and Yong Man Ro 

Data representation learning without labels has attracted increasing attention due to its nature that does not require human annotation. Recently, as data samples are acquired in multi-sensory environments, representation learning has been extended to bimodal data, especially sound and image which are closely related to basic human senses. Existing sound and image representation learning methods necessarily require a large number of sound and image with corresponding pairs. Therefore, it is difficult to ensure the effectiveness of the methods in the weakly paired condition, which lacks paired bimodal data. In fact, according to human cognitive studies, the cognitive functions in the human brain for a certain modality can be enhanced by receiving other modalities, even not directly paired ones. Based on the observation, we propose a new problem to deal with the weakly paired condition: How to boost a certain modal representation even by using other unpaired modal data. To address the issue, we introduce a novel bimodal associative memory (BMA-Memory) with key-value switching that can store bimodal features in sound-image sub-memories and naturally associate with one another. BMA-Memory enables to build sound-image association with small paired bimodal data and to boost the built association with easily obtainable large amount of unpaired data. Through the proposed associative learning, it is possible to reinforce the representation of a certain modality (e.g., sound) even by using other unpaired modal data (e.g., images).

1. Authors: Taeheon Kim, Hong Joo Lee, and Yong Man Ro

Title: MAP: Multispectral Adversarial Patch to Attack Person Detection

Recently, multispectral person detection has shown great performances in real world applications such as autonomous driving and security systems. However, the reliability of person detection against physical attacks has not been fully explored yet in multispectral person detectors. To evaluate the robustness of multispectral person detectors in the physical world, we propose a novel Multispectral Adversarial Patch (MAP) generation framework. MAP is optimized with a Cross-spectral Mapping(CSM) and Material Emissivity(ME) loss. This paper is the first to evaluate the reliability of a multispectral person detector against physical attack. Throughout experiments, our proposed adversarial patch successfully attacks the person detector and the Average Precision (AP) score is dropped by 90.79% in digital space and 73.34% in physical space.

2. Authors: Sungjune Park, Dae Hwi Choi, Jung Uk Kim, and Yong Man Ro

Title: Robust Thermal Infrared Pedestrian Detection by Associating Visible Pedestrian Knowledges

Recently, pedestrian detection on thermal infrared images has shown the robust pedestrian detection performance. In this paper, we propose a novel thermal infrared pedestrian detection framework which can associate and utilize the complementary pedestrian knowledge from visible images. Motivated by that humans can associate useful information from other sensors to perform a more reliable decision, we devise a Visible-sensory Pedestrian Associating (VPA) Memory to conduct the robust pedestrian detection by utilizing complementary visible-sensory pedestrian knowledge explicitly. The VPA Memory is trained to store the pedestrian information of visible images and associate it with a given thermal infrared pedestrian knowledge via the memory associating learning. We verify the effectiveness of the proposed framework by conducting extensive experiments, and it achieves state-of the-art pedestrian detection performances on thermal infrared images.

Title: Distinguishing Homophenes using Multi-head Visual-audio Memory for Lip Reading

Authors: Minsu Kim, Jeong Hun Yeo, and Yong Man Ro

Recognizing speech from silent lip movement, which is called lip reading, is a challenging task due to 1) the inherent information insufficiency of lip movement to fully represent the speech, and 2) the existence of homophenes that have similar lip movement with different pronunciations. In this paper, we try to alleviate the aforementioned two challenges in lip reading by proposing a Multi-head Visual-audio Memory (MVM). Firstly, MVM is trained with an audio-visual dataset and remembers audio representations by modelling the inter-relationships of a paired audio-visual representations. At the inference stage, visual input alone can extract the saved audio representation from the memory by examining the learned inter-relationships. Therefore, the lip reading model can complement the insufficient visual information with the extracted audio representations. Secondly, MVM is composed of multi-head key memories for saving visual features and one value memory for saving audio knowledge, which is designed to distinguish the homophenes. With the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input lip movement. This also can be viewed as an explicit implementation of the one-to-many mapping of viseme-to-phoneme. Moreover, MVM is employed in multi-temporal levels to consider the context when retrieving the memory and distinguish the homophenes. Extensive experimental results verify the effectiveness of the proposed method in lip reading and in distinguishing the homophenes.

Title: SyncTalkFace: Talking Face Generation with Precise Lip-syncing via Audio-Lip Memory

Authors: Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro

The challenge of talking face generation from speech lies in aligning two different modal information, audio and video, such that the mouth region corresponds to input audio. Previous methods either exploit audio-visual representation learning or leverage intermediate structural information such as landmarks and 3D models. However, they struggle to synthesize fine details of the lips varying at the phoneme level as they do not sufficiently provide visual information of the lips at the video synthesis step. To overcome this limitation, our work proposes Audio-Lip Memory that brings in visual information of the mouth region corresponding to input audio and enforces fine-grained audio-visual coherence. It stores lip motion features from sequential ground truth images in the value memory and aligns them with corresponding audio features so that they can be retrieved using audio input at inference time. Therefore, using the retrieved lip motion features as visual hints, it can easily correlate audio with visual dynamics in the synthesis step. By analyzing the memory, we demonstrate that unique lip features are stored in each memory slot at the phoneme level, capturing subtle lip motion based on memory addressing. In addition, we introduce visual-visual synchronization loss which can enhance lip-syncing performance when used along with audio-visual synchronization loss in our model. Extensive experiments are performed to verify that our method generates high-quality video with mouth shapes that best align with the input audio, outperforming previous state-of-the-art methods.

Title: Towards Versatile Pedestrian Detector with Multisensory-Matching and Multispectral Recalling Memory

Authors: Jung Uk Kim, Sungjune Park, and Yong Man Ro

Recently, automated surveillance cameras can change a visible sensor and a thermal sensor for all-day operation. However, existing single-modal pedestrian detectors mainly focus on detecting pedestrians in only one specific modality (i.e., visible or thermal), so they cannot effectively cope with other modal inputs. In addition, recent multispectral pedestrian detectors have shown remarkable performance by adopting multispectral modalities, but they also have limitations in practical applications (e.g., different Field-of-View (FoV) and frame rate). In this paper, we introduce a versatile pedestrian detector that shows robust detection performance in any single modality. We propose a multisensory-matching contrastive loss to reduce the difference between the visual representation of pedestrians in the visible and the thermal modalities. Moreover, to make the proposed method perform robust detection on a single modality, we design a Multispectral Recalling (MSR) Memory. The MSR Memory enhances the visual representation of the single modal features by recalling that of the multispectral modalities. To guide the MSR Memory to store the contexts of the multispectral modalities, we introduce a multispectral recalling loss. It enables the pedestrian detector to encode more discriminative features in a single input modality. We would like to insist that our method is a step forward detector that can be applied to a variety of real-world applications. The comprehensive experimental results verify the effectiveness of the proposed method.

Title: Robust Perturbation for Visual Explanation: Cross-checking Mask Optimization to Avoid ClassDistortion

Authors: Junho Kim, Seongyeop Kim, Seong Tae Kim, and Yong Man Ro

Along with the outstanding performance of the deep neural networks (DNNs), considerable research efforts have been devoted to finding ways to understand the decision of DNNs structures. In the computer vision domain, visualizing the attribution map is one of the most intuitive and understandable ways to achieve human-level interpretation. Among them, perturbation-based visualization can explain the “black box” property of the given network by optimizing perturbation masks that alter the network prediction of the target class the most. However, existing perturbation methods could make unexpected changes to network predictions after applying a perturbation mask to the input image, resulting in a loss of robustness and fidelity of the perturbation mechanisms. In this paper, we define class distortion as the unexpected changes of the network prediction during the perturbation process. To handle that, we propose a novel visual interpretation framework, Robust Perturbation, which shows robustness against the unexpected class distortion during the mask optimization. With a new cross-checking mask optimization strategy, our proposed framework perturbs the target prediction of the network while upholding the non-target predictions, providing more reliable and accurate visual explanations. We evaluate our framework on three different public datasets through extensive experiments. Furthermore, we propose a new metric for class distortion evaluation. In both quantitative and qualitative experiments, tackling the class distortion problem turns out to enhance the quality and fidelity of the visual explanation in comparison with the existing perturbation-based methods.

Title: Speech Reconstruction with Reminiscent Sound via Visual Voice Memory

Authors: Joanna Hong, Minsu Kim, Se Jin Park, Yong Man Ro

The goal of this work is to reconstruct speech from silent video, in both speaker dependent and speaker independent ways. Unlike previous works that have been mostly restricted to a speaker dependent setting, we propose Visual Voice memory to restore essential auditory information to generate proper speech from different speakers and even unseen speakers. The proposed memory takes additional auditory information that corresponds to the input face movements and stores the auditory contexts that can be recalled by the given input visual features. Specifically, the Visual Voice memory contains value and key memory slots, where value memory slots are for saving the audio features, and key memory slots are for storing the visual features in the same location of the saved audio features. Guiding each memory to properly save each feature, the model can adequately produce the speech through auxiliary information of audio. Hence, our method employs both video and audio information during training time, but does not require any additional auditory input in the inference time. Our key contributions are: (1) proposing the Visual Voice memory that brings rich information of audio that complements the visual features, thus producing high-quality speech from silent video, and (2) enabling multi-speaker and speaker independent training by memorizing auditory features and the corresponding visual features. We validate the proposed framework on GRID and Lip2Wav datasets and show that our method surpasses the performance of previous works. Moreover, we experiment on both multi-speaker and speaker independent settings and verify the effectiveness of the Visual Voice memory. We also demonstrate that the Visual Voice memory contains meaningful information to reconstruct speech.

Title: M-CAM: Visual Explanation of Challenging Conditioned Dataset with Bias-reducing Memory

Authors: Seongyeop Kim and Yong Man Ro

We introduce a framework that enhances visual explanation of class activation map (CAM) with key-value memory structure for deep networks. We reveal challenging conditions inherently existing in several datasets that degrade the visual explanation quality of existing CAM-based visual explanation methods (e.g. imbalanced data, multi-object co-occurrence) and try to solve it with the proposed framework. The proposed Bias-reducing memory module learns spatial feature representation of different classes from trained networks and stores each different semantic information in separate memory slots, while it does not require any modification to the existing networks. Furthermore, we propose a novel visual explanation method accompanied by a memory slot searching algorithm to retrieve semantically relevant spatial feature representation from the memory module and make visual explanation of network decisions. We evaluate our visual explanation framework with datasets of challenging conditions including several medical image datasets and multiclass classification datasets. We qualitatively and quantitatively compare it with existing CAM-based methods to demonstrate the strength of our framework.

Title: Lip to Speech Synthesis with Visual Context Attentional GAN

Authors: Minsu Kim, Joanna Hong, and Yong Man Ro

In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis. Specifically, the proposed VCA-GAN synthesizes the speech from local lip visual features by finding a mapping function of visemes-to-phonemes, while global visual context is embedded into the intermediate speech representation to refine the coarse speech representation in details. To achieve this, a visual context attention module is proposed where it encodes global representations from the local visual features and provides the desired global visual context corresponding to the given coarse speech representation to the generator. In addition to the explicit modelling of local and global visual representations, a synchronization technique is introduced through contrastive learning that guides the generator to synthesize a speech in sync with the given input lip movements. Extensive experiments demonstrate that the proposed VCA-GAN outperforms existing state-of-the-art and is able to effectively synthesize the speech from multi-speaker that has been barely handled in the previous works.

Title: Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck

Authors: Junho Kim*, Byung-Kwan Lee*, and Yong Man Ro (*: equally contributed)

Adversarial examples, generated by carefully crafted perturbation, have attracted considerable attention in research fields. Recent works have argued that the existence of the robust and non-robust features is a primary cause of the adversarial examples, and investigated their internal interactions in the feature space. In this paper, we propose a way of explicitly distilling feature representation into the robust and non-robust features, using Information Bottleneck. Specifically, we inject noise variation to each feature unit and evaluate the information flow in the feature representation to dichotomize feature units either robust or non-robust, based on the noise variation magnitude. Through comprehensive experiments, we demonstrate that the distilled features are highly correlated with adversarial prediction, and they have human-perceptible semantic information by themselves. Furthermore, we present an attack mechanism intensifying the gradient of non-robust features that is directly related to the model prediction, and validate its effectiveness of breaking model robustness.

Title: CroMM-VSR: Cross-Modal Memory Augmented Visual Speech Recognition

Authors: Minsu Kim, Joanna Hong, Se Jin Park, Yong Man Ro

Visual Speech Recognition (VSR) is a task that recognizes speech from external appearances of the face (i.e., lips) into text. Since the information from the visual lip movements is not sufficient to fully represent the speech, VSR is considered as one of the challenging problems. One possible way to resolve this problem is additionally utilizing audio which contains rich information for speech recognition. However, the audio information could not be always available such as in long-distance or crowded situations. Thus, it is necessary to find a way that successfully provides enough information for speech recognition with visual inputs only. In this paper, we alleviate the information insufficiency of visual lip movement by proposing a cross-modal memory augmented VSR with Visual-Audio Memory (VAM). The proposed framework tries to utilize the complementary information of audio even when the audio inputs are not provided at the inference time. Concretely, the proposed VAM learns to imprint audio features of short clip-level into a memory network using the corresponding visual features. To this end, the VAM contains two memories, lip-video key and audio value. The audio value memory is guided to imprint the audio feature and the lip-video key memory is guided to memorize the location of the imprinted audio. By doing this, the VAM can exploit rich audio information by accessing the memory using visual inputs only. Thus, the proposed VSR framework can refine the prediction with the imprinted audio information during inference time where the audio inputs are not provided. We validate the proposed method on popular benchmark databases, LRW, LRW-1000, GRID, and LRS2. Experimental results show that the proposed method achieves state-of-the-art performance on both word- and sentence-level visual speech recognition. In addition, we verify the learned representations inside the VAM contain meaningful information for VSR by examining and visualizing the learned representations.

2022년도 전기 박사과정 (국비), 석사과정 (국비 및 KAIST 장학), 산학장학생 (KEPSI, EPSS, LGenius) 등을 모집합니다.

(http://admission.kaist.ac.kr/graduate/)

모집 연구분야:

 - Deep learning (XAI, adversarial attack/defense, multimodal)

 - Machine learning with visual data

 - Computer vision (object segmentation/detection/classification)

 - multimodal (Vision-Language) Deep learning

 - Defense security

현재 진행중인 연구과제:

 - Explainable (Interpretable) Deep learning

 - Adversarial defense in Deep learning

 - Deep learning algorithms (detection/classification/segmentation) in computer vision

 - Multimodal deep learning

최근 연구실 연구결과 - 링크 (LINK)

최근 연구실 석박사과정 딥러닝 관련 해외 학회 발표실적 - 링크 (LINK)

최근 연구실 석박사과정 해외 저널 실적 - 링크 (LINK)

을 참고하세요.

연구실 입학 문의는 노용만 교수님(ymro@kaist.ac.kr)께 이메일/사전미팅 하기 바랍니다.

Title: Assessing Individual VR Sickness through Deep Feature Fusion of VR Video and Physiological Response

Authors: Sangmin Lee, Seongyeop Kim, Hak Gu Kim, and Yong Man Ro

Recently, VR sickness assessment for VR videos is highly demanded in industry and research fields to address VR viewing safety issues. Especially, it is difficult to evaluate VR sickness of individuals due to individual differences. To achieve the challenging goal, we focus on deep feature fusion of sickness-related information. In this paper, we propose a novel deep learning-based assessment framework which estimates VR sickness of individual viewers with VR videos and corresponding physiological responses. We design the content stimulus guider imitating the phenomenon that humans feel VR sickness. The content stimulus guider extracts a deep stimulus feature from a VR video to reflect VR sickness caused by VR videos. In addition, we devise the physiological response guider to encode physiological responses that are acquired while humans experience VR videos. Each physiology sickness feature extractor (EEG, ECG, and GSR) in the physiological response guider is designed to suit their physiological characteristics. Extracted physiology sickness features are then fused into a deep physiology feature that comprehensively reflects individual deviations of VR sickness. Finally, the VR sickness predictor assesses individual VR sickness effectively with the fusion of the deep stimulus feature and the deep physiology feature. To validate the proposed method extensively, we built two benchmark datasets which contain 360-degree VR videos with physiological responses (EEG, ECG, and GSR) and SSQ scores. Experimental results show that the proposed method achieves meaningful correlations with human SSQ scores. Further, we validate the effectiveness of the proposed network designs by conducting analysis on feature fusion and visualization.

Title: Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video

Authors: Minsu Kim*, Joanna Hong*, Se Jin Park, and Yong Man Ro (*: equally contributed)

In this paper, we introduce a novel audio-visual multi-modal bridging framework that can utilize both audio and visual information, even with uni-modal inputs. We exploit a memory network to achieve the multi-modal bridging, where the memory network consists of two modality-specific memories: source-key and target-value memories. These two modality-specific memories save a source and a target modal representations, respectively. Then, an associative bridge is constructed between the source-key memory and the target-value memory, regarding the interrelationship between the two memories. By learning the interrelationship through the associative bridge, it is possible to access the target-value memory using source modality and source-key memory without target modality. Accordingly, the proposed framework can recall the target modal representations with source modal inputs only and provides rich information for its downstream tasks. We apply the proposed framework to two tasks: lip reading and speech reconstruction from silent video. Through the proposed associative bridge and modality-specific memories, each task knowledge is enriched with the recalled audio context, achieving state-of-the-art performance. We also verify that the associative bridge properly relates the source and target memories.

Title: Robust Small-scale Pedestrian Detection with Cued Recall via Memory Learning

Authors: Jung Uk Kim*, Sungjune Park*, and Yong Man Ro (*: equally contributed)

Although the visual appearances of small-scale objects are not well observed, humans can recognize them by associating the visual cues of small objects from their memorized appearance. It is called cued recall. In this paper, motivated by the memory process of humans, we introduce a novel pedestrian detection framework that imitates cued recall in detecting the small-scale pedestrians. We propose a large-scale embedding learning with the large-scale pedestrian recalling memory (LPR Memory). The purpose of the proposed large-scale pedestrian embedding learning is to memorize and recall the large-scale pedestrian appearance via the LPR Memory. To this end, we employ the large-scale pedestrian exemplar set, so that, the LPR Memory can recall the information of the large-scale pedestrians from the small-scale pedestrians. Comprehensive quantitative and qualitative experimental results validate the effectiveness of the proposed framework with the LPR Memory.

1. Authors: Junho Kim, Minsu Kim, Yong Man Ro

Title: 'Interpretation of Lesional Detection via Counterfactual Generation'

2. Authors: Hong Joo Lee, Yong Man Ro

Title: 'Adversarially Robust Multi-Sensor Fusion Model Training via Random Feature Fusion for Semantic Segmentation'

3. Authors: Byeong Cheon Kim*, Youngjoon Yu*, Yong Man Ro *equally contributed first author

Title: 'Robust Decision-based black-box adversarial attack via Coarse-to-fine Random Search'

Title: Adversarially Robust Hyperspectral Image Classification via Random Spectral Sampling and Spectral Shape Encoding

Authors: Sungjune Park, Hong Joo Lee, Yong Man Ro

Although the hyperspectral image (HSI) classification has adopted deep neural networks (DNNs) and shown remarkable performances, there is a lack of studies of the adversarial vulnerability for the HSI classifications. In this paper, we propose a novel HSI classification framework robust to adversarial attacks. To this end, we focus on the unique spectral characteristic of HSIs (i.e., distinctive spectral patterns of materials). With the spectral characteristic, we present the random spectral sampling and spectral shape feature encoding for the robust HSI classification. For the random spectral sampling, spectral bands are randomly sampled from the entire spectrum for each pixel of the input HSI. Also, the overall spectral shape information, which is robust to adversarial attacks, is fed into the shape feature extractor to acquire the spectral shape feature. Then, the proposed framework can provide the adversarial robustness of HSI classifiers via randomization effects and spectral shape feature encoding. To the best of our knowledge, the proposed framework is the first work dealing with the adversarial robustness in the HSI classification. In experiments, we verify that our framework improves the adversarial robustness considerably under diverse adversarial attack scenarios, and outperforms the existing adversarial defense methods.

Title: Uncertainty-Guided Cross-Modal Learning for Robust Multispectral Pedestrian Detection

Authors: Jung Uk Kim, Sungjune Park, Yong Man Ro

Multispectral pedestrian detection has received great attention in recent years as multispectral modalities (i.e. color and thermal) can provide complementary visual information. However, there are major inherent issues in multispectral pedestrian detection. First, the cameras of the two modalities have different field-of-views (FoVs), so that image pairs are often miscalibrated. Second, modality discrepancy is observed, because image pairs are captured at different wavelengths. In this paper, to alleviate these issues, we propose a new uncertainty-aware multispectral pedestrian detection framework. In our framework, we consider two types of uncertainties: (1) Region of Interest (RoI) uncertainty and (2) predictive uncertainty. For the miscalibration issue, we propose RoI uncertainty which represents the reliability of the RoI candidates. With the RoI uncertainty, when combining two modal features, we devise uncertainty-aware feature fusion (UFF) module to reduce the effect of RoI features with high RoI uncertainty. We also propose uncertainty-aware cross-modal guiding (UCG) module for the modality discrepancy. In the UCG module, we use the predictive uncertainty, which indicates how reliable the prediction of the RoI feature is. Based on the predictive uncertainty, the UCG module guides the feature distribution of high predictive uncertain (less reliable) modality to resemble that of low predictive uncertain (more reliable) modality. The UCG module can encode more discriminative features by guiding feature distributions of two modalities to be similar. With comprehensive experiments on the public multispectral datasets, we verified that our method reduces the effect of the miscalibration and alleviates the modality discrepancy, outperforming existing state-of-the-art methods.

Title: Uncertainty-Guided Cross-Modal Learning for Robust Multispectral Pedestrian Detection

Authors: Jung Uk Kim, Sungjune Park, Yong Man Ro

Multispectral pedestrian detection has received great attention in recent years as multispectral modalities (i.e. color and thermal) can provide complementary visual information. However, there are major inherent issues in multispectral pedestrian detection. First, the cameras of the two modalities have different field-of-views (FoVs), so that image pairs are often miscalibrated. Second, modality discrepancy is observed, because image pairs are captured at different wavelengths. In this paper, to alleviate these issues, we propose a new uncertainty-aware multispectral pedestrian detection framework. In our framework, we consider two types of uncertainties: (1) Region of Interest (RoI) uncertainty and (2) predictive uncertainty. For the miscalibration issue, we propose RoI uncertainty which represents the reliability of the RoI candidates. With the RoI uncertainty, when combining two modal features, we devise uncertainty-aware feature fusion (UFF) module to reduce the effect of RoI features with high RoI uncertainty. We also propose uncertainty-aware cross-modal guiding (UCG) module for the modality discrepancy. In the UCG module, we use the predictive uncertainty, which indicates how reliable the prediction of the RoI feature is. Based on the predictive uncertainty, the UCG module guides the feature distribution of high predictive uncertain (less reliable) modality to resemble that of low predictive uncertain (more reliable) modality. The UCG module can encode more discriminative features by guiding feature distributions of two modalities to be similar. With comprehensive experiments on the public multispectral datasets, we verified that our method reduces the effect of the miscalibration and alleviates the modality discrepancy, outperforming existing state-of-the-art methods.

2021년도 후기 박사과정 (KAIST 장학), 석사과정 (국비 및 KAIST 장학), 산학장학생 (KEPSI, EPSS, LGenius) 등을 모집합니다.

(http://admission.kaist.ac.kr/graduate/)

모집 연구분야:

 - Deep learning (XAI, adversarial attack/defense, multimodal)

 - Machine learning with visual data

 - Computer vision (object segmentation/detection/classification)

 - multimodal (Vision-Language) Deep learning

 - Defense security

현재 진행중인 연구과제:

 - Explainable (Interpretable) Deep learning

 - Adversarial defense in Deep learning

 - Deep learning algorithms (detection/classification/segmentation) in computer vision

 - Multimodal deep learning

최근 연구실 연구결과 - 링크 (LINK)

최근 연구실 석박사과정 딥러닝 관련 해외 학회 발표실적 - 링크 (LINK)

최근 연구실 석박사과정 해외 저널 실적 - 링크 (LINK)

을 참고하세요.

연구실 입학 문의는 노용만 교수님(ymro@kaist.ac.kr)께 이메일/사전미팅 하기 바랍니다.

Title: Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning

Authors: Sangmin Lee, Hak Gu Kim, Dae Hwi Choi, Hyung-Il Kim, Yong Man Ro

Our work addresses long-term motion context issues for predicting future frames. To predict the future precisely, it is required to capture which long-term motion context (e.g., walking or running) the input motion (e.g., leg movement) belongs to. The bottlenecks arising when dealing with the long-term motion context are: (i) how to capture the long-term motion context naturally matching input sequences with limited dynamics, (ii) how to capture the long-term motion context with high-dimensionality (e.g., motion complexity). To address the issues, we propose novel motion context-aware video prediction. To solve the bottleneck (i), we introduce a long-term motion context memory (LMC-Memory) with memory alignment learning. The proposed memory alignment learning enables to store long-term motion contexts into the memory and to match them with sequences including limited dynamics. As a result, the long-term context can be recalled from the limited input sequence. In addition, to resolve the bottleneck (ii), we propose memory query decomposition to store local motion context (i.e., low-dimensional dynamics) and recall the suitable local context for each local part of the input individually. It enables to boost the alignment effects of the memory. Experimental results show the proposed method outperforms other sophisticated RNN-based methods, especially in the long-term condition. Further, we validate the effectiveness of the proposed network designs by conducting ablation studies and memory feature analysis.

Title: Towards Robust Training of Multi-Sensor data Fusion Network Against Adversarial Examples in Semantic Segmentation

Authors: Youngjoon Yu, Hong Joo Lee, Byeong Cheon Kim, Jung Uk Kim, and Yong Man Ro

The success of multi-sensor data fusions in deep learning appears to be attributed to the use of complementary information among multiple sensor datasets. Compared to their predictive performance, relatively less attention has been devoted to the adversarial robustness of multi-sensor data fusion models. To achieve adversarial robust multi-sensor data fusion networks, we propose here a novel robust training scheme called Multi-Sensor Cumulative Learning (MSCL). The motivation behind the MSCL method is based on the way human beings learn new skills. The MSCL allows the multi-sensor fusion network to learn robust features from individual sensors, and then learn complex joint features from multiple sensors just as people learn to walk before they run. The step wise framework of MSCL enables the network to incorporate pre-trained knowledge of robustness with new joint information from multiple sensors. Extensive experimental evidence validated that the MSCL outperforms other multi-sensor fusion training in defending against adversarial examples.

Dr. Seong Tae Kim, who received Ph.D (Advisor: Prof. Yong Man Ro) in 2019, is appointed as an assistant professor of the department of computer science and engineering at Kyung Hee University. He has established a research laboratory named of ‘Augmented Intelligence Lab’ https://sites.google.com/view/augilab).

Title: Visual Comfort Aware-Reinforcement Learning for Depth Adjustment of Stereoscopic 3D Images

Authors: Hak Gu Kim, Minho Park, Sangmin Lee, Seongyeop Kim, Yong Man Ro

Depth adjustment aims to enhance the visual experience of stereoscopic 3D (S3D) images, which accompanied with improving visual comfort and depth perception. For a human expert, the depth adjustment procedure is a sequence of iterative decision making. The human expert iteratively adjusted the depth until he is satisfied with the both levels of visual comfort and the perceived depth. In this work, we present a novel deep reinforcement learning (DRL)-based approach for depth adjustment named VCA-RL (Visual Comfort Aware Reinforcement Learning) to explicitly model human sequential decision making in depth editing operations. We formulate the depth adjustment process as a Markov decision process where actions are defined as camera movement operations to control the distance between the left and right cameras. Our agent is trained based on the guidance of an objective visual comfort assessment metric to learn the optimal sequence of camera movement actions in terms of perceptual aspects in stereoscopic viewing. With extensive experiments and user studies, we show the effectiveness of our VCA-RL model on three different S3D databases.

Title: Towards a Better Understanding of VR Sickness: Physical Symptom Prediction for VR Contents

Authors: Hak Gu Kim, Sangmin Lee, Seongyeop Kim, Heoun-taek Lim, and Yong Man Ro

We address the black-box issue of VR sickness assessment (VRSA) by evaluating the level of physical symptoms of VR sickness. For the VR contents inducing the similar VR sickness level, the physical symptoms can vary depending on the characteristics of the contents. Most of existing VRSA methods focused on assessing the overall VR sickness score. To make better understanding of VR sickness, it is required to predict and provide the level of major symptoms of VR sickness rather than overall degree of VR sickness. In this paper, we predict the degrees of main physical symptoms affecting the overall degree of VR sickness, which are disorientation, nausea, and oculomotor. In addition, we introduce a new large-scale dataset for VRSA including 360 videos with various frame rates, physiological signals, and subjective scores. On VRSA benchmark and our newly collected dataset, our approach shows a potential to not only achieve the highest correlation with subjective scores, but also to better understand which symptoms are the main causes of VR sickness.

CUA Loss: Class Uncertainty-Aware Gradient Modulation for Robust Object Detection

Authors: Jung Uk Kim, Seong Tae Kim, Hong Joo Lee, Sangmin Lee, and Yong Man Ro

Recently, a wide range of research on object detectionhas shown breakthrough performance. However, in a

challenging environment, such as occlusion and small object cases, object detectors still produce inaccurate or erroneous predictions. To effectively cope with such conditions, most of the existing methods have suggested loss functions to guide the object detectors by modulating the magnitude of their loss. However, when modulating the loss function, they are highly dependent on the classification score of the object detector. It is a known fact that deep neural networks tend to be overconfident in their predictions. In this paper, to alleviate the problem of the object detectors which heavily rely on the prediction in the training phase, we devise a novel loss function called class uncertainty-aware (CUA) loss. CUA loss considers the predictive ambiguity as well as the predictions on classification score when modulating loss function. In addition to the classification score, CUA loss further modulates the loss gradient in an increasing way when the object detectors output an uncertain prediction. Therefore, object detectors with CUA loss effectively cope with challenging environments where prediction result is uncertain. With comprehensive experiments on three public datasets (i.e. PASCAL VOC, MS COCO, and Berkeley DeepDrive), we verified that our CUA loss enhanced the accuracy of the object detectors and outperformed previous state-of-the-art loss functions.

Title: Robust Multispectral Pedestrian Detection via Uncertain-Aware Cross-Modal Learning

Authors: Sungjune Park, Jung Uk Kim, Yeon Gyun Kim, Sang-Keun Moon and Yong Man Ro

With the development of deep neural networks, multispectral pedestrian detection has been received a great attention by exploiting complementary properties of multiple modalities (e.g., color-visible and thermal modalities). Previous works usually rely on network prediction scores in combining complementary modal information. However, it is widely known that deep neural networks often show overconfident problem which results in limited performance. In this paper, we propose a novel uncertainty-aware cross-modal learning to alleviate the aforementioned problem in multispectral pedestrian detection. First, we extract

object region uncertainty which represents the reliability of object region features in multiple modalities. Then, we combine each modal object region feature considering object region uncertainty. Second, we guide the classifier of detection framework with soft target labels to be aware of the level of object region uncertainty in multiple modalities. To verify the effectiveness of the proposed methods, we conduct extensive experiments with various detection frameworks on two public datasets (i.e., KAIST Multispectral Pedestrian Dataset and CVC-14).

Title: Unsupervised Disentangling of Viewpoint and Residues Variations by Substituting Representations for Robust Face Recognition

Authors: Minsu Kim, Joanna Hong, Junho Kim, Hong Joo Lee, Yong Man Ro

It is well-known that identity-unrelated variations (e.g., viewpoint or illumination) degrade the performances of face recognition methods. In order to handle this challenge, a robust method for disentangling the identity and view representations has drawn an attention in the machine learning area. However, existing methods learn discriminative features which require a manual supervision of such factors of variations. In this paper, we propose a novel disentangling framework through modeling three representations of identity, viewpoint, and residues (i.e., identity and pose unrelated) which do not require supervision of the variations. By jointly modeling the three representations, we enhance the disentanglement of each representation and achieve robust face recognition performance. Further, the learned viewpoint representation can be utilized for pose estimation or editing of a posed facial image. Extensive quantitative and qualitative evaluations verify the effectiveness of our proposed method which disentangles identity, viewpoint, and residues of facial images. 

2021년도 전기 박사과정, 석사과정, 산학장학생 (KEPSI, EPSS, LGenius) 등을 모집합니다.

(http://admission.kaist.ac.kr/graduate/)

모집 연구분야:

 - Deep learning (XAI, adversarial attack/defense, multimodal)

 - Machine learning with visual data

 - Computer vision (object segmentation/detection/classification)

 - multimodal (Vision-Language) Deep learning

 - Medical imaging/ Defense security

현재 진행중인 연구과제:

 - Explainable (Interpretable) Deep learning

 - Adversarial attack/defense in Deep learning

 - Deep learning algorithms (detection/classification/segmentation) in computer vision

 - Multimodal deep learning

최근 연구실 연구결과 - 링크 (LINK)

최근 연구실 석박사과정 딥러닝 관련 해외 학회 발표실적 - 링크 (LINK)

최근 연구실 석박사과정 해외 저널 실적 - 링크 (LINK)

을 참고하세요.

연구실 입학 문의는 노용만 교수님(ymro@kaist.ac.kr)께 이메일/사전미팅 하기 바랍니다.

Title: Dual-Branch Structured De-Striping Convolution Network Using Parametric Noise Model

Authors: Jongho Lee and Yong Man Ro

Abstract: The stripe fixed pattern noise (FPN) of the infrared image significantly corrupts the image quality so that the infrared imaging system suffers from the degradation of observability and detectability during operation. Therefore, the FPN de-striping method, which eliminates stripe patterns without substantial loss of image information, remains a core technology in the field of infrared image processing. In this paper, we propose the dual-branch structured based FPN de-striping deep convolutional neural network to effectively extract the structural features of the FPN and preserve the image details in the single infrared image. In addition, we have established the parametric FPN model through an infrared image diagnosis experiment based on the physical principle of the infrared detector signal response. We have optimized each parameter of the FPN model using measured data, which acquired on a wide range of detector temperatures. Further, we generate the training data using our FPN model to ensure stable learning performance against various stripe patterns. We performed comparative experiments with state-of-the-art methods using artificially corrupted infrared images and real corrupted infrared images, and our proposed method achieved outstanding de-striping results in both qualitative and quantitative evaluation compared with existing methods.

Title: Robust Ensemble Model Training via Random Layer Sampling Against Adversarial Attack

Authors: Hakmin Lee*, Hong Joo Lee*, Seong Tae Kim, and Yong Man Ro

* Both authors contributed equally to this work.

Deep neural networks have achieved substantial achievements in several computer vision areas, but have vulnerabilities that are often fooled by adversarial examples that are not recognized by humans. This is an important issue for security or medical applications. In this paper, we propose an ensemble model training framework with random layer sampling to improve the robustness of deep neural networks. In the proposed training framework, we generate various sampled model through the random layer sampling and update the weight of the sampled model. After the ensemble models are trained, it can hide the gradient efficiently and avoid the gradient-based attack by the random layer sampling method. To evaluate our proposed method, comprehensive and comparative experiment have been conducted on three datasets. Experimental results show that the proposed method improves the adversarial robustness.

Title: SACA Net: Cybersickness Assessment of Individual Viewers for VR Content via Graph-based Symptom Relation Embedding

Authors: Sangmin Lee, Jung Uk Kim, Hak Gu Kim, Seongyeop Kim, and Yong Man Ro

Recently, cybersickness assessment for VR content is in demand to deal with viewing safety issues. Assessing physical symptoms of individual viewers is challenging but important to provide detailed and personalized guides for viewing safety. In this paper, we propose a novel symptom-aware cybersickness assessment network (SACA Net) that quantifies physical symptom levels for assessing cybersickness of individual viewers. SACA Net is designed to utilize the relational characteristics of symptoms for complementary effects among relevant symptoms. The proposed network consists of three main parts: a stimulus symptom context guider, a physiological symptom guider, and a symptom relation embedder. The stimulus symptom context guider and the physiological symptom guider extract symptom features from VR content and human physiology, respectively. The symptom relation embedder refines the stimulus-response symptom features to effectively predict cybersickness by embedding relational characteristics with graph formulation. For validation, we utilize two public 360-degree video datasets that contain cybersickness scores and physiological signals. Experimental results show that the proposed method is effective in predicting human cybersickness with physical symptoms. Further, latent relations among symptoms are interpretable by analyzing relational weights in the proposed network.

1. Authors: Jung Uk Kim*, Sungjune Park*, Yong Man Ro *equally contributed first author

Title: 'Towards Human-Like Interpretable Object Detection Via Spatial Relation Encoding'

2. Authors: Eun Sung Kim*, Jung Uk Kim*, Sangmin Lee, Sang-Keun Moon, Yong Man Ro *equally contributed first author

Title: 'Class Incremental Learning With Task-Selection'

3. Authors: Seongyeop Kim, Sangmin Lee, Yong Man Ro

Title: 'Estimating Vr Sickness Caused By Camera Shake In Vr Videography'

4. Authors: Minsu Kim, Hong Joo Lee, Sangmin Lee, Yong Man Ro

Title: 'Robust Video Facial Authentication With Unsupervised Mode Disentanglement'

5. Authors: Joanna Hong, Jung Uk Kim, Sangmin Lee, Yong Man Ro

Title: 'Comprehensive Facial ____Expression____ Synthesis Using Human-Interpretable Language'

6. Authors: Junho Kim, Minsu Kim, Jung Uk Kim, Hong Joo Lee, Sangmin Lee, Joanna Hong, Yong Man Ro

Title: 'Learning Style Correlation For Elaborate Few-Shot Classification'

7. Authors: Dae Hwi Choi, Hong Joo Lee, Sangmin Lee, Jung Uk Kim, Yong Man Ro

Title: 'Fake Video Detection With Certainty-Based Attention Network'

8. Authors: Byeong Cheon Kim, Jung Uk Kim, Hakmin Lee, Yong Man Ro

Title: 'Revisiting Role Of Autoencoders In Adversarial Settings'

2020년도 후기 박사과정(KAIST장학), 석사과정(국비 및 KAIST장학), 산학장학생 (KEPSI, EPSS, LGenius) 등을 모집합니다.

(http://admission.kaist.ac.kr/graduate/)

모집 연구분야:

 - Deep learning (XAI, adversarial defense, multimodal)

 - Machine learning with visual data

 - Computer vision (object segmentation/detection/classification)

 - multimodal (Vision-Language) Deep learning

 - Medical imaging/ Defense security

현재 진행중인 연구과제:

 - Explainable (Interpretable) Deep learning

 - Adversarial defense in Deep learning

 - Deep learning algorithms (detection/classification/segmentation) in computer vision

 - Multimodal deep learning

최근 연구실 연구결과 - 링크 (LINK)

최근 연구실 석박사과정 딥러닝 관련 해외 학회 발표실적 - 링크 (LINK)

최근 연구실 석박사과정 해외 저널 실적 - 링크 (LINK)

을 참고하세요.

연구실 입학 학생 (국비, KAIST장학생, 지도교수 사전선택 KAIST장학생)은 노용만 교수님(ymro@kaist.ac.kr)께 이메일/사전미팅 하기 바랍니다.

Title: Robust Video Frame Interpolation with Exceptional Motion Map

Authors: Minho Park, Hak Gu Kim, Sangmin Lee, and Yong Man Ro,

Video frame interpolation has increasingly attracted attention in computer vision and video processing fields. When motion patterns in a video are complex, large and non-linear (exceptional motion), the generated intermediate frame is blurred and likely to have large artifacts. In this paper, we propose a novel video frame interpolation considering the exceptional motion patterns. The proposed video frame interpolation takes into account an exceptional motion map that contains the location and intensity of the exceptional motion. The proposed method consists of three parts, which are optical flow based frame interpolation, exceptional motion detection, and frame refinement. The optical flow based frame interpolation predicts an optical flow which is used to synthesize the pre-generated intermediate frame. The exceptional motion detection detects the position and intensity of complex and large motion with the current frame and the previous frame sequence. The frame refinement focuses on the exceptional motion region of the pre-generated intermediate frame by using the exceptional motion map. The proposed video frame interpolation can be robust against the exceptional motion including complex and large motion. Experimental results showed that the proposed video frame interpolation achieved high performance on various public video datasets and especially on videos with exceptional motion patterns.

Title: Structure Boundary Preserving Segmentation for Medical Image with Ambiguous Boundary

Authors: Hong Joo Lee, Jung Uk Kim, Sangmin Lee, Hak Gu Kim and Yong Man Ro

In this paper, we propose a novel image segmentation method to tackle two critical problems of medical image, which are (i) ambiguity of structure boundary in the medical image domain and (ii) uncertainty of the segmented region without specialized domain knowledge. To solve those two problems in automatic medical segmentation, we propose a novel structure boundary preserving segmentation framework. To this end, the boundary key point selection algorithm is proposed. In the proposed algorithm, the key points on the structural boundary of the target object are estimated. Then, a boundary preserving block (BPB) with the boundary key point map is applied for predicting the structure boundary of the target object. Further, for embedding experts’ knowledge in the fully automatic segmentation, we propose a novel shape boundary-aware evaluator (SBE)with the ground-truth structure information indicated by experts. The proposed SBE could give feedback to the segmentation network based on the structure boundary key point. The proposed method is general and flexible enough to be built on top of any deep learning-based segmentation network. We demonstrate that the proposed method could surpass the state-of-the-art segmentation network and improve the accuracy of three different segmentation network models on different types of medical image datasets.

Title: TOWARDS HIGH-PERFORMANCE OBJECT DETECTION: TASK-SPECIFIC DESIGN CONSIDERING CLASSIFICATION AND LOCALIZATION SEPARATION

Authors: Jung Uk Kim, Seong Tae Kim, Eun Sung Kim, Sang-Keun Moon, Yong Man Ro

Object detection performs two tasks (classification and localization) simultaneously. Two tasks share a similarity: they need robust features that effectively represents the visual appearance of the objects. However, two tasks also have different properties. First, classification mainly requires features from discriminative parts of an object to determine the object category, whereas localization mainly requires features from the entire object regions for localizing by drawing a bounding box. Second, classification has a translation invariant property, whereas localization has a translation variant property. In order to increase the efficiency of object detection, it is necessary to design a network in consideration of the commonalities and differences of two tasks. In this work, we simply modified layers of the existing object detection networks into three parts by considering such characteristics: lower-layer feature sharing part, layer separation part, and feature fusion part. As a result, the performance of the proposed method was noticeably improved by properly sharing, separating, and fusing layers of the existing object detection networks

Title: VIDEO FRAME INTERPOLATION VIA EXCEPTIONAL MOTION-AWARE SYNTHESIS

Authors: Minho Park, Sangmin Lee, Yong Man Ro

In this paper, we propose a novel video frame interpolation method via exceptional motion-aware synthesis, in which accurate optical flow could be estimated even with exceptional motion patterns. Specifically, we devise two deep learning modules: exceptional motion detection and frame interpolation with refined flow. The motion detection module detects the position and intensity of exceptional motion patterns in current frame given the past frame sequence. The flow refinement module refines the pre-estimated optical flow for synthesizing the intermediate frame using the information of exceptional motion. The proposed modules improve the quality of the synthesized intermediate frame by making the optical flow robust against exceptional case of motion. Experimental results showed that the proposed method outperforms the state-of-the-art methods qualitatively and quantitatively.

IVIST: Interactive VIdeo Search Tool in VBS 2020                           

Authors: Sungjune Park, Jaeyub Song, Minho Park, Yong Man Ro

This paper presents a new video retrieval tool, Interactive VIdeo Search Tool (IVIST), which participates in the 2020 Video Browser Showdown (VBS). As a video retrieval tool, IVIST is equipped with proper and high performing functionalities such as object detection, dominant-color finding, scene-text recognition and text-image retrieval. These functionalities are constructed with various deep neural networks. By adopting these functionalities, IVIST performs well in searching users’ desirable videos. Furthermore, due to user-friendly user interface, IVIST is easy to use even for novice users. Although IVIST is developed to participate in VBS, we hope that it will be applied as a practical video retrieval tool in the future, dealing with actual video data on the Internet.

Face Tells Detailed Εxpression: Generating Comprehensive Facial Εxpression Sentence through Facial Action Units

Authors: Joanna Hong, Hong Joo Lee, Yelin Kim, Yong Man Ro

Human facial ｅxpression plays the key role in the understanding of the social behavior. Many deep learning approaches present facial emotion recognition and automatic image captioning considering human sentiments. However, most current deep learning models for facial ｅxpression analysis do not contain comprehensive, detailed information of a single face. In this paper, we newly introduce a text-based facial ｅxpression description using several essential components describing comprehensive facial ｅxpression: gender, facial action units, and corresponding intensities. Then, we propose comprehensive facial ｅxpression sentence generating model along with facial ｅxpression recognition model for a single facial image to verify the effectiveness of our text-based dataset. Experimental results show that the proposed two models are supporting each other improving their performances: the text-based facial ｅxpression description provides comprehensive semantic information to the facial emotion recognition model. Also, the visual information from the emotion recognition model guides the facial ｅxpression sentence generation to produce a proper sentence describing comprehensive description.

Title: Encoding Features Robust to Unseen Modes of Variation with Attentive Long Short-Term Memory

Authors: Wissam J. Baddar and Yong Man Ro

Abstract: Long short-term memory (LSTM) is a type of recurrent neural networks that is efficient for encoding spatio-temporal features in dynamic sequences. Recent work has shown that the LSTM retains information related to the mode of variation in the input sequence which reduces the discriminability of the encoded features. To encode features robust to unseen modes of variations, we devise an LSTM adaptation named attentive mode variational LSTM. The proposed attentive mode variational LSTM utilizes the concept of attention to separate the input sequence into two parts: (1) task-relevant dynamic sequence features and (2) task-irrelevant static sequence features. The task-relevant features are used to encode and emphasize the dynamics in the input sequence. The task-irrelevant static sequence features are utilized to encode the mode of variation in the input sequence. Finally, the attentive mode variational LSTM suppresses the effect of mode variation with a shared output gate and results in a spatio-temporal feature robust to unseen variations. The effectiveness of the proposed attentive mode variational LSTM is verified using two tasks: facial ____expression____ recognition and human action recognition. Comprehensive and extensive experiments have verified that the proposed method encodes spatio-temporal features robust to variations unseen during the training.

The title of the paper is "Weather Condition Robust Infrared Image Enhancement via Domain Transfer without Training Dataset Pair (기상 상황에 강인한 적외선 영상 개선을 위한 학습 쌍이 필요 없는 도메인 변환)".