Shiyi Lan 兰石懿

I am a Staff Research Scientist at Waymo. My goal is to build general-purpose multimodal agents that perceive, reason, plan, and act across digital and physical environments. I focus on language-grounded perception, long-horizon decision-making, and robust interaction under uncertainty.

My research spans VLM post-training, multimodal state and world modeling, Reinforcement Learning, and closed-loop perception-to-action learning. I have applied these capabilities to embodied decision-making and end-to-end autonomous driving. I received my Ph.D. in Computer Science from the University of Maryland, College Park in 2022, advised by Prof. Larry S. Davis, and my B.S. from Fudan University in 2018.

Email: voidrank [at] gmail [dot] com

CV   /   Google Scholar   /   GitHub

Waymo, Staff Research Scientist 2026–Present
Building Gemini-based multimodal agents that translate complex visual observations into language-grounded state representations, enabling long-horizon reasoning, decision-making, and robust action in long-tail scenarios. Developing post-training methods for diffusion-based Gemini VLMs and improving RL infrastructure for diffusion models, with autonomous driving as the primary embodied application.

NVIDIA Research, Senior Research Scientist 2023–2026
Core member of Nemotron-Diffusion-VL, responsible for data recipes, evaluation, VLM implementation, and training infrastructure. Led research on multimodal perception-to-action agents, including language-grounded scene understanding, trajectory generation, teacher-student distillation, closed-loop evaluation, and reinforcement learning, applied to end-to-end autonomous driving through the HydraMDP family and OmniDrive.

NVIDIA Research, Research Scientist 2022–2023
Core team member of NV-CLIP; led pre-training and resolved loss-divergence issues. Developed MMAL, the CVPR 2023 mask auto-labeling framework.

NVIDIA Research, Research Intern 2020
Researched weakly supervised instance segmentation on MS COCO.

ByteDance AI Lab, Research Intern 2018
Contributed to a recommendation-system warmup project that reduced daily warmup cost by five million CNY.

• Nemotron-Diffusion-VL — Core member of NVIDIA's first diffusion VLM.
• End-to-End Driving at Scale — 1st place at CVPR 2025 and CVPR 2024 (Team Lead).
• 3D Occupancy Challenge — 1st place at CVPR 2023.
• Robust Vision Challenge — 1st place in the 2022 semantic segmentation track.
• MS COCO 2017 — 1st place in object detection and 2nd place in instance segmentation.

For a complete list, please see my Google Scholar profile.

The first-place weakly supervised instance segmentation model using box labels.

The multi-representation, multi-scale, mutual-relation 3D object detector with transformers.

3D Object Detection with the effective dynamic attention module.

A real-time and high-performance object detector.

Modeling point cloud by leveraging geometric information of point clouds.

Generating instance segmentation candidates by leveraging multi-scale feature pyramids.

• ICPC 2015 Shenyang Regional, Silver Medal (Rank 18/≈300)
• National Olympiad in Informatics of China 2013, Bronze (Rank 122/≈400)

• Temporal-based perception for autonomous systems and applications, US Patent 12,625,926 (2026) (Choi, Lopez, Lan, Asgarieh, Yu)
• Occupancy prediction using forward-backward view transformation, US Patent 12,515,706 (2026) (Li, Yu, Austin, Lan, Kautz, Lopez)
• Point-level supervision for video instance segmentation, US Patent App. 18/395,198 (Yu, Huang, Huang, Lan, Radhakrishnan, Alvarez)
• Class agnostic object mask generation, US Patent 12,614,284 (2026) (Lan, Yu, Radhakrishnan, Lopez, Anandkumar)
• Object detection, instance segmentation, and semantic correspondence from bounding box supervision, US Patent App. 17/177,068 (Yu, Lan, Choy, Radhakrishnan, Liu, Zhu, Anandkumar)