Huan Ling

I am a senior Research Scientist at NVIDIA Spatial Intelligence (TorontoAI) Lab. My research focuses on building foundational generative models. I aim to develop models capable of simulating and generating realistic, diverse, and controllable environments—including video and 3D dynamic scenes — with a focus on enabling physical AI scene-level generation, and new forms of generation and interaction of creative content.

Contact me at: linghuan at cs.torototo.edu

Github / Google Scholar / Twitter

• Oct. 2023: See Through the Eyes of Generative Models, UBC
• July 2023: Vision generation and perception using Diffusion Models, ByteDance AD Core(Slides)
• June 2023: Image, Video and 3D Content Creation with Diffusion Models, BAAI 2023 (Slides)
• June 2023: Align your Latents: VideoLDM, Shanghai AI Lab (Slides)
• July 2021: GANs for 2D Vision Perception, Walmart CV Conf (Slides)
• June 2021: GANs for 2D Vision Perception , IIIS, Tsinghua University (Slides)

Selected publications. Full list at Google Scholar.

Generative Models:

Enhancing 3D reconstructions and novel-view synthesis via single step diffusion inference.

A neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework.

A generative video model with precise Camera Control and temporal 3D Consistency with a 3D cache

Media: Two Minute Papers

Large-scale multimodal control for conditional world generation.

NVIDIA's Cosmos product and open-source models.

Media: Two Minute Papers; 机器之心

A scalable and expressive framework for high-quality dynamic 3D scene reconstruction.

We propose a framework that aligns dynamic 3D Gaussians with text-driven 4D generations.

We present a latent diffusion framework for high-resolution video synthesis. A follow-up open sourced model, Stable Video Diffusion (SVD), is available at huggingface, featuring enhanced datasets and fine-tuned results.

Media: Two Minute Papers

EditGAN enables fine-grained and high-quality semantic edits to images by directly manipulating the latent space of GANs with explicit control over object-level attributes.

Media: Two Minute Papers

Generative Representation Learning:

A novel 3D object detection approach that leverages geometry-aware features derived from diffusion models for more robust 3D understanding.

We propose a self-supervised pretraining framework that uses generative models to supervise image encoders without requiring labeled data.

We extend DatasetGAN to synthesize large-scale datasets like ImageNet with dense pixel-wise annotations, significantly reducing the manual labeling burden.

DatasetGAN leverages the rich latent space of GANs to synthesize annotated datasets with minimal human effort, enabling pixel-level labeling from a few examples.

We introduce a variational framework for completing partially occluded objects, producing plausible and diverse amodal completions for scene understanding.

3D Vision:

GANverse3D

We unify GANs and differentiable rendering for interpretable 3D reconstruction and inverse graphics from single images using GAN priors and neural rendering techniques.

Two-minute paper

We propose DIB-R, a differentiable renderer that enables training of 3D object predictors end-to-end with supervision from 2D images alone.

Interactive Annotation:

ScribbleBox is a practical system that allows users to quickly annotate video object segmentations with scribbles, enabling high-quality masks with minimal effort.

We introduce Curve-GCN, a real-time annotation tool that models object contours as closed curves and uses graph convolution to refine annotations interactively.

Polygon-RNN++ is an annotation system that speeds up segmentation mask creation by predicting object boundaries with minimal clicks via recurrent polygon prediction.

Fun note from year 2023: Yes, we took a bite of RLHF back in 2017 :)

We introduce an early reinforcement learning with human feedback (RLHF) approach for image captioning, where a model learns to improve descriptions through natural language feedback rather than numerical rewards.

I borrowed the template from ✩, ✩,✩,✩.