Ailing Zeng 曾爱玲

Ph.D. Department of Computer Science and Engineering The Chinese University of Hong Kong Senior Researcher Tencent Google Scholar

Biography

I am currently a senior researcher at Tencent AI Lab. Previously, I spent two wonderful years at International Digital Economy Academy (IDEA), leading human-centric perception and generation research team. I obtained my Ph.D. from the Department of Computer Science and Engineering, the Chinese University of Hong Kong, supervised by Prof. Qiang Xu. I was a visiting scholar in the Robotics Institute, Carnegie Mellon University.

My research targets to build multi-modal human-like intelligent agents on scalable big data, especially for Large Motion Models to capture, understand, interact, and generate the motion of humans, animals, and the world. Specifically,

1) Human-centric visual perception with large-scale data and general models: AiOS, SMPLer-X, OSX, DW-Pose, ED-Pose, SmoothNet, DeciWatch

2) Large-scale multi-modality datasets: Motion-X, UBody, Uni-KPT, BallPlay, HuMMan, Human-Art

3) Human-centric generative models: MotionCraft, HumanSD, PhysHOI, Dreamwaltz, HumanTOMATO, DiffSHEG

4) Interactive AI & Human-in-the-loop techniques: X-Pose, Click-Pose, Grounded-SAM

5) Previously, time series analysis and forecasting: LTSF-Linear, SCINet, FITS

I'm looking for self-motivated interns and full-time engineers/researchers. Feel free to reach out if you are interested.

News

• [2024.10] I serve as an Area Chair for CVPR 2025.

• [2024.09] Two papers are accepted to SIGGRAPH Asia and NeurIPS D&B 2024!

• [2024.07] Three papers are accepted to ECCV 2024!

• [2024.05] LTSF-Linear was selected as the most Influential Paper in AAAI 2023!

• [2024.05] One paper on whole-body motion generation is accepted to ICML 2024!

• [2024.04] We are hosting the ECCV 2024 Tutorial on “ Recent Advances in Video Content Understanding and Generation” (VENUE).

• [2024.02] Four papers are accepted to CVPR 2024!

• [2024.01] Three papers are accepted to ICLR 2024!

• [2023.12] We are hosting the CVPR 2024 Workshop on “ Computer Vision with Humans in the Loop” (CVHL).

• [2023] 13 papers are accepted to CVPR/ICLR/NeurIPS/ICCV/AAAI 2023!

• [2022] 4 papers are accepted to ECCV/NeurIPS 2022!

• [2022] Invited Talk at OpenMMlab about Human Motion Modeling from a Time Series Perspective. [Slides] [Video]

Selected Research

The Dawn of Video Generation: Preliminary Explorations with SORA-like Models [Website] Ailing Zeng*, Yuhang Yang*, Weidong Chen, Wei Liu technical report, 2024 A systemantic empirical study on 13 SORA-like text-to-video, image-to-video, video-to-video models.

MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls [Code] Yuxuan Bian, Ailing Zeng#, Xuan Ju, Xian Liu, Zhaoyang Zhang, Wei Liu, Qiang Xu technical report, 2024 MotionCraft unifies (text2motion, speech2gesture, music2dance) whole-body motion generation within a plug-and-play multimodal DiT model.

DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion [Code] Yukun Huang*, Jianan Wang*, Ailing Zeng, Zheng-Jun Zha, Lei Zhang, Xihui Liu Extended Version of DreamWaltz [Neurips 2023] It creates high-quality text-driven 3D avatars and expressive whole-body animation via the proposed Skeleton-guided Score Distillation and Hybrid 3D Gaussian Avatar Representation.

SkillMimic: Learning Reusable Basketball Skills from Demonstrations [Code] [Data] Yinhuai Wang, Qihan Zhao, Runyi Yu, Ailing Zeng#, Jing Lin, Zhengyi Luo, Hok Wai Tsui, Jiwen Yu, Xiu Li, Qifeng Chen, Jian Zhang#, Lei Zhang, Ping Tan technical report, 2024 It proposes a unified reward to learn diverse basketball skills for physically and dynamically plausible humanoid-ball motion simulation from the proposed human-ball motion datasets.

Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation [Code] Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Heng Pan, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, Qifeng Chen SIGGRAPH Asia, 2024 A keypoint controllable diffusion-based framework for expressive image portrait animation.

X-Pose: Detecting Any Keypoints [Code] Jie Yang, Ailing Zeng#, Ruimao Zhang#, Lei Zhang European Conference on Computer Vision (ECCV), 2024 UniPose supports textual and visual prompts to detect arbitrary keypoints (e.g., from articulated, rigid, to soft objects).

HumanTOMATO: Text-aligned Whole-body Motion Generation [Code] Shunlin Lu*, Ling-Hao Chen*, Ailing Zeng#, Jing Lin, Ruimao Zhang#, Lei Zhang, Heung-Yeung Shum# The 41st International Conference on Machine Learning (ICML), 2024 The first text-driven whole-body motion generation method with an explicit text-motion alignment.

Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks [Code] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, Lei Zhang Technical report, 2024 Combining the segment anything model (SAM) with X, e.g., Grounding DINO or OSX for various visual tasks.

PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction [Homepage] [Code] Yinhuai Wang, Jing Lin, Ailing Zeng#, Zhengyi Luo, Jian Zhang#, Lei Zhang Technical report, 2023 The physics-based whole-body Human-Object Interaction (HOI) imitation approach for dynamic HOI and a proposed BallPlay dataset.

AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation [Homepage] [Code] Qingping Sun*, Yanjun Wang*, Ailing Zeng, Wanqi Yin, Chen Wei, Wenjia Wang, Haiyi Mei, Chi Sing Leung, Ziwei Liu, Lei Yang, Zhongang Cai Rank Top-1 on AGORA SMPL-X leaderboard! [link] The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024 An end-to-end DETR-based model for multi-person whole-body mesh recovery.

FreeMan: Towards Benchmarking 3D Human Pose Estimation under Real-World Conditions [Homepage] [Code] [Data] Jiong Wang, Fengyu Yang, Wenbo Gou, Bingliang Li, Danqi Yan, Ailing Zeng, Yijun Gao, Junle Wang, Yanqing Jing, Ruimao Zhang The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024 The first large-scale real-world multi-view dataset comprises 11M frames from 8k sequences captured by synchronizing 8 smartphones.

GPAvatar: Generalizable and Precise Head Avatar from Image(s) [Homepage] [Code] Xuangeng Chu, Yu Li, Ailing Zeng, Tianyu Yang, Lijian Lin, Yunfei Liu, Tatsuya Harada The Twelfth International Conference on Learning Representations (ICLR), 2024 A framework for 3D head avatar reconstruction from one or several images in a single forward pass.

Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset [Homepage] [Code] Jing Lin∗, Ailing Zeng∗#, Shunlin Lu∗, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, Lei Zhang Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023 A large-scale 3D expressive whole-body human motion dataset (over 10M frames) with SMPL-X, text, audio, and RGB modalities.

SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation [Homepage] [Code] Zhongang Cai*, Wanqi Yin*, Ailing Zeng, Chen Wei, Qingping Sun, Yanjun Wang, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, Chen Change Loy, Lei Yang, Ziwei Liu Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023 Rank Top-1 on 7 SMPL/SMPL-X benchmarks! SMPL-X estimation with a systematic investigation on 32 datasets and model scaling-up.

DreamWaltz: Make a Scene with Complex 3D Animatable Avatars [Homepage] [Code] Yukun Huang*, Jianan Wang*, Ailing Zeng, He Cao, Xianbiao Qi, Yukai Shi, Zheng-Jun Zha, Lei Zhang Conference on Neural Information Processing Systems (NeurIPS), 2023 A text-driven 3D animatable avatar creation framework with complex scenes.

Effective Whole-body Pose Estimation with Two-stages Distillation [Code] Zhendong Yang, Ailing Zeng, Chun Yuan, Yu Li The Thirty-Fourth IEEE/CVF Conference on International Conference on Computer Vision (ICCV), CV4Metaverse Workshop, 2023 Rank Top 1 on COCO-WholeBody Benchmark. A better alternative to OpenPose.

HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation [Homepage] [Code] Xuan Ju*, Ailing Zeng*, Chenchen Zhao*, Jianan Wang, Lei Zhang, Qiang Xu (Oral) The Thirty-Fourth IEEE/CVF Conference on International Conference on Computer Vision (ICCV), 2023 HumanSD highlights multi-scenario human-centric image generation with precise pose control.

Neural Interactive Keypoint Detection [Code] Jie Yang, Ailing Zeng#, Feng Li, Shilong Liu, Ruimao Zhang#, Lei Zhang The Thirty-Fourth IEEE/CVF Conference on International Conference on Computer Vision (ICCV), 2023 The first end-to-end neural interactive keypoint detection/annotation framework significantly reduces 10+ times labeling costs.

One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer [Homepage] [Code] Rank Top-1 on AGORA SMPL-X leaderboard! (2023.04) [link] Jing Lin, Ailing Zeng#, Haoqian Wang, Lei Zhang, Yu Li The Thirty-Fourth IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023 The first one-stage framework for 3D whole-body mesh recovery and an large-scale upper-body dataset.

Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenes [Homepage] [论文解读] [Code] Xuan Ju, Ailing Zeng#, Jianan Wang, Qiang Xu, Lei Zhang The Thirty-Fourth IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023 Human-Art contains 50k high-quality images with over 123k person instances from 5 natural and 15 artificial scenarios, which are annotated with bounding boxes, keypoints, self-contact points, and text information for humans represented in both 2D and 3D.

Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation [论文解读] [Code] Jie Yang, Ailing Zeng#, Shilong Liu, Feng Li, Ruimao Zhang#, Lei Zhang Eleventh International Conference on Learning Representations (ICLR), 2023 We present a novel end-to-end framework with Explicit box Detection for multi-person Pose estimation.

Are Transformers Effective for Time Series Forecasting? [Code] [bilibili] Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu (Oral) the most Influential Paper in AAAI 2023, Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI), 2023 We try to question the validity of Transformer-based time series forecasting solutions via a linear layer.

DeciWatch: A Simple Baseline for 10× Efficient 2D and 3D Pose Estimation [Website] [Code] [Data] Ailing Zeng, Xuan Ju, Lei Yang, Ruiyuan Gao, Xizhou Zhu, Bo Dai, Qiang Xu European Conference on Computer Vision (ECCV), 2022 DeciWatch achieves 10+ efficiency improvement over existing works without any performance degradation for video-based 2D/3D human pose estimation.

SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos [Website] [Code] [知乎] Ailing Zeng, Lei Yang, Xuan Ju, Jiefeng Li, Jianyi Wang, Qiang Xu European Conference on Computer Vision (ECCV), 2022 SmoothNet is a plug-and-play refinement network to improve temporal smoothness and per-frame precision of any existing pose estimators.

HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling [Website] [Data] Zhongang Cai*, Daxuan Ren*, Ailing Zeng*, Zhengyu Lin*, Tao Yu*, Wenjia Wang*, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang^, Ziwei Liu^ (Oral) European Conference on Computer Vision (ECCV), 2022 HuMMan is a large-scale multi-modal 4D human dataset with 1000 human subjects, 400k sequences and 60M frames.

SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach [Code] Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, Stephen Lin 2020 European Conference on Computer Vision (ECCV’20) We design a split-and-recombine approach to improve generalization performance in 3D human pose estimation, especially on rare and unseen poses.

Mentoring

I will provide research and engineering guidance as much as possible based on the situation of each intern. Long-term goal-oriented research is welcome!

• Kenkun Liu, CUHK(SZ)

• Yinhuai Wang, PKU(SZ)

• Shunlin Lu, CUHK(SZ),(2023.01-now), two first-author works along human motions.

• Jing Lin, THU(SZ)->MPI,(2022.09-2023.11), two first-author and two co-author works along 3D digital human.

• Xuan Ju, CUHK->Tencent,(2022.06-2023.10), three first-author works along generative models.

• Jie Yang, CUHK(SZ)->BIGAI,(2022.05-2023.11), four first-author works along end-to-end keypoint detection.

Working Experience

• Researcher at International Digital Economy Academy. (Apr. 2022 to Apr. 2024)
  Worked on human-centric computer vision.

• Research intern at Microsoft Asia Research. (Dec. 2019 to Sep. 2020)
  Worked on improving the generalization ability of single-view 3D human pose estimation from either images or videos.

• In recent years, I have been a AC/reviewer of the following conferences and journals: CVPR, ICCV, ECCV, ICLR, ICML, NeurIPS, AAAI, TPAMI, IJCV, etc.

Honors & Awards

• Shenzhen Artificial Intelligence Natural Science Award, 2023

• Shenzhen Pengcheng special talent award, 2023

• Full Postgraduate Studentship, CUHK (2017 - 2021)

• Excellent League Member, Top 1% student of Xiamen University

• National Scholarship (2015)