Ailing Zeng 曾爱玲

Ph.D. Department of Computer Science and Engineering The Chinese University of Hong Kong Researcher International Digital Economy Academy (IDEA) Google Scholar Resume IDEA-GenAI, IDEA-Research

Biography

I am a researcher of Computer Vision and Robotics at the International Digital Economy Academy (IDEA). Previously, I obtained my Ph.D. degree from the Department of Computer Science and Engineering, the Chinese University of Hong Kong, supervised by Prof. Qiang Xu. I was a visiting scholar in the Robotics Institute, Carnegie Mellon University.

My research targets to build multi-modal human-like intelligent agents on scalable big data, especially for Large Motion Models to capture, understand, interact, and generate the motion of humans, animals, and the world. Specifically,

1) Human-centric visual perception with large-scale data and general models: AiOS,SMPLer-X, OSX, DW-Pose, ED-Pose, SmoothNet, DeciWatch

2) Large-scale multi-modality datasets: Motion-X, UBody, Uni-KPT, BallPlay, HuMMan, Human-Art

3) Human-centric generative models: HumanSD, PhysHOI, Dreamwaltz, HumanTOMATO, DiffSHEG

4) Interactive AI & Human-in-the-loop techniques: Uni-Pose, Click-Pose, Grounded-SAM

5) Previously, time series analysis and forecasting: LTSF-Linear, SCINet, FITS

Now, I'm looking for self-motivated interns, especially for video generation. Please send me your resume by email.

News

• [2024.04] We are hosting the ECCV 2024 Tutorial on “Recent Advances in Video Content Understanding and Generation (VENUE).”

• [2024.02] Four papers are accepted to CVPR 2024!

• [2024.01] Three papers are accepted to ICLR 2024!

• [2023.12] We are hosting the CVPR 2024 Workshop on “Computer Vision with Humans in the Loop” (CVHL).

• [2023.09] LTSF-Linear was selected as the Top-3 Influential Paper in AAAI 2023!

• [2023] 13 papers are accepted to CVPR/ICLR/NeurIPS/ICCV/AAAI 2023!

• [2022] 4 papers are accepted to ECCV/NeurIPS 2022!

• [2022] Invited Talk at OpenMMlab about Human Motion Modeling from a Time Series Perspective. [Slides] [Video]

Publications

Open-World Human-Object Interaction Detection via Multi-modal Prompts Jie Yang* , Bingliang Li* , Ailing Zeng#, Lei Zhang, Ruimao Zhang# The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024 A Multi-modal Prompt-based HOI detector designed to leverage both textual descriptions and visual exemplars for open-world generalization.

AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation [Homepage] [Code] Qingping Sun*, Yanjun Wang*, Ailing Zeng, Wanqi Yin, Chen Wei, Wenjia Wang, Haiyi Mei, Chi Sing Leung, Ziwei Liu, Lei Yang, Zhongang Cai The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024 An end-to-end DETR-based model for multi-person whole-body mesh recovery.

PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code [Homepage] [Code] Xuan Ju, Ailing Zeng#, Yuxuan Bian, Shaoteng Liu, Qiang Xu# The Twelfth International Conference on Learning Representations (ICLR), 2024 A Plug-and-Play inversion for diffusion-based image editing and a prompt-based image editing benchmark (PIE-Bench).

FITS: Modeling Time Series with 10k parameters [Code] Zhijian Xu, Ailing Zeng, Qiang Xu (spotlight)The Twelfth International Conference on Learning Representations (ICLR), 2024 An extremely lightweight yet powerful model for time series analysis.

GPAvatar: Generalizable and Precise Head Avatar from Image(s) [Homepage] [Code] Xuangeng Chu, Yu Li, Ailing Zeng, Tianyu Yang, Lijian Lin, Yunfei Liu, Tatsuya Harada The Twelfth International Conference on Learning Representations (ICLR), 2024 A framework for 3D head avatar reconstruction from one or several images in a single forward pass.

UniPose: Detecting Any Keypoints [Code] Jie Yang, Ailing Zeng#, Ruimao Zhang#, Lei Zhang tech. report, 2023 UniPose supports textual and visual prompts to detect arbitrary keypoints (e.g., from articulated, rigid, to soft objects).

Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset [Homepage] [Code] Jing Lin∗, Ailing Zeng∗#, Shunlin Lu∗, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, Lei Zhang Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023 A large-scale 3D expressive whole-body human motion dataset (over 10M frames) with SMPL-X, text, audio, and RGB modalities.

SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation [Homepage] [Code] Zhongang Cai*, Wanqi Yin*, Ailing Zeng, Chen Wei, Qingping Sun, Yanjun Wang, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, Chen Change Loy, Lei Yang, Ziwei Liu Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023 Rank Top-1 on 7 SMPL/SMPL-X benchmarks! SMPL-X estimation with a systematic investigation on 32 datasets and model scaling-up.

DreamWaltz: Make a Scene with Complex 3D Animatable Avatars [Homepage] [Code] Yukun Huang*, Jianan Wang*, Ailing Zeng, He Cao, Xianbiao Qi, Yukai Shi, Zheng-Jun Zha, Lei Zhang Conference on Neural Information Processing Systems (NeurIPS), 2023 A text-driven 3D animatable avatar creation framework with complex scenes.

A Comprehensive Benchmark for Neural Human Radiance Fields Kenkun Liu, Derong Jin, Ailing Zeng#, Xiaoguang Han, Lei Zhang Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023 A thorough study of unified evaluation settings and metrics to establish a fair and reasonable benchmark for human NeRF models.

Effective Whole-body Pose Estimation with Two-stages Distillation [Code] Zhendong Yang, Ailing Zeng, Chun Yuan, Yu Li The Thirty-Fourth IEEE/CVF Conference on International Conference on Computer Vision (ICCV), CV4Metaverse Workshop, 2023 Rank Top 1 on COCO-WholeBody Benchmark. A better alternative to OpenPose.

HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation [Homepage] [Code] Xuan Ju*, Ailing Zeng*, Chenchen Zhao*, Jianan Wang, Lei Zhang, Qiang Xu (Oral) The Thirty-Fourth IEEE/CVF Conference on International Conference on Computer Vision (ICCV), 2023 HumanSD highlights multi-scenario human-centric image generation with precise pose control.

Neural Interactive Keypoint Detection [Code] Jie Yang, Ailing Zeng#, Feng Li, Shilong Liu, Ruimao Zhang#, Lei Zhang The Thirty-Fourth IEEE/CVF Conference on International Conference on Computer Vision (ICCV), 2023 The first end-to-end neural interactive keypoint detection/annotation framework significantly reduces 10+ times labeling costs.

From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels [Code] Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, Yu Li The Thirty-Fourth IEEE/CVF Conference on International Conference on Computer Vision (ICCV), 2023 This work unifies the formulations of the two tasks by decomposing and reorganizing the generic KD loss into a Normalized KD loss and customized soft labels.

One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer [Homepage] [Code] Rank Top-1 on AGORA SMPL-X leaderboard! (2023.04) [link] Jing Lin, Ailing Zeng#, Haoqian Wang, Lei Zhang, Yu Li The Thirty-Fourth IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023 The first one-stage framework for 3D whole-body mesh recovery and an large-scale upper-body dataset.

Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenes [Homepage] [论文解读] [Code] Xuan Ju, Ailing Zeng#, Jianan Wang, Qiang Xu, Lei Zhang The Thirty-Fourth IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023 Human-Art contains 50k high-quality images with over 123k person instances from 5 natural and 15 artificial scenarios, which are annotated with bounding boxes, keypoints, self-contact points, and text information for humans represented in both 2D and 3D.

Lite DETR: An Interleaved Multi-Scale Encoder for Efficient DETR [Code] Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lionel Ni, Lei Zhang The Thirty-Fourth IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023 A simple, effective, and general design for lightweight DETR models.

Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation [论文解读] [Code] Jie Yang, Ailing Zeng#, Shilong Liu, Feng Li, Ruimao Zhang#, Lei Zhang Eleventh International Conference on Learning Representations (ICLR), 2023 We present a novel end-to-end framework with Explicit box Detection for multi-person Pose estimation.

Are Transformers Effective for Time Series Forecasting? [Code] [bilibili] Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu (Oral) Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI), 2023 We try to question the validity of Transformer-based time series forecasting solutions via a linear layer.

DeciWatch: A Simple Baseline for 10× Efficient 2D and 3D Pose Estimation [Website] [Code] [Data] Ailing Zeng, Xuan Ju, Lei Yang, Ruiyuan Gao, Xizhou Zhu, Bo Dai, Qiang Xu European Conference on Computer Vision (ECCV), 2022 DeciWatch achieves 10+ efficiency improvement over existing works without any performance degradation for video-based 2D/3D human pose estimation.

SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos [Website] [Code] [知乎] Ailing Zeng, Lei Yang, Xuan Ju, Jiefeng Li, Jianyi Wang, Qiang Xu European Conference on Computer Vision (ECCV), 2022 SmoothNet is a plug-and-play refinement network to improve temporal smoothness and per-frame precision of any existing pose estimators.

HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling [Website] [Data] Zhongang Cai*, Daxuan Ren*, Ailing Zeng*, Zhengyu Lin*, Tao Yu*, Wenjia Wang*, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang^, Ziwei Liu^ (Oral) European Conference on Computer Vision (ECCV), 2022 HuMMan is a large-scale multi-modal 4D human dataset with 1000 human subjects, 400k sequences and 60M frames.

SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction [Code] Minhao Liu, Ailing Zeng, Qiuxia Lai, Qiang Xu Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022 We propose a novel and general CNN-based backbone for the time series forecasting problem, achieving SOTAs on several real-world datasets.

T-WaveNet: Tree-Structured Wavelet Neural Network for Sensor-Based Time Series Analysis [Code] Minhao Liu, Ailing Zeng, Qiuxia Lai, Qiang Xu 2022 Tenth International Conference on Learning Representations (ICLR'22) We introduce a tree-structured wavelet deep neural network to effectively extract more discriminative and expressive feature representations in time series signals.

Learning Skeletal Graph Neural Networks for Hard 3D Pose Estimation Ailing Zeng, Xiao Sun, Lei Yang, Nanxuan Zhao, Minhao Liu, Qiang Xu 2021 IEEE/CVF International Conference on Computer Vision (ICCV’21) We explore a novel skeletal graph neural network to deal with hard poses in 3D pose estimation.

Human Pose Regression with Residual Log-likelihood Estimation [Code] [知乎] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu and Cewu Lu (Oral) 2021 IEEE/CVF International Conference on Computer Vision (ICCV’21) We propose a novel regression paradigm, which is superior to the heatmap-based methods for the first time, especially on multi-person pose estimation.

SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach [Code] Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, Stephen Lin 2020 European Conference on Computer Vision (ECCV’20) We design a split-and-recombine approach to improve generalization performance in 3D human pose estimation, especially on rare and unseen poses.

Mentoring

I will provide research and engineering guidance as much as possible based on the situation of each intern. Long-term goal-oriented research is welcome!

• Kenkun Liu, CUHK(SZ)

• Yinhuai Wang, PKU(SZ)

• Linghao Chen, THU(SZ)

• Shunlin Lu, CUHK(SZ),(2023.01-now), two first-author works along human motions.

• Jing Lin, THU(SZ)->MPI,(2022.09-2023.11), two first-author and two co-author works along 3D digital human.

• Xuan Ju, CUHK->Tencent,(2022.06-2023.10), three first-author works along generative models.

• Jie Yang, CUHK(SZ)->BIGAI,(2022.05-2023.11), four first-author works along end-to-end keypoint detection.

Working Experience

• Researcher at International Digital Economy Academy. (Apr. 2022 to now)
  Worked on human-centric computer vision.

• Research intern at Sensetime Research. (Nov. 2020 to Apr. 2022)
  Worked on robust and efficient 2D/3D human pose estimation and reconstruction from videos.

• Research intern at Microsoft Asia Research. (Dec. 2019 to Sep. 2020)
  Worked on improving the generalization ability of single-view 3D human pose estimation from either images or videos.

• Research intern at Baidu Research. (June 2018 to Sep. 2018)
  Worked on 2D human pose estimation.

• TMT investment intern at ASB Ventures. (March 2017 to Aug. 2017)
  Analyzed technological feasibility of start-up companies.

• In recent years, I have been a reviewer of the following conferences and journals: CVPR, ICCV, ECCV, ICLR, ICML, NeurIPS, AAAI, TPAMI, IJCV, etc.

Honors & Awards

• Shenzhen Artificial Intelligence Natural Science Award, 2023

• Shenzhen Pengcheng special talent award, 2023

• Full Postgraduate Studentship, CUHK (2017 - 2021)

• Excellent League Member, Top 1% student of Xiamen University

• National Scholarship (2015)