Chih-Hsiang, Hsu (許智翔)

---

Contents

• 1 Research
  • 1.1 Introduction
  • 1.2 Paper survey
• 2 Meeting slides
• 3 Contact

Research

---

Introduction

We aims to do sports analysis by reconstructing 3D human pose from video source. This task can be split into two steps:

• 2D keypoints estimation: estimating the 2D keypoints on human body, e.g. DEKR.
• 2D-to-3D lifting: reconstruct 3D human pose from a sequence of 2D keypoints, e.g. Poseformer

Paper survey

• Human pose prior
  • ICCV 2021 - Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flows
    • Paper: https://arxiv.org/abs/2107.13788
    • Code: GitHub
  • CVPR 2023 - GFPose Learning 3D Human Pose Prior with Gradient Fields
    • Paper: https://arxiv.org/abs/2212.08641
    • Code: GitHub
    • Notes: Using ground truth depth in normalization
• Absolute depth
  • 3DV 2019 - Multi-Person 3D Human Pose Estimation from Monocular Images
    • Paper: https://arxiv.org/abs/1909.10854
    • Notes: Estimating multi-person depth with focal length and translation optimization
  • ICCV 2019 - Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image
    • Paper: https://arxiv.org/abs/1907.11346
    • Code: GitHub
    • Notes: Using bounding box in image and the 3d posture knowledge to estimate absolute depth
  • ICCV 2019 - Optimizing Network Structure for 3D Human Pose Estimation
    • Paper: https://ieeexplore.ieee.org/document/9010332
    • Code: GitHub
    • Notes: Using ground truth depth in normalization
  • ArXiv 2019 - PoseLifter: Absolute 3D human pose lifting network from a single noisy 2D human pose
    • Paper: https://arxiv.org/abs/1910.12029
    • Code: GitHub
    • Notes: Predicting canonical root depth
  • ECCV 2020 - SMAP: Single-Shot Multi-Person Absolute 3D Pose Estimation
    • Paper: https://zju3dv.github.io/SMAP/
    • Code: GitHub
    • Notes: Normalizing depth
  • CVPR 2022 - Ray3D: ray-based 3D human pose estimation for monocular absolute 3D localization
    • Paper: https://openaccess.thecvf.com/content/CVPR2022/papers/Zhan_Ray3D_Ray-Based_3D_Human_Pose_Estimation_for_Monocular_Absolute_3D_CVPR_2022_paper.pdf
    • Code: GitHub
    • Notes: Embedding 2d poses in projective space
• Occlusion handling
  • ICCV 2019 - Occlusion-Aware Networks for 3D Human Pose Estimation in Video
    • Paper: https://openaccess.thecvf.com/content_ICCV_2019/papers/Cheng_Occlusion-Aware_Networks_for_3D_Human_Pose_Estimation_in_Video_ICCV_2019_paper.pdf
    • Notes: Detecting occlusion by cylinder-man model
  • AAAI 2020 - 3D Human Pose Estimation Using Spatio-Temporal Networks with Explicit Occlusion Training
    • Paper: https://arxiv.org/abs/2004.11822
    • Notes: Masking keypoints in training process
  • ECCV 2022 - Explicit Occlusion Reasoning for Multi-person 3D Human Pose Estimation
    • Paper: https://arxiv.org/abs/2208.00090
    • Code: GitHub
    • Notes: Detecting occlusion by Skeleton-guided human Shape Fitting
• 2d keypoints estimation
  • CVPR 2019 - Deep High-Resolution Representation Learning for Human Pose Estimation
    • Paper: https://arxiv.org/abs/1902.09212
    • Code: GitHub
  • CVPR 2021 - Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression
    • Paper: https://arxiv.org/abs/2104.02300
    • Code: GitHub
    • Notes: best solution for 2d human pose estimation
• Multi-hypotheses pose estimation
  • ICCV 2017 - Generating Multiple Diverse Hypotheses for Human 3D Pose Consistent with 2D Joint Detections
    • Paper: https://arxiv.org/abs/1702.02258
  • CVPR 2019 - Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Network
    • Paper: https://arxiv.org/abs/1904.05547
    • Code: GitHub
  • CVPR 2022 - MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation
    • Paper: https://arxiv.org/abs/2111.12707
    • Code: GitHub
    • Notes: there are several unsolved problems in this paper, which are mentioned in meeting slide 2022/11/02
  • CVPR 2023 - Diffpose: Toward more reliable 3d pose estimation
    • Paper: https://arxiv.org/abs/2211.16940
    • Code: GitHub
  • ICCV 2023 - Diffusion-based 3d human pose estimation with multi-hypothesis aggregation
    • Paper: https://arxiv.org/abs/2303.11579
    • Code: GitHub
  • CVPR 2024 - Ktpformer: Kinematics and trajectory prior knowledge-enhanced transformer for 3d human pose estimation
    • Paper: https://arxiv.org/abs/2404.00658
    • Code: GitHub
  • CVPR 2024 - Finepose: Fine-grained prompt-driven 3d human pose estimation via diffusion models
    • Paper: https://arxiv.org/abs/2405.05216
    • Code: GitHub
• Geometry constraints on human pose estimation
  • CVPR 2019 - Self-Supervised Learning of 3D Human Pose using Multi-view Geometry
    • Paper: https://arxiv.org/abs/1903.02330
    • Code: GitHub
    • Notes: MPJPE (L2-norm loss) is said to be not sufficient
  • IEEE 2021 - Anatomy-aware 3D Human Pose Estimation with Bone-based Pose Decomposition
    • Paper: https://arxiv.org/abs/2002.10322
    • Code: GitHub
• Human pose estimation
  • CVPR 2019 - 3D human pose estimation in video with temporal convolutions and semi-supervised training
    • Paper: https://arxiv.org/abs/1811.11742
    • Code: GitHub
  • CVPR 2019 - Semantic Graph Convolutional Networks for 3D Human Pose Regression
    • Paper: https://arxiv.org/abs/1904.03345
    • Code: GitHub
  • 2020 - A Graph Attention Spatio-temporal Convolutional Network for 3D Human Pose Estimation in Video
    • Paper: https://arxiv.org/abs/2003.14179
    • Code: GitHub
  • ICCV 2021 - 3D Human Pose Estimation with Spatial and Temporal Transformers
    • Paper: https://arxiv.org/abs/2103.10455
    • Code: GitHub
  • IEEE 2022 - Limb Pose Aware Networks for Monocular 3D Pose Estimation
    • Paper: https://ieeexplore.ieee.org/document/9663053
    • Notes: Using additional information from raw images
  • CVPR 2022 - Mixste: Seq2seq mixed spatiotemporal encoder for 3d human pose estimation in video
    • Paper: https://arxiv.org/abs/2203.00859
    • Code: GitHub
  • CVPR 2023 - 3d human pose estimation with spatio-temporal criss-cross attention
    • Paper: https://openaccess.thecvf.com/content/CVPR2023/papers/Tang_3D_Human_Pose_Estimation_With_Spatio-Temporal_Criss-Cross_Attention_CVPR_2023_paper.pdf
    • Code: GitHub
  • CVPR 2023 - Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation
    • Paper: https://arxiv.org/abs/2303.17472
    • Code: GitHub
• Transformer on computer vision
  • ICLR 2021 - An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
    • Paper: https://arxiv.org/abs/2010.11929
    • Code: GitHub
• Object pose estimation in 6-dof
  • CVPR 2022 - EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation
    • Paper: https://arxiv.org/abs/2203.13254
    • Code: GitHub
• IMU related
  • CVPR 2019 - On the Continuity of Rotation Representations in Neural Networks
    • Paper: https://openaccess.thecvf.com/content_CVPR_2019/papers/Zhou_On_the_Continuity_of_Rotation_Representations_in_Neural_Networks_CVPR_2019_paper.pdf
    • Notes: representation of rotation
  • CVPR 2023 - Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors
    • Paper: https://arxiv.org/abs/2203.08528
    • Code: GitHub
  • TMLR 2023 - High fidelity neural audio compression
    • Paper: https://openreview.net/pdf?id=ivCd8z8zR2
    • Code: GitHub
    • Notes: imu data encoder

Contact

---

• Email: andy.hsu@nas.mirlab.org
• Lab: Multimedia Information Retrieval Lab