Webcam 1: MoveNet AI Powered 3D Human Pose Tracking and Analysis

This notebook is the webcam 1 client-side of the AI-Powered 3D Human Pose Tracking and Analysis Project.

Goal:

  • Detect 2D pose using MoveNet
  • Send the skeleton key points to the server through Socket.IO
  • Save the data in a CSV file

How to run this project

  1. Run Server
  2. Open Receiver
  3. Run Webcam 1 and this Webcam 2

2D Pose Detection with MoveNet TensorFlow.js

MoveNet Model is the latest pre-trained machine learning library released by TensorFlow team as part of a larger Pose Detection TensorFlow.js models set. MoveNet is a convolutional neural network model that runs on RGB images and predicts human joint locations of a single person that runs faster than real-time ( 30+ FPS).

The pose Animator meshes SVG, face landmarks detection, and motion capture with pose estimation to allow real-time human pose estimation in the browser.

We encouraged you to check our 2D Webcam Pose Detection With PoseNet to compare the robustness of the two pose detection models. Compared to TensorFlow's older model - PoseNet, the MoveNet model has the combination of speed and accuracy needed that allows a more accurate depiction of difficult poses and motions.

The new Pose Detection API supports multiple models for running real-time pose estimation by simply estimating where key body joints are. Although impressive, the pose is still in 2D.

  • [Compare with PoseNet](/content/@mt-cs/posenet-webcam-1 "2D Webcam Pose Detection With PoseNet"/index.html)

Tracking two angles to get 3D pose tracking

By positioning two webcams, pointed at orthogonal directions, we can combine the two 2D poses into a single 3D pose stream.

Traditional 3D motion capture systems require a sophisticated setup and are very costly. This system makes 3D pose capture accessible to much more people.

Image: Camera Setup

Multi-Webcam 3D Pose tracking application

Whether it's for games, motion analysis in health and fitness, virtual reality development, or medical examinations, many applications require that people and their movements be captured digitally in 3D in real-time. Until now, this was possible only with expensive systems using multiple cameras and/or by having people wear special suits and tracking points or tracking devices. This project offers the possibility to turn smartphones and webcams into multi-view body tracking systems without using any trackers.

The long-term goal of this project is to interact in the VR world without controllers or wearables. By detecting 3D gestures in non-costly ways, we hope to reduce the need for body trackers, hardware usage, and also computational power and make 3D interaction in VR more accessible. Full-body tracking for skeletal animation allows for more user expression that could humanize users in the VR world. There are endless use cases including hand motion interaction with data, video games, and virtual meetings in 3D.

Let's Run Webcam 1

Turn on live feed from your webcam:

Webcam off

Toggle on and off to reset webcam if you get the video element has not loaded data:

Reset webcam

source = undefined

Movenet will run webcam feed through a machine learning model and output an object stored in the first element of the array below:

Pose Detection Controls

#feffad

Point/Line Color

#5bbeaa

Text Color

Skeleton Line Width

Show Face Points

Show Point Labels

Socket.io

Socket.IO is a library that enables real-time, bidirectional, and event-based communication between the browser and the server. It is divided into two parts:

  • Server-Side: it is a Node.js server
  • Client-Side: it is a Javascript client library for the browser (which can be also run from Node.js), In this project the client is both webcam 1 and webcam 2 ObservableHQ notebook.

While data can be sent in multiple forms, JSON is the simplest.

Enter your server URL:

Configure the socket server URL

socket = Socket {io: Manager, nsp: "/", json: Socket, ids: 0, acks: Object, receiveBuffer: Array(0), sendBuffer: Array(0), connected: false, disconnected: true, flags: Object, subs: Array(3), _callbacks: Object}

io = ƒ(uri, opts)

sendKeypoints = ƒ(keypoints, minConfidence, ctx, scale)

Saving Data

We are saving the skeleton data points in an array for the purpose of project demo and further data analysis.

Run to make sure the array contains skeleton data:

Array(0) []

Then run cell below to download the CSV file:

RuntimeError: unable to load module

Utility functions below:

markPoses = ƒ(canvas, poses, showLabels, showFacePoints, pointColor, textColor)

xyWebcam1 = Array(0) []

Libraries

model = "MoveNet"

poseDetection = Object {SupportedModels: Object, calculators: Object, createDetector: ƒ(model, modelConfig), movenet: Object, util: Object}

import { tf, requireTF, drawPose, drawLine, drawPoint, facePoints, sparkbar} from "@mt-cs/movenet-posenet-image-pose-tracking"

import { file, slider, text, select, color} from "@jashkenas/inputs"

import { toggle} from "@kelleyvanevert/little-things"

import { serialize} from "@palewire/saving-csv"

Credits

Developed for the Kineviz and University of San Francisco Summer Project by:

Sponsor: Weidong Yang

Code credits:

Acknowledgments

We sincerely thank:

  • Weidong Yang and the Kineviz Team for our biweekly meetings
  • Nikko Sacramento for plotting the 3D skeleton in GraphXR and running the data in VR
  • Sean Li for debugging the cors issue
  • Thom Lauret for the more robust AWS server
  • Professor Mario Lim for helpful crossover network connection discussions
  • Alex Law for posting the blogpost and voice over
  • Cynthia Yu for being our model

Contact

The notebook was implemented by Barbora Novakova and Marisa Tania. Feel free to reach out!