Project documentation

Abstract

The AI Fitness Assistant is a browser-based computer vision system that tracks resistance-training exercises in real time from an ordinary webcam or phone camera. Google’s MediaPipe Pose model locates 33 body landmarks in every frame; the application converts them into joint angles and body positions, smooths the signal, and runs a per-exercise state machine that only counts a repetition once its full range of motion is complete.

The Exercise Lab ships 31 trackers: 29 repetition counters and 2 timed holds. Form checks turn the on-screen skeleton red and explain the fault in plain words. All inference runs on the user’s device, so no video or body data ever leaves the browser. In automated testing every tracker passes all 100 scripted scenarios on both 16:9 and 4:3 cameras, including 33 movements that must not count as a rep.

Problem statement & motivation

Resistance exercise provides proven cardiovascular, metabolic and musculoskeletal health benefits. However, inexperienced lifters face substantial risks of joint strain and acute injury when they train with poor posture or a compromised range of motion, and most train without a coach watching.

Wearable inertial measurement units (IMUs) and motion-capture suits can measure movement precisely, but they need specialised, expensive hardware. This project investigates whether a consumer webcam and edge machine learning can deliver useful biomechanical feedback directly in a web browser.

The aim is not to replace a coach. It is to answer, immediately and objectively, the two questions every lifter asks: did that rep count? and was my position safe?

Objectives & requirements

System architecture

Every tracker runs the same six-step pipeline on each camera frame. Only step 5, the exercise’s own rules, differs between trackers.

1. 1CameraFrames from getUserMedia
2. 2Pose modelMediaPipe Pose finds 33 landmarks
3. 3GeometryJoint angles & body ratios
4. 4Smoothing~160 ms moving average
5. 5State machineThe exercise’s rep rules
6. 6FeedbackSkeleton, meter, cues, count

Components

Lifecycle of a set

1. Start opens the camera and, the first time only, creates the pose model. Later Starts reuse the same model and call reset(): MediaPipe’s WebAssembly runtime cannot be rebuilt on a page, so closing and re-creating it would leave the tracker silent.
2. Each frame is sent to the model. Its results arrive in onResults, which measures the body, advances the state machine and redraws the skeleton.
3. Stop ends the set: the camera turns off and the rep count, hold timer, smoothing history and form state all reset to zero. A frame still being processed when Stop is pressed is ignored, so it cannot leave stale state for the next set.

Pose landmarks

MediaPipe Pose, based on the BlazePose network [1], returns 33 landmarks per frame. Each has an x and y position as a fraction of the frame (0–1), a relative depth z, and a visibility score. The trackers run the full model (complexity 1) with landmark smoothing on and detection and tracking confidence set to 0.7.

A landmark is only trusted when its visibility is at least 0.5, and an angle is only measured when all three of its points pass. A limb hidden from the camera therefore drops out instead of producing a wild reading.

Measuring joint angles

A joint angle uses three landmarks: for the elbow, the shoulder (A), elbow (B) and wrist (C). The two-argument arctangent gives the direction of each limb segment, and their difference is the angle at B.

Landmark x is a fraction of the frame’s width and y of its height. On a 16:9 webcam one unit of x is 1.78 times longer than one unit of y, so using the raw numbers skews every angle that isn’t perfectly horizontal or vertical. The trackers rescale x by the video’s aspect ratio first:

Measured effect

The same two photographs were padded into three frame shapes and run through the real pose model. A good measurement should not depend on the frame shape.

For the squat the spread across frame shapes falls from 14.5° uncorrected to 8.3° corrected. MediaPipe’s 3D world landmarks [3] were the most stable (3.5°) but read a straight leg as about 168°, too close to the 160–165° lockout thresholds. The trackers therefore use aspect-corrected 2D angles, which keep the thresholds’ meaning.

Distance-independent measures

Some movements are judged by position rather than angle. Each is scaled by the body itself, so the result doesn’t change with distance from the camera:

• Lat Pulldown: wrist height above or below the shoulders, measured in shoulder widths.
• Calf Raises: heel height above the toes as a fraction of shin length, compared with a resting baseline captured at Start.
• Pull-ups: head height compared with the hands, which mark the bar.
• Plank, Burpees, torso-lean checks: torso tilt measured in true pixel proportions.

Smoothing the signal

Landmarks jitter slightly from frame to frame. Each reading is averaged over the last ~160 ms (at most five frames). A fixed five-frame average, used previously, lags more as the frame rate drops. On a slower phone it can smooth away the bottom of a fast rep before the angle crosses its threshold.

The table shows a simulated squat rep (knee 170° → 85° → 170°) and the lowest smoothed angle each method reaches. The rep counts only if that value drops below the 100° threshold.

At 30 fps the two methods are identical; the time window only changes behaviour on slower devices.

Counting reps

Each tracker is a finite state machine with two thresholds and a gap between them. For a squat, the knee angle dropping below 100° arms the rep (DOWN), and standing back up (knees above 160° and hips above 150°) while armed counts it (UP). The gap stops jitter around a single line from double-counting, and the required order means half-reps and bounces never count.

Angles alone can’t tell exercises apart: a standing curl bends the elbow exactly like a push-up. So every tracker first checks that the body is in the exercise’s position (lying back, hinged forward, hanging from the bar, in a split stance) and only then lets the angles move the state machine. Outside that position the panel shows an amber cue such as “GET INTO A PUSH-UP POSITION”, and nothing counts.

How trackers read the body

Every completed rep dispatches a repCompleted browser event with the current count; routine mode listens for it to advance to the next exercise.

Exercise reference

The exact rules each tracker applies. Angles are in degrees; “armed” is the position that must be reached before a rep can count.

Feedback & interface

Controls: Start, Stop, Flip Camera (front or rear, with mirroring handled) and Hide/Show Skeleton.

Privacy & performance

Evaluation

1. Synthetic skeletons. An automated browser harness replaces the pose model with skeletons of exactly known joint angles and runs each tracker’s own page code. There are 100 scenarios covering all 31 trackers:
   • 16:9 and 4:3 cameras;
   • a person close to the camera and twice as far;
   • fast reps at about 9 fps;
   • 33 movements that must not count, such as half squats, squats on the lunge and step-up pages, push-ups or rows done standing, curls on the dips, press, pull-up, tricep and single-arm row pages, a press out to a T, a deadlift without knee lockout (or on the kettlebell page), standing toe touches on the sit-up page, a burpee without the jump, half jumping jacks, and sliding the legs out on the glute bridge;
   • form faults with the exact message expected (rounded back, sagging or piked hips, bent arms or legs, drifting elbows, leaning off the wall), and form that must not be flagged (looking ahead during a row, a 25° lean on the lat pulldown);
   • timed holds, and Stop/Start resets.
2. Real model. The real MediaPipe Pose package ran against a fake webcam feed of photographs. Every tracker went through three Start/Stop sessions.
3. Measurement study. The angle and smoothing comparisons in sections 06 and 07.

The 13 scenarios the previous engine failed exposed real bugs. On a 16:9 camera Deadlift, Romanian Deadlift and Wall Sit never registered. Pull-ups could not count from a dead hang, and Shoulder Press counted on the way down. Curls added reps to Shoulder Press and Overhead Tricep Extension, and Lat Pulldown depended on distance from the camera. Fast reps were missed at low frame rates.

A later technique audit found that many trackers checked the right joint but not the right exercise. Run against the previous pages, the 73-scenario technique suite passed 36. Push-ups, bench press, rows and kickbacks counted while standing. Lunges counted squats, and Bulgarian Split Squats never counted because the rear knee stays bent. Dips counted curls, and the press counted arms straightened out to a T. Deadlifts and squats counted without a full lockout, and sit-ups counted standing toe touches. Every tracker now checks the position first and the lockout fully, and each of these cases is a test.

These tests prove each tracker does what its rules say. They do not prove the thresholds suit every body: no study with real athletes has been run yet (see section 13).

Limitations & future work

Known limitations

• Monocular depth ambiguity: a single 2D camera loses accuracy when a limb moves towards or away from it, which is why most trackers ask for a side-on view.
• Self-occlusion: overlapping limbs lower the model’s confidence, and a hidden joint drops out of the measurement.
• Lighting and background: dim light or a busy background adds landmark jitter.
• Heuristic thresholds: the angles come from common coaching cues, not from a measured population, and may need tuning for mobility or body proportions.
• One person in frame: other people in view can confuse the model.
• No spine landmarks: the pose model has no points along the back, so back rounding is read from where the ear sits against the hip-to-shoulder line. A rounded lower back with the head held up can pass.
• Leaning towards the camera: on the trackers you face, a lean forwards or backwards changes little in 2D, so lean checks there catch sideways leaning best.

Future work

• A validation study with recorded, hand-labelled reps from real athletes.
• Optional 3D world-landmark angles, with thresholds re-tuned on that dataset.
• Per-user calibration of range of motion.
• Temporal Convolutional Networks (TCN) for continuous phase prediction, and dual-camera fusion.
• Per-set statistics such as tempo and depth, and adaptive filters such as the 1€ filter [5].

Student project contributors

References

1. Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., & Grundmann, M. (2020). BlazePose: On-device Real-time Body Pose tracking. arXiv preprint arXiv:2006.10204.
2. Lugaresi, C., et al. (2019). MediaPipe: A Framework for Building Perception Pipelines. Third Workshop on Computer Vision for AR/VR at IEEE CVPR.
3. Grishchenko, I., Bazarevsky, V., Zanfir, A., et al. (2022). BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation. arXiv preprint arXiv:2206.11678.
4. Baechle, T. R., & Earle, R. W. (2008). Essentials of Strength Training and Conditioning. National Strength and Conditioning Association (NSCA), Human Kinetics.
5. Casiez, G., Roussel, N., & Vogel, D. (2012). 1€ Filter: A Simple Speed-based Low-pass Filter for Noisy Input in Interactive Systems. Proceedings of CHI 2012, ACM.
6. W3C. Media Capture and Streams (the getUserMedia API). https://www.w3.org/TR/mediacapture-streams/