AI - Fitness Game Using Body Pose Detection (Difficulty: 3)

CreatiCode

Key Topics Covered

Introduction

In this tutorial, you will build a fitness game using body pose detection. Different target poses will be shown on the stage, and the players need to match that pose as fast as they can.

Step 1 - A New Project

Please create a new project in the CreatiCode playground. Remove the sprite named “Sprite1”, and rename the “Empty1” sprite to “Main”.

Step 2 - Run Body Part Detection

When the green flag is clicked, we should run the body part detection block right away. It will turn on the camera and analyze the camera images in real-time. You should set “debugging” to “yes”, so it will automatically highlight the key body parts.

A table named “i” should be automatically created in the “Variables” category. Please check the checkbox for the table “i”,

then run the program. You should see the key body parts highlighted, and the content of the table “i” will be updated in real time:

Step 3 - Add the Squat Costume

Next, uncheck the table “i”, and add a costume for a person in the “squat” pose. You can pick the costume named “Dorian-d”, and rename it “Squat”.

Please set the sprite size to 130, and move it to the bottom right of the stage. This will be the first target pose for the player.

Step 4 - Analyze the Squat Pose

Although we know the position of each body part, we still don’t know the pose of the player. To automatically determine whether the player is in the squat pose, we can focus on these 4 points: left/right hip and left/right knee.

The key observation is that when the player is in the squat pose, the distance between the 2 knee points should be much larger than the distance between the 2 hip points.

To keep our logic simple, we will assume the player’s body is not tilted left or right, so we can calculate these distances using the x positions of the 4 body parts only:

Knee Distance = Right Knee X - Left Knee X
Hip Distance = Right Hip X - Left Hip X

Step 5 - Read from the Table

Now let’s create 4 new variables to represent the X positions of these 4 body parts: “left hip x”, “left knee x”, “right hip x”, “right knee x”.

We can read their values from the table “i”. Check the checkbox for the table “i” again to examine its content. For example, for the x position of the left hip, we can use the “x” column, and we find its row number is 12:

Therefore, we can read this value into the “left hip x” variable like this:

Here are the blocks to read the values of all 4 variables:

Step 6 - Calculate the Distances

Now we have the x positions of these 4 body points, we can calculate the distances between the 2 knees and the 2 hips.

Please make 2 new variables “knee distance” and “hip distance” to represent these 2 distances.

The “hip distance” is the difference between the x positions of the right hip and the left hip:

Similarly, the knee distance is the difference between the x positions of the right knee and the left knee. So here is the complete program after this step:

Step 7 - Calculate the Pose based on the Distances

Now our program knows the 2 distance values, we can compare them to determine if the player is in a squat pose. We can use a simple rule like this: if the knee distance is more than double the hip distance, we call it a good squat pose.

Let’s make a new variable “pose”. We will set its value to “Unknown” by default, but if the knee distance is indeed more than double the hip distance, we set “pose” to “Squat” instead.

Step 8 - Reorganize the Blocks

Since all the new blocks we have added serve one purpose of calculating the pose of the player, we should make a new block using them. This new block will be called “calculate pose”, and we’ll move all the new blocks to its definition:

Step 9 - Guard Against Invalid Values

When a body part is not detected from the camera image, the corresponding item in the table will be set to -10000. If any of the 4 variables for hip/knee x positions is -10000, then we should not use that data.

Therefore, we should add some additional logic to ignore invalid data points, such as requiring all 4 variables to be greater than -250. Note that if the body part is visible, its x position should never be less than -240.

Step 10 - Calculate the Pose Repeatedly

After the player moves and changes pose, the numbers in the table “i” will be updated continuously. Therefore, we should repeatedly calculate the player’s pose. This can be done using a forever loop:

Step 11 - Check for “Squat” Pose

Now we can repeatedly check the value of the “pose” variable. As soon as its value changes to “squat”, we can show a success message.

Here is what you would get if you try to run the program:

Step 12 - Set a Target Pose

Now we have the basic game mechanics worked out. The next step is to make it more fun. We will give the player more than one target pose, so the player has to try to complete these poses to earn points.

As a first step, we should create a new variable “target pose”, which will represent the next target pose the player needs to make.

We can rewrite our code using this new variable:

Step 13 - Toggle the Target Pose between “Squat” and “Stand”

When the user has completed the target pose, we should change the target to a new pose. To keep it simple, let’s make the target toggle between 2 poses: “Squat” and “Stand”.

We can define a new block “change target pose” for this purpose. If the value of “target pose” is “Squat”, then the next target should be “Stand”. Otherwise, if the value of “target pose” is not “Squat”, it must be “Stand”, so we just need to switch it back to “Squat”.

Step 14 - Make the “Stand” Costume

To show that the target pose has changed, we should change the costume of the sprite as well.

Please find the “Dorian-c” costume and add it to your sprite. Rename it as “Stand”.

We still need to make some small adjustments to the costume. The leg on the left is reaching out sideways. We can select the whole leg and rotate it to an upright position:

In addition, the head is looking sideways, which may confuse the player. We can simply remove this head, and copy the head from the “Squat” costume:

Step 15 - Switching the Costume

When we change the “target pose”, we should change the costume at the same time, so that the player sees what’s the next pose he or she needs to make:

Step 16 - Visual/Sound Effects

To give the player clear feedback that the target pose has been completed, we should show some visual effects. For example, we can make the sprite brighter for a short time before changing to the next costume. Also, we can play a sound for collecting a coin as well:

Step 17 - Detect if the player is in “Stand” pose

In the “calculate pose” block’s definition, currently we only check for the “Squat” pose. We also need to check if the player is in “Stand” pose. To keep it simple, we can still look at the distance between the knees and the distance between the hips. If the player is standing up, the knee distance should just be slightly more than the hip distance. For example, we can require that the knee distance is less than 1.2 times the hip distance:

Now our game is ready. Whenever the player hits a target pose, we switch to the next target:

Creative Ideas

Here are some fun ways for you to build upon this project:

Game Rules: You can design different rules for playing the game. For example, you can give the player a time limit (e.g. 1 minute), and whoever earns more coins wins. You can also require the player to complete 60 target poses, and whoever finishes in the shortest time wins.
Refine the Pose Detection: Currently we only look at knees and hips. You can also add additional checks on the arms. For example, you can require the arms to be held horizontally. You can compare the Y position of the wrists with the shoulders.
More Target Poses: You can try to add more fun poses. For example, make the player do an “X” pose like below. You will need to use the X and Y positions of the wrists and ankles to detect such a pose.

2-Player Mode: When 2 players are standing in front of the camera, both can be detected by the body pose detection engine. This allows you to create 2-player games. For example, whenever player A makes a pose, we set that to be the target for player B. This way, player B has to copy player A’s poses.