Navigation

    CreatiCode Scratch Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • CreatiCode

    AI - Fitness Game Using Body Pose Detection (Difficulty: 3)

    Tutorials
    1
    1
    713
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • info-creaticode
      CreatiCode last edited by admin

       

      Key Topics Covered

       

      • Using variables
      • Reading from tables
      • Graphic Effects
      • Body Part Detection AI

       
       

      Introduction

       

      In this tutorial, you’ll create a fitness game using AI-powered body pose detection. The game will display a target pose on the screen, and the player must quickly match that pose using their body.

      6375abe6-43a0-42e8-b412-921e31f5af5a.gif

       
       

      Step 1 - Start a New Project

       

      Create a new project in the CreatiCode Playground.
      Delete the sprite named “Sprite1” and rename the “Empty1” sprite to “Main”.

       
       

      Step 2 - Enable Body Part Detection

       

      Add the body part detection block. Set debugging to “yes” to highlight the key body points on the video feed.

      29ec46b8-175b-4b5f-90be-ef68fd78ad3b-image.png

       
      A new table named “i” will automatically appear under the Variables category. Check its box to display it on stage.

      0e41c5fa-9538-4ee5-a285-7687d4b08472.gif

       
      Once you run the program, you’ll see the body part markers on screen, and the table data will update in real time:

      c2890b2e-b478-49f0-84e0-0727fbe2e099.gif

       
      Note: To capture your entire body in the camera frame, you’ll need to stand farther away—but this makes it difficult to read the table data on the screen. A good solution is to work with a classmate: one person performs the poses in front of the camera, while the other monitors how the data in the table changes.

       
       

      Step 3 - Add the Squat Costume

       

      Now, uncheck the “i” table and add a costume representing a squat pose. Use the costume named “Dorian-d”, and rename it “Squat”.

      c54d7698-fb24-4b6f-b6ed-e30cf55bf876.gif

       
      Set its size to 130 and position it at the bottom right of the stage. This will serve as the first target pose.

      6f23c3cd-a65e-4cbb-9d74-2fff4fcfef84.png

       
       

      Step 4 - Understand the Squat Pose

       

      We know where each body part is, but we don’t yet know how to consistently recognize the squat pose. To do that, focus on the X-positions of these 4 parts:

      • Left hip
      • Right hip
      • Left knee
      • Right knee

      In a squat pose, the knees spread out wider than the hips. So:

      • Knee Distance = Right Knee X - Left Knee X
      • Hip Distance = Right Hip X - Left Hip X

      After we calculate these 2 distances, we can use them to determine when the player is in a squat pose.

      564edd7e-dbc2-43e3-b030-4bef8cfd2354.png

       
       

      Step 5 - Read Data from the Table

       

      Create 4 variables to represent the X positions of these 4 body parts: “left hip x”, “left knee x”, “right hip x”, “right knee x”.

      Display the “i” table and note each part’s row number. For example, the left hip X position is in row 12, under the column x:

      9fbcfc48-51c3-4c06-8b09-3fe970bdc235.gif

       
      Therefore, we can read this value into the “left hip x” variable like this:

      48ef9065-aa52-4846-a0a0-ece0e53606a2-image.png

       
      Repeat this process for all 4 variables:

      46aa8f81-a178-4ba0-a59d-ffabf38fa990-image.png

       
       

      Step 6 - Calculate Distances

       

      Create 2 more variables:

      • hip distance
      • knee distance

      Now calculate the distances:

      • hip distance = right hip x - left hip x
      • knee distance = right knee x - left knee x
         

      0a66bf99-a1e5-4366-9894-285493778f00-image.png

       
       

      Step 7 - Determine Pose from Distances

       

      Create a variable called pose. Use a simple rule like the one below to classify the pose:

      • If knee distance > 2 times of hip distance, then it’s a Squat
      • Otherwise, it’s Unknown

      6f52d490-a270-45f4-b671-3f643a7f0b1a-image.png

       
       

      Step 8 - Organize with a Custom Block

       

      Group the logic for pose detection into a new block called “calculate pose”, and move the related code into it.

      17815e92-c6fa-44ff-ab73-939e4708d462-image.png

       
       

      Step 9 - Handle Invalid Data

       

      Sometimes body parts are not detected and return a value of -10000 in the table.

      To avoid errors, only calculate pose when all 4 X values are > -250.

      8de3dba3-dc3b-44e5-a4d2-f9abeb9c7c76-image.png

       
       

      Step 10 - Continuously Calculate Pose

       

      Since the table updates continuously as the player moves, wrap the pose calculation inside a forever loop:

      d18a8fac-b1e3-4863-afb8-4da7f244d2ae-image.png

       
       

      Step 11 - Detect Success

       

      Now check if pose = “Squat”. If so, show a success message:

      86acf572-f9f1-4fde-8e63-bc70d826472d-image.png

       
      Result:

      succ1.gif

       
       

      Step 12 - Set the Target Pose

       

      Now we have the basic game mechanics worked out. The next step is to make it more fun. We will give the player more than one target pose, so the player has to try to complete these poses to earn points.

      As a first step, we should create a new variable “target pose”, which will represent the next target pose the player needs to make.

      We can rewrite our code using this new variable:

      f1930974-4250-489f-a3f2-63893c63b721-image.png

       
       

      Step 13 - Switch Between Squat and Stand

       

      When the user has completed the target pose, we should change the target to a new pose. To keep it simple, let’s make the target toggle between 2 poses: “Squat” and “Stand”.

      We can define a new block “change target pose” for this purpose. If the value of “target pose” is “Squat”, then the next target should be “Stand”. Otherwise, if the value of “target pose” is not “Squat”, it must be “Stand”.

      3f92d377-3152-473b-bcfd-532376dc2faf-image.png

       
       

      Step 14 - Create the Stand Pose Costume

       

      To show that the target pose has changed, we should change the costume of the sprite as well.

      Please find the “Dorian-c” costume and add it to your sprite. Rename it as “Stand”.

      ccbb49ba-1841-4bbd-bdeb-91790727ddf6-image.png

       
      We need to make some small adjustments to the costume. The leg on the left is reaching out sideways. We can select the whole leg and rotate it to an upright position:

      leg.gif

       
      In addition, the head is looking sideways, which may confuse the player. We can remove this head, and copy the head from the “Squat” costume:

      head.gif

       
       

      Step 15 - Switch Costume with Target Pose

       

      When changing the target pose, also update the costume accordingly:

      ffaa0bc1-4366-4650-ba05-315bbdd2cf40-image.png

       
       

      Step 16 - Add Visual/Sound Effects

       

      To give the player clear feedback that the target pose has been completed, we should show some visual effects. For example, we can make the sprite brighter for a short time before changing to the next costume. Also, we can play a sound for collecting a coin as well:

      7b07560d-03f8-41fc-b935-9ab4b946f189-image.png

       
       

      Step 17 - Detect the Stand Pose

       

      In the “calculate pose” block’s definition, currently we only check for the “Squat” pose. We also need to check if the player is in “Stand” pose. To keep it simple, we can still look at the distance between the knees and the distance between the hips. If the player is standing up, the knee distance should just be slightly more than the hip distance. For example, we can require that the knee distance is less than 1.2 times the hip distance:

      1e58e459-0f12-44e7-93e9-f605f171133f-image.png

       
      Now our game is ready. Whenever the player hits a target pose, we switch to the next target:

      6375abe6-43a0-42e8-b412-921e31f5af5a.gif

       
       

      Additional Challenges

       

      Here are some fun ways for you to build upon this project:

      • Game Rules: You can design different rules for playing the game. For example, you can give the player a time limit (e.g. 1 minute), and whoever earns more coins wins. You can also require the player to complete 60 target poses, and whoever finishes in the shortest time wins.

      • Improve the Pose Detection: Currently we only look at knees and hips. You can also add additional checks on the arms. For example, you can require the arms to be held horizontally. You can compare the Y position of the wrists with the shoulders.

      • Different Target Poses: You can try to use some other poses. For example, make the player do an “X” pose like below. You will need to use the X and Y positions of the wrists and ankles to detect such a pose. Note that you can also generate more positions using the AI image generation tool (first generate an image for a standing person, then generate a variation).

      31cb114d-7979-4809-aa66-c20cecbebd5b.png

      • 2-Player Mode: When 2 players are standing in front of the camera, both can be detected by the body pose detection engine. This allows you to create 2-player games. For example, whenever player A makes a pose, we set that to be the target for player B. This way, player B has to copy player A’s poses.
      1 Reply Last reply Reply Quote 0
      • Pinned by  info-creaticode info-creaticode 
      • First post
        Last post