Navigation

    CreatiCode Scratch

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups

    AI - Counting Fingers with Hand Detection (Premium, Difficulty 3)

    Tutorials
    1
    1
    212
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • info-creaticode
      CreatiCode last edited by info-creaticode

       

      Key Topics Covered

      • Using variables
      • Reading from tables
      • AI for Hand Detection

       
       

      Introduction

      In this tutorial, you will build a small project that counts how many fingers are stretching out:

       
       

      Step 1 - A New Project

      Please create a new project in the CreatiCode playground. Remove the sprites named “Sprite1”. We will only need to use the “Empty” sprite.

       
       

      Step 2 - Add Number Costumes

      We will be showing the number we have recognized by switching to different costumes. Please switch to the “Costumes” tab, add the “Glow-0” costume, and remove the original costume:

       
      Similarly, please add the costumes “Glow-1” to “Glow-5”, so there are 6 costumes in total. Costume number 1 should show 0, and costume number 6 should show 5.

       
       

      Step 3 - Start Hand Detection

      Now let’s switch to the “Code” tab, and add the hand detection block from the “AI” category (premium subscription needed).

      0d65cfc5-d1d2-4141-b597-8b5950369323-image.png

       
      When you click the green flag, this block will turn on the camera on your computer, and try to find the hands in the camera video.

       
      As shown, for each hand, 21 dots are drawn to indicate the key “landmarks”: 4 landmarks for each finger, and 1 landmark for the wrist.

       
       

      Step 4 - Review the Data Table

      A new table named “i” is automatically added to the project when you add the hand detection block. Go to the “Variables” category to find the table name “i”, then click its checkbox to display its content on the stage.

       
      Now you should see the table being updated in real-time on the stage. For each hand, 47 rows of data will be added to this table:

       
       

      Step 5 - Observe the Curling Angle for Each Finger

      For this project, we simply need to know whether each finger is stretched out or folded. We can look at the first 5 rows of data in the table, and focus on the “CURL” column.

      This column tells us the curling angle of each finger. If the finger is stretched out in a straight line, then this angle should be close to 180 degrees. As the finger curls, this angle will gradually reduce towards 0.

      For example, observe how the “curl” value changes as you curl your index finger:

      1.gif

       
      Therefore, to determine if a finger is stretched out, we can simply check if its curling angle is greater than a threshold value, such as 150 degrees.

       
       

      Step 6 - A Counter Variable

      Now our task is to count the number of fingers that are stretched out. We will need a counter variable, which can help us do the counting. Please create a new variable named “counter”.

       
       

      Step 7 - A Forever Loop

      To get a real-time count of the fingers, we will need to repeatedly check the fingers. Let’s use a forever loop, which sets the “counter” to 0 by default.

      a1cf27fc-102d-4e6a-a5ae-5e60aa81388e-image.png

       
       

      Step 8 - Check If There is At Least One Hand

      We do not need to do any work if no hand is detected in the camera video. The easiest way to check that is to look at the number of rows in the table “i”. If the row count is at least 47, that means the AI has found at least one hand:

      356c34cb-b788-4edf-b688-1306a1fb3c2f-image.png

       
      Note that the “row count” block is also in the “Variables” category.

       
       

      Step 9 - Check Row 1 to Row 5

      Since we will be looking at data in rows 1 to 5, we should create a new variable named “row”, and use a for-loop to make this variable go from 1 to 5:

      6a8e12d2-ff26-419e-93e2-82c5f22b7561-image.png

       
       

      Step 10 - Read the “curl” Value for Each Row

      We can read the value in the “curl” column for any row like this:

       
      To read all 5 rows one by one, we just need to use the “row” variable to specify the row number in the block:

      2c39ff15-268a-4cc0-9402-008acd639af5-image.png

       
       

      Step 11 - Increase the Counter

      When we find the curling angle is more than 150 degrees for any of the 5 fingers, we will increase the counter by 1:

      c1692e1b-24f6-4ec7-9e21-df1829ccc3ad-image.png

       
      After the for-loop completes, the “counter” variable will contain the correct count of stretched-out fingers.

       
       

      Step 12 - Switch the Costume

      For our last step, we just need to switch to the corresponding costume based on the “counter” variable. When “counter” is 0, we should show costume number 1, and when “counter” is 1, we should show costume number 2 … Therefore, a simple pattern is that we always show the costume numbered “counter + 1”. Here is how to do that in blocks:

      0a9f98ae-795b-4e6e-8f35-3b39f7e63916-image.png

       
      Here is what you should get:

       
       

      Enhancement

      This finger-counting program is very simple. There are a lot of places where you can make improvements. Here are some ideas:

      • Recognize 6: When you show the number “6”, currently the program still reports “2”, since 2 fingers are stretched out. Can you make the program smarter? How about other digits like 7, 8 and 9?

      • 2 Hands: You can try to recognize 2-digit numbers with 2 hands

      • Other Gestures: Besides numbers, you can use the “curl” angle to help recognize simple hand gestures, such as “OK”.

      1 Reply Last reply Reply Quote 0
      • Pinned by  info-creaticode info-creaticode 
      • First post
        Last post