AI - Counting Fingers with Hand Detection (Premium, Difficulty 3)

CreatiCode

Key Topics Covered

Introduction

In this tutorial, you will build a small project that counts how many fingers are stretching out:

Step 1 - A New Project

Please create a new project in the CreatiCode playground. Remove the sprites named “Sprite1”. We will only need to use the “Empty” sprite.

Step 2 - Add Number Costumes

We will be showing the number we have recognized by switching to different costumes. Please switch to the “Costumes” tab, add the “Glow-0” costume, and remove the original costume:

Similarly, please add the costumes “Glow-1” to “Glow-5”, so there are 6 costumes in total. Costume number 1 should show 0, and costume number 6 should show 5.

Step 3 - Start Hand Detection

Now let’s switch to the “Code” tab, and add the hand detection block from the “AI” category (premium subscription needed).

When you click the green flag, this block will turn on the camera on your computer, and try to find the hands in the camera video.

As shown, for each hand, 21 dots are drawn to indicate the key “landmarks”: 4 landmarks for each finger, and 1 landmark for the wrist.

Step 4 - Review the Data Table

A new table named “i” is automatically added to the project when you add the hand detection block. Go to the “Variables” category to find the table name “i”, then click its checkbox to display its content on the stage.

Now you should see the table being updated in real-time on the stage. For each hand, 47 rows of data will be added to this table:

Step 5 - Observe the Curling Angle for Each Finger

For this project, we simply need to know whether each finger is stretched out or folded. We can look at the first 5 rows of data in the table, and focus on the “CURL” column.

This column tells us the curling angle of each finger. If the finger is stretched out in a straight line, then this angle should be close to 180 degrees. As the finger curls, this angle will gradually reduce towards 0.

For example, observe how the “curl” value changes as you curl your index finger:

Therefore, to determine if a finger is stretched out, we can simply check if its curling angle is greater than a threshold value, such as 150 degrees.

Step 6 - A Counter Variable

Now our task is to count the number of fingers that are stretched out. We will need a counter variable, which can help us do the counting. Please create a new variable named “counter”.

Step 7 - A Forever Loop

To get a real-time count of the fingers, we will need to repeatedly check the fingers. Let’s use a forever loop, which sets the “counter” to 0 by default.

Step 8 - Check If There is At Least One Hand

We do not need to do any work if no hand is detected in the camera video. The easiest way to check that is to look at the number of rows in the table “i”. If the row count is at least 47, that means the AI has found at least one hand:

Note that the “row count” block is also in the “Variables” category.

Step 9 - Check Row 1 to Row 5

Since we will be looking at data in rows 1 to 5, we should create a new variable named “row”, and use a for-loop to make this variable go from 1 to 5:

Step 10 - Read the “curl” Value for Each Row

We can read the value in the “curl” column for any row like this:

To read all 5 rows one by one, we just need to use the “row” variable to specify the row number in the block:

Step 11 - Increase the Counter

When we find the curling angle is more than 150 degrees for any of the 5 fingers, we will increase the counter by 1:

After the for-loop completes, the “counter” variable will contain the correct count of stretched-out fingers.

Step 12 - Switch the Costume

For our last step, we just need to switch to the corresponding costume based on the “counter” variable. When “counter” is 0, we should show costume number 1, and when “counter” is 1, we should show costume number 2 … Therefore, a simple pattern is that we always show the costume numbered “counter + 1”. Here is how to do that in blocks:

Here is what you should get:

Enhancement

This finger-counting program is very simple. There are a lot of places where you can make improvements. Here are some ideas:

Recognize 6: When you show the number “6”, currently the program still reports “2”, since 2 fingers are stretched out. Can you make the program smarter? How about other digits like 7, 8 and 9?

2 Hands: You can try to recognize 2-digit numbers with 2 hands
Other Gestures: Besides numbers, you can use the “curl” angle to help recognize simple hand gestures, such as “OK”.