AI for Hand Detection

CreatiCode

Description

You can use the following AI block to automatically recognize the user’s hands from the front camera of the computer:

This block takes 3 input parameters:

Output Table: the first input allows you to select a table to be used to store the output of this block. Note that when you add this block to the code editor, a new table will be created automatically, and you can change that table’s name if you like. See below for an explanation of the data written in the table.
Debug: if this input is “yes”, then the outline of the hand will be drawn over the camera video like this. Note that a total of 21 points will be identified: the wrist, and then 4 points for each of the 5 fingers. The points are drawn as red dots for the left hand and blue dots for the right hand.

Show Video: if this input is “yes”, then the video from the camera will be displayed on the stage as a background. If this input is “no”, then the video will not be displayed, which will make the program run faster. If your program simply uses the data in the output table, then you can choose not to show the video.

The Output Table

When no hand is detected in the camera video, the output table will be empty. It contains these 7 columns.

When one hand is detected, the table will be populated with 47 rows of data. If 2 hands are detected, another 47 rows will be added below, making it a total of 94 rows.

For each hand, there are 3 types of data, and each type uses a different subset of the columns:

Finger Pose

For each hand, the first 5 rows contain the curl angle and direction of each finger. These 5 rows contain enough information for most of the simple use cases.

Only the first 4 columns are populated with data for these 5 rows:

Hand: This column tells you whether this is the left hand or the right hand.
Part: This column tells you which finger this row is describing.
Curl: This is the curl angle of that finger. It should be a number between 0 and 180. For example, for the index finger, when it is straight, the curling angle is 180 degrees. As you curl your finger, this angle drops down to as low as about 40 degrees.
Dir: This is the pointing direction of this finger. It uses the same direction system as the “point in direction” motion block. For example, when the finger is pointing up, the direction is about 0 degrees. When the finger is pointing to the right, the direction is about 90 degrees. For the left direction, it is about -90 degrees. For the down direction, it is about 180 or -180 degrees. Of course, the actual direction value will not be exact, so your program needs to check if the value is in a range. For example, if the direction is more than -30 or less than 30, you can treat it as pointing up.

2D Finger Position On Stage

Below the 5 rows of finger pose data, there are 21 rows of data describing the X and Y positions of the 21 points. 4 columns are populated with data:

Hand: This column tells you whether this is the left hand or the right hand.
Part: This column tells you which of the 21 points this row is describing. The first row is the “wrist” point, and then there are 4 rows for each of the 5 fingers. For example, for the thumb, the 4 points are “thumb 1”, “thumb 2”, “thumb 3” and “thumb 4”, from bottom up.

X and Y: These values use the same position system as the sprites, with the center of the stage at X = 0 and Y = 0. For example, in the picture below, the wrist is at the bottom center of the stage window, so its X position is around 0, and its Y position is around -180.

3D Finger Position On Stage

Below the 21 rows of 2D finger position data, there are another 21 rows of data describing the 3D positions of the same 21 points. 5 columns are populated with data:

Hand: This column tells you whether this is the left hand or the right hand.
Part: This column tells you which of the 21 points this row is describing.
X/Y/Z: These values use a 3D position system. If you use a small box to represent each point, then when you hold your hand forward, these boxes will form a 3D hand facing forward (same direction as your real hand):

You can open this link to play with this demo:

https://play.creaticode.com/projects/ef0d19a26944613ac1805db7