AI - Recognize 0's and 1's (Difficulty: 4)

CreatiCode

Key Topics Covered

Introduction

There are many AI applications that can recognize numbers automatically. For example, when we deposit a check, we can take a picture of it on the phone, and the app will recognize the account numbers and the deposit amount written on the check.

In this tutorial, you will build a smart program that can recognize whether a hand-written number is 0 or 1:

Step 1 - Create a New Project

First, please create a new project, remove Sprite1 with Cody, and rename the “Empty1” sprite to “AI”. Please also add an empty sprite named “Pen”.

The "Pen’ sprite will allow the user to draw a number on the stage, and the “AI” sprite will contain code to guess whether this number is 0 or 1.

Step 2 - The Drawing Area

We’ll start with the “Pen” sprite. First, when the program starts, let’s draw a square area to indicate where the user can draw the number. This can be done using the “draw rectangle” block in the “Looks” category. Note that this square will be located at the center, with both width and height of 200. Please set its “fill” color to 100% transparent, and pick any border color you like.

Here is what the square looks like:

Step 3 - Initialize the “Pen” Extension

Next, please add the “Pen” extension, and set the pen’s color to black, size to 10. We will also erase all drawings from the pen.

Step 4 - A Forever Loop To Move the Pen Sprite

We will allow the user to use the mouse pointer to draw. To do that, we need to make the Pen sprite follow the pointer. You can use a forever loop to make the Pen sprite keep going to the position of the mouse pointer.

Now if you move the mouse, you will see the position of the “Pen” sprite changes with it:

Step 5 - Mouse Down to Draw

To use the mouse pointer as the pen, we can check if the mouse is pressed down. Whenever it is, we will put the pen down as well; whenever the mouse is not pressed down, we will lift the pen up.

Now our drawing area is ready for drawing:

Step 6 - The “Clear” Button

To allow the user to clear the drawing and restart, let’s add a “clear” button. Please load the “widget” extension, then find the “add button” block.

Run the program again, and you will see the button at the bottom:

When the button is clicked, we need to erase all the drawings:

Step 7 - The “Guess” Button

Please add another button “Guess” at the bottom right. When it is clicked, we’ll send a message to the “AI” sprite to take a guess of the number.

Step 8 - Prepare the AI Sprite

Now we are ready to work on the AI sprite. First, please draw a small square in the “AI” sprite’s costume, so we can see where it is:

Step 9 - Scan the Middle Row

When the “AI” sprite receives the “guess” message, it should scan the middle row from left to right. We can create a new variable “x”, and use a for-loop to set x to go from -100 to 100 at a step size of 5.

Click the “Guess” button to test it:

Step 10 - Touching Black Before and After Each Move

Now we are going to check whether our AI sprite is touching the black color of the drawing at each step. We will use 2 new variables to keep track: “touching before move” and “touching after move”. They will be set to the value of “touching black color” before and after the “go to” block. To view the values in slow motion, we will add a 1-second wait after each move:

Now let’s test it. When our AI sprite touches the black color for the first time, “touching before move” will still be false, but “touching after move” will become “true”.

Step 11 - Stamping the Strokes

Now we can detect a black stroke using the 2 variables we have created. We will make a stamp whenever the AI sprite finds that it is touching the black color after the move, but not touching it before the move.

Now let’s test it. No matter how many strokes we draw, the AI sprite will make a stamp the first time it touches each stroke:

Step 12 - Counting the Strokes

Now we can count how many strokes our AI sprite is detecting. We will use a new variable named “counter”, and instead of making a stamp, we also increase the counter by 1.

Run the project, and you will find that the counter will tell us how many vertical strokes are there. More specifically, it is telling how many “edges” are there, with each edge defined by going from white to black color.

Step 13 - 0 or 1?

Finally, we are ready to answer the question: whether the user has drawn a 0 or 1?

The insight is that when the user draws a 1, there is only one edge; but if the user draws a 0, there are 2 edges.

Therefore, we can take a guess based on the value of “counter”. If it is 1, then the user has drawn a “1”. If it is 2, then the user has drawn a “0”. Of course, to keep it simple, we are assuming the user can only draw a 0 or 1.

Please test it a few times:

Step 14 - Handling Smaller 0’s

When the user draws a very small or very thin “0”, we can only detect one edge, so we will guess incorrectly.

That is because our AI sprite box is touching both strokes at the same time. So to fix it, we simply need to make our AI sprite much smaller, such as a size of 5:

Step 15 - Speed Up

Lastly, to speed up our AI detector, we can make a new block named “guess”, and run it without screen refresh:

Then we move all the blocks into that block’s definition stack:

Now our program runs much faster:

Challenges

This tutorial shows a very common technique that analyzes an image using edge detection. As a challenge, please try to extend this program to recognize some other numbers. You need some smart ideas to handle them in a robust way.

For example, if you are trying to recognize the number “6”, you will need to scan all rows, not just the middle row. Then you can check if the number of edges is 1 at the top, becomes 2 towards the bottom, and then reverts to 1 at last.

Similarly, if you are trying to recognize the number “8”, you can check if the number of edges is 2 at the top and bottom, but 1 in the middle.

Here is a demo of an enhanced version of the project: