AI - Recognize 0's and 1's (Difficulty: 4)

CreatiCode

Key Topics Covered

Introduction

Many AI programs can recognize handwritten numbers. For example, when we deposit a check using a phone app, it can detect the account number and the amount written on the check just from a picture.

In this tutorial, you’ll build a smart program that can recognize whether a handwritten number is 0 or 1:

Step 1 - Create a New Project

Start by creating a new Scratch project. Then:

Delete Sprite1 (Cody).
Rename the “Empty1” sprite to AI.
Add a new empty sprite named Pen.

The Pen sprite will be used to draw numbers on the screen. The AI sprite will contain the code to guess if the number is a 0 or a 1.

Step 2 - Initialize the “Pen” Extension

Next, please add the “Pen” extension, and set the pen’s color to black, size to 10. We will also erase all existing drawings from the pen.

Step 3 - Draw the Drawing Area

When the project starts, we’ll draw a square that shows the area where users can write their numbers. Use the “draw rectangle” block from the Pen extension (not from the “Looks” category).

Move the sprite to the center of the stage.
Set the rectangle’s width and height to 200.
Set the fill color to 100% transparent.
Choose any border color you like.

Here’s how it will look:

Step 4 - A Forever Loop To Move the Pen Sprite

We want to let users draw using the mouse. So, we’ll make the Pen sprite follow the mouse pointer.

After drawing the rectangle, use a forever loop to keep the Pen sprite moving to the mouse’s position:

Try it out! Move your mouse around and watch how the Pen sprite changes its x and y positions accordingly:

Step 5 - Mouse Down to Draw

To draw using the mouse, we’ll check whether the mouse is clicked (pressed down).

If the mouse is down, put the pen down.
If it’s not, lift the pen up.

Now our drawing area is ready for drawing:

Step 6 - The “Clear” Button

To allow the user to clear the drawing and restart, let’s add a “clear” button. Find the “add button” block from the “widget” category, and add the block at the very top:

Run the program again, and you will see the button at the bottom:

Now add 4 more blocks: when the clear button (“button1”) is clicked, we need to

Erase all the drawings
Move the Pen sprite to the center and redraw the rectangle.

Now you can try to draw something and erase it with the clear button.

Step 7 - The “Guess” Button

Add another button named “Guess” in the bottom right corner.

When this button is clicked, we’ll send a message to the AI sprite so it can guess the number:

Step 8 - Prepare the AI Sprite

Now let’s switch to work on the AI sprite. First, please draw a small square in the center of its costume, so we can see where it is on the stage. Make sure it is not too big or too small.

Step 9 - Scan the Middle Row

When the AI sprite gets the “guess” message, it should scan the middle row from left to right.

Make a variable named x.
Use a for-loop to set x from -100 to 100, increasing by 5 each step.

Click the “Guess” button to test it:

Step 10 - Touching Black Before and After Each Move

Now let’s check if the AI sprite is touching black color (the user’s drawing) at each step.

Make two variables: “touching before move” and “touching after move”.
Record their values before and after the sprite moves.
Add a 1-second wait after each move so you can observe the values.

Now let’s test it. You should observe the following:

When the AI sprite is starting from the left, both variables are false;
When the sprite touches the black color for the first time, “touching before move” remains as false, but “touching after move” becomes “true”.

Step 11 - Detecting New Edges

We can now detect the left edges of black strokes using the two variables.

If the sprite starts touching black after a move, but was not touching black before, that’s a new edge.
When that happens, make a stamp for testing purposes.

Now let’s test it. No matter how many lines we draw, the AI sprite will make a stamp the first time it touches each line:

Step 12 - Counting the Edges

Instead of stamping, let’s count how many times we detect a new edge.

Make a new variable called “counter”.
Initialize counter as 0
Every time we detect a new edge, increase the counter by 1.

Run the project. The counter tells how many vertical edges the AI sees—each edge is a transition from white to black.

Step 13 - 0 or 1?

Now we can guess the number!

If the user draws a 1, the counter shows 1 edge.
If they draw a 0, it shows 2 edges.

So we use the value of the counter to guess:

If counter = 1 → it’s a “1”
If counter = 2 → it’s a “0”

(We’re assuming users will only draw 0 or 1.)

Please test it a few times:

Step 14 - Handling Smaller 0’s

When the user draws a very small or narrow “0”, we can only detect one edge, so we will guess incorrectly.

That is because our AI sprite box is touching both strokes at the same time. So to fix it, we simply need to make our AI sprite much smaller, such as a size of 5:

Step 15 - Speed Up

Lastly, to speed up the scanning, we can make a new block named “guess”, and run it without screen refresh:

Then we move all the blocks into that block’s definition stack:

Now our program runs much faster, because we won’t update the stage when the AI sprite moves across the stage:

Additional Challenges

This tutorial shows a very common technique that analyzes an image using edge detection. As a challenge, please try to extend this program to recognize some other numbers. You need some smart ideas to handle them in a robust way.

For example, if you are trying to recognize the number “6”, you will need to scan all rows, not just the middle row. Then you can check if the number of edges is 1 at the top, becomes 2 towards the bottom, and then reverts to 1 at last.

Similarly, if you are trying to recognize the number “8”, you can check if the number of edges is 2 at the top and bottom, but 1 in the middle.

Since the user might draw the number in the top or bottom part of the box, you may also need a preparation step, where you scan the image to find out the range of Y positions where there is any drawing. Then you can only focus on those rows.

Here is a demo of an enhanced version of the project: