Autograded Coding Quizzes for Assessment

CreatiCode

Formative Assessment for Learning Scratch

MIT’s Scratch platform offers a dynamic environment for open-ended projects, where millions of students develop and share creative games and stories. However, assessing student learning in Scratch-based coding classes has traditionally been limited to multiple-choice questions, which may not fully capture a student’s coding skills.

The CreatiCode platform enhances Scratch by enabling hands-on coding problems for formative assessment. This allows teachers to identify gaps in student understanding early and provide tailored learning materials to support individual progress.

Overview

CreatiCode enables teachers to create automatically graded hands-on coding quizzes, such as Parsons Problems. These assessments provide a more effective alternative to multiple-choice questions by directly evaluating students’ coding skills in a practical context.

This process takes 4 steps:

Create a new project, which asks the students to complete a task by adding/changing the blocks in it.
Add “secret” code to this project to automatically grade the student’s work and submit the result. Auto-grading can be done using ChatGPT or your own logic.
Create a new “quiz” using the project
Create a new “assignment” using the quiz that includes due dates.

Now, let’s walk through this process with a complete example.

Create a Coding Challenge Project

Step 1 - Remix a Starting Template

First, please remix the following project:

https://play.c reaticode.com/projects/6593e905a3c834dd5277053f/editor

Step 2 - Overview of the First Sprite

You should see 2 sprites in this project. The first sprite is called “Sprite1”.

This is where the student will make changes to complete the challenge. This example is a Parsons problem, since all the blocks needed by the student are given in the “short list” on the left, so students just need to compose them into a working program. A rich-text comment is added to the green flag block, which explains the task:

You can change the short list to any set of blocks. For example, you can provide more blocks than necessary, or no blocks in the short list at all.

You can also change which blocks are given in the editor as the starting point of the quiz. For example, you can add more blocks so students only need to work on the new blocks they have just learned. You can even provide an incorrect program for the students to fix.

In this example, we are asking students to attach blocks to the “when green flag clicked” block, which is the most natural starting point, since students can simply click the green flag to test their work. However, you might choose to ask the student to add new blocks below a different type of hat block, such as “when I receive message” or “when this sprite is clicked”, etc.

Step 3 - Overview of the Secret Sprite

The second sprite in this project is named “Secre_t”. Note that you need to rename it to “Secret” so that it is hidden from any other user. That is, when students open or remix this project, they will not see this Secret sprite.

This sprite contains 2 stacks of blocks. The first stack is used to store the correct solution, and the second stack uses ChatGPT to evaluate the student’s code against some rubric. We will look into each stack below.

Step 4 - Compose the Correct Solution in Sprite1

To automatically evaluate whether a student’s program is correct or not, it is often necessary to know the correct solution. This is especially important if we need to use ChatGPT to evaluate the student’s code, as it sometimes fails to correctly analyze student’s code without a correct solution as reference.

Therefore, in this step, let’s build a solution program based on the challenge:

Step 5 (Optional) - Set up the project’s initial state

When the student’s code is triggered by the “when green flag clicked” block, there is one common issue: we can not specify the initial state of the program. For example, suppose we want to test the student to always initialize a variable before using it. The best way to do that is to set that variable to a random value BEFORE the student’s code runs, so if the student’s code doesn’t initialize that variable correctly, it will be using a random variable value.

To make this easy, the CreatiCode playground offers a new block “prepare for green flag click”, and any code attached to this block will run before the blocks under the green flag block. For example, we can initialize the “number” variable like this:

Add Code to Grade the Student Solution Automatically

One of the key advantages of creating quiz problems using the CreatiCode platform is that the project for students to work on and the logic to evaluate student solution are both stored in the same project. This makes it much easier to create or update such quiz problems.

After the teacher shares the project with their students, the students will remix this project, and implement their solution. Then, when they click the green flag, the logic in the “Secret” sprite will be triggered to automatically grade the student’s solution.

Step 1 - Save the Correct Solution as Text

Since we can not operate on blocks directly, the first step is to convert the correct solution to text format.

To do that, switch to the “Secret” sprite, and run these 2 blocks:

The first block will read all the information (e.g. blocks, costumes, etc) about Sprite1, and store them in the given list “script”. The first item in the list will be the text representation of the code blocks in Sprite1, and we will store them in the variable “solution code”.

In this example, the “solution code” variable will have the following value, which is the program in Sprite1 in pseudo code format:

Step 2 - Read the Student’s Code

To automatically grade the student’s code, we also have to read the student’s code from the Sprite1 into “my code”, similar to how we read the reference solution. This is done whenever the green flag is clicked, as we assume the student has remixed our quiz project and completed the code changes.

For example, suppose this is the student’s solution, which fails to change the “number” variable by 1 in the repeat loop:

Then “my code” will have the following value:

Step 3 - Send Quiz Information to ChatGPT

At this point, we know both the correct solution and the student’s solution (both in pseudo code format), and there are several methods for analyzing the correctness of the student’s solution. The most generic and robust solution is to ask ChatGPT to do the grading for us. We will need to compose a fairly long prompt to fully explain what we need ChatGPT to do, and provide all the information it needs.

To get started, we need to tell ChatGPT its task, and give it the quiz’s rubric and the reference solution:

Note that to change this code based on your quiz problem, you need to make 2 changes:

Update the rubric (the green text) in the comment box. The rubric should describe what the student’s code is expected to do, and any other requirements like which blocks should be used. Since we are only giving information to ChatGPT at this step, we only need to ask ChatGPT to respond with a “Yes”. This is a commonly used technique to avoid long responses from ChatGPT.
After updating the rubric, copy the entire content of the comment box and paste it into the “set [info prompt] to” block below it. After that, the “info prompt” and the “solution code” will be joined into the final request we send to ChatGPT.

Step 4 - Ask ChatGPT to Evaluate the Student’s Code

Now, we are finally ready to ask ChatGPT to evaluate the student’s code against the requirement. Since we have already told ChatGPT about the quiz, the rubric and a reference solution, we only need to give it any additional requirements on the format of its response and the actual code from the student:

The prompt asks ChatGPT to respond with “PASS” or “FAIL” first. And in case of “FAIL”, it will ask ChatGPT to also provide additional explanations.

Note that you don’t have to modify this part of the code in most situations. However, if you have some special requirements on how ChatGPT should write the feedback, you can modify the prompt in the comment box, then copy it into the “set [evaluate prompt] to” block.

Step 5 - Display ChatGPT’s feedback to the student

After running the blocks above, the “result” variable will contain the response from ChatGPT. We can show that in an “alert” window like this:

Step 6 - Submit the Evaluation Result

Lastly, we will need to submit the evaluation result to the CreatiCode server, so that we can review all results as a teacher.

We can check if the result starts with “FAIL” or not, and set the assignment to be “fail” or “pass”. We will also store the student’s code and the full response from ChatGPT. You will learn where to view these results below.

Step 7 - Share the Project

Now, we are done with creating the quiz problem. It has to be shared so that students can open and remix it. Usually, you can use “share unlisted” (so only users with the exact URL can open the project) or “share with password” (users need to input a password to open the project).

Alternative Methods for Evaluating Student Code Automatically

Instead of asking ChatGPT to grade the student’s solution, there are 2 other methods worth mentioning here.

First, for coding challenges with very simple and fixed solutions, we can analyze the student’s code text directly. For example, we can check if the value of “my code” contains the text “change [number v] by (1)”, or we can check the value of the “number” variable after a few seconds. Of course, this method may not work well when there are multiple ways to solve the challenge.

Second, a slightly more robust method is to use the console log. To do that, the quiz can require the students to write code to print certain information in the console in our rubric. For example, instead of “say” the number, ask the student to print 1, 2, 3 in the console panel using the print block:

Next, in the Secret sprite, we can wait a few seconds after the green flag is clicked, then check the console log’s content. We can read its content using the “get console log” block, then verify if the log is the same as expected. You can specify the expected output directly like “1\n2\n3”. Alternatively, you can run the correct solution, save the console log from it into a variable, then compare that with the console log produced by the student’s code when the student runs the program.

Create a Quiz

After creating a coding project with autograding logic, we can create a quiz using that project.

First, go to the “My Stuff” page in the CreatiCode playground, and find the “Quizzes” tab on the left:

Next, click the “Add a new quiz” button, and fill in the following information:

You can assign any name to this quiz, and copy the URL of the project you created above. The “Custom XO Prompt” is reserved for the future, and you can leave it as blank for now. The “Notes” is any additional notes you would like to add for yourself. After clicking “Add”, you will see a new quiz in the list:

For each quiz, there are 3 links available:

You can click “View Project” to open the project associated with that quiz to review or update it.
You can click “Edit” to change the name, project link or note for this quiz.
You can click “Delete” to remove this quiz.

Create a Class Assignment

Now, we can create an assignment using the quiz created above. The key difference between an assignment and a quiz is that assignments have due dates, and they can be assigned to a class of students.

Step 1 - Create a class and add your students

If you haven’t done so, you need to create a class under “My Classes”, as described in this tutorial: Teacher Only: How to Manage Students in the “My Classes” Page

Step 2 - Create a new assignment

To create an assignment for any class, select that class in the “My Classes” page, and switch to the “Assignments” tab, which will display all assignments you have created for this class:

Click the “+ New assignment” button on the top right to add a new assignment:

For “Quiz”, you can select a quiz from the dropdown. If you don’t see the new quiz you just added, refresh the page to update the list.
For “CreatiCode XO”, you can choose to disable it so that when the student opens this project in the playground, the CreatiCode XO button will be hidden.

Step 3 - Students work on the quiz

After the start time of the assignment, students will see a “start” link next to the assignment on their “My Class” page:

When a student clicks “start”, the project shared by the teacher will be remixed into a new project in this student’s account. This way, the student can make changes to it and save it.

Whenever they click the green flag button to test their solution, the evaluation results will be submitted to the CreatiCode database by the code in the Secret sprite (invisible to the students). If they run it again, the new result will overwrite the previous ones.

Also, the student can choose to restart the project using the “restart” link after it has started:

When “restart” is clicked, a new remix project will be created from the original project shared by the teacher.

Step 4 - Teacher Reviews Results

Teachers can review these results using the “Activity” link next to the assignment:

The results will be displayed in a popup window. It will show the “Status” of “Pass” or “Fail”, as well as the explanation and code submitted from the Secret code.