3 Practice testing

In this chapter you’ll learn how to use AI to test yourself. Practice testing, or retrieval practice, is one of the most effective ways to consolidate learning, and it can be applied in several formats. The evidence on this is about as consistent as evidence in psychology gets: testing yourself beats re-reading and re-watching, free recall beats recognition, and harder testing beats easy testing. This chapter was written with a data skills course in mind but it applies to any type of course.

You should have a specific week/chapter/lecture of your course to work with. For PsyTeachR courses like Applied Data Skills, we’ll be using the Intended Learning Outcomes, functions used, and glossary to help instruct the AI. If you’re not working through a course, or it’s not a PsyTeachR course, it would be helpful to have a list of functions, concepts, or skills that you want to test.

It’s very important that you use Copilot through your UofG account so that the course content you share is not used to train the model, and remember from chapter 1 that teaching materials are copyrighted. You should also ensure that you have the consent of your lecturer to do this (if you’re enrolled on Applied Data Skills, you have our consent to upload any course material to Copilot, but not to platforms like ChatGPT).

3.1 Prompts or the agent?

Copilot has a Study and Learn agent with a Quiz mode built in (described later in this chapter) and you might reasonably ask why we’re going to spend most of this chapter writing our own prompt instead. Three reasons. First, the agent is built for facts and vocabulary; it does MCQs and flashcards well and coding problems and error mode badly, and if you’re on a data skills course the coding questions are the ones you need. Second, our prompt lets you control what the AI is allowed to use: which datasets, which functions, what’s been covered and what hasn’t. The agent doesn’t give you that control, and questions that use things you haven’t been taught are demoralising rather than useful. Third, you won’t always have this agent. You might use a different AI, Microsoft might rename or remove it, and the skill of setting up an AI to test you properly is one you can take anywhere. Use the agent when it’s convenient; know how to build your own.

3.2 Question types

The first thing we’re going to do is set-up several different prompts to create different types of practice questions. Which of these you end up using will depend on what you’re trying to study. Different question formats test different aspects of your knowledge. Here’s what each type is for and what the AI will produce when you ask for it.

3.2.1 Questions that test recognition

3.2.1.1 Multiple choice questions

MCQs test recognition and your ability to spot the best answer among distractors. Each item has a stem (the question) followed by four labelled options (A–D). Only one is correct. MCQs are efficient for covering a wide range of content, easy to self-check, and useful for practising discrimination between similar concepts. However, they can also encourage recognition more than recall, and can sometimes be answered by guessing or test-wise strategies.

Example:

Which of the following functions is used to create a scatterplot in R? A) geom_bar() B) geom_point() C) geom_boxplot() D) geom_histogram()

3.2.1.2 True-or-false (TOF)

TOF questions are a quick way to check factual accuracy and basic conceptual understanding and involve a single statement for you to judge as true or false. TOF questions are very quick to complete and good for simple checks of knowledge but have a high chance of guessing correctly (50%) and can oversimplify complex concepts and have limited diagnostic value.

Example:

True or false? The mean() function in R returns the median of a numeric vector.

3.2.1.3 Fill-in-the-blanks (coding)

These questions strengthen fluency with R syntax by making you supply missing functions or arguments but are easier than writing out the full code from scratch These questions involve a line of R code with a missing element. These questions support recall of key syntax and reduce cognitive load compared to writing from scratch but they can be too easy and may not transfer well to real coding tasks.

Example:

ggplot(mtcars, aes(x = wt, y = mpg)) + ____()

3.2.1.4 Fill-in-the-blanks (theory)

Similarly, for theory, FITB helps memorise key terms, definitions, or concepts without giving the full answer. These questions will be a sentence with 1–2 blanks, sometimes with hints. FITB is good for reinforcing vocabulary and key concepts and are quick to create and practise. But, there is a risk of rote memorisation without deeper understanding and answers may sometimes be ambiguous.

Example:

“A variable that can take on any value between two points is called a ______.”

3.2.2 Questions that test production

3.2.2.1 Short-answer-questions

SAQs require you to recall and explain in your own words, building deeper understanding. These are focused, open questions that should be answered in <100 words. SAQs promote active recall and deeper processing and are flexible enough to test conceptual understanding. However, they are harder to self-mark and may be more time-consuming to generate and answer.

Example:

Explain the difference between a categorical and a continuous variable.

3.2.2.2 Coding problems

Coding problems let you apply your skills to solve a real task, similar to authentic assessments. They usually involve a short programming challenge using a real dataset. Coding problems closely mimic real-world problem solving, encourage transfer of knowledge, and consolidate multiple skills at once. The higher difficulty can be discouraging for beginners and they may be harder to self-assess without feedback.

Example:

“Using the penguins dataset, create a boxplot of body mass grouped by species.”

3.2.2.3 Error mode

Error mode is debugging practice which builds resilience and problem-solving skills, and teaches you to spot common mistakes. It involves a runnable piece of R code that contains one plausible error. Error mode develops error-detection and debugging skills and mirrors real-world coding experience. However, it can be frustrating for novices and requires baseline knowledge to be effective.

Example:

ggplot(mtcars, aes(x = wt, y = mpg))
geom_point()

(Error: missing + before geom_point())

3.3 Practical tip: Built-in datasets

When practising coding questions with AI, you need a dataset you can actually run code on. A common problem with AI-generated exercises is that it invents datasets or column names that do not exist, which makes it impossible to test your solution. One way to avoid this is to rely on built-in datasets.

Built-in datasets in R are sample datasets that come pre-loaded with the software or with specific packages. They cover a variety of domains (e.g., cars, gemstones, movies, penguins) and are designed to help you practise data manipulation, analysis, and visualisation without importing external files.

You can get a full list of available datasets by running data() in the console. Base R provides some, and additional packages (e.g., tidyverse, palmerpenguins) add more. Remember that a package must be loaded before its datasets are accessible.

# see list of datasets
data()

# load tidyverse to get access to extra sets
library(tidyverse)

# see list of datasets expanded to include tidyverse ones
data()

# load in dataset to environment so it can be used
data("starwars")

3.4 Question prompt

We will now design a prompt that will set-up Copilot to give you different types of questions.

Activity 1

Edit the below template to suit your needs. For example, change the role, the sources you’re going to use, and if relevant, the datasets it should draw on in constraints. You can also change the other details although most of them should work for you without any changes.

Template prompt to copy and paste into Copilot

Role: You are a strict practice-testing tutor for second year undergraduate students learning R.

Sources: I will paste Intended Learning Outcomes (ILOs), function lists, and glossary terms from my course Applied Data Skills.

Protocol:

Ask one question at a time of the requested type.
Do not reveal the answer until I reply.
Do not give the answer away in the suggested follow-up prompts. For example, for MCQs, give suggested prompts for each answer option, not just the correct answer.
After I answer, mark it, then give a short explanation (2–4 sentences).

Constraints:

All questions must align to ILOs and be challenging but fair.
For coding questions, avoid imaginary datasets or columns. Instead, use only use the following datasets: starwars, diamonds, penguins.

Controls I will use:

type: … (see list below)
calibrate: harder | easier

Question Types (with rules):

MCQ (Multiple choice) – 4 options (A–D), 1 correct. Plausible distractors. After marking, explain each option briefly.
TOF (True/False) – One statement; avoid trivially true/false. After marking, if false, rewrite as a correct statement.
SAQ (Short answer) – Ask about one concept. I should be able to answer in <100 words. After marking, provide a 2–3 point ideal outline.
FITB_code (Fill-in-the-blank: coding) – One line of R with a missing function/argument. Must run on an approved dataset. State expected output shape (e.g., “tibble: 3 × 2”).
FITB_theory (Fill-in-the-blank: theory) – A statement with up to 2 blanks. Give part-of-speech hints (e.g., [noun]).
CP (Coding problem) – Minimal complete example using an approved dataset. Specify columns that exist. Do not invent data.
EM (Error mode) – Provide a minimal reproducible example that fails using a specified dataset. The code should include the call to load the dataset and any required packages. Include exactly one plausible beginner-level error but give no hint as to what the error is. State intended outcome. Hold back the fix until after I attempt a solution. Then reveal the correction and what it teaches.

First action: Confirm readiness and ask me to provide the sources.

Save it somewhere

You will want this prompt again and again so keep a copy in a text file rather than scrolling back through your chat history. If you use Claude or ChatGPT for anything outside University work, both let you save a prompt like this so that it’s available in every chat without pasting it in: Claude calls these skills, ChatGPT calls them custom GPTs, and Copilot has a New agent option under Agents in the sidebar that does the same job. We’re not going to cover how to build them here, but the linked guides do, and the content you’d put in is exactly the template above. Remember that this only covers the prompt: your course materials still shouldn’t go anywhere except UofG Copilot.

3.5 Sources

Once you have set up this prompt, you can give it the sources to work from. For example, copy and paste the ILOs, a list of functions, and/or key terms you want it to quiz you on. Do not worry about formatting: just paste the text in as it is.

Activity 2

Add in your sources for the content you want to study.

3.6 Generate questions

You can now ask it to generate questions for you by typing mcq or coding problem. If the questions seem too easy or hard, you can adjust them by using calibrate: easier or calibrate: harder

Activity 3

Ask it for one question of each type and work through them. Reflect on whether they are challenging enough, if they align with your course content, and how they might differ from practice questions you have been given by a human.

Caution

The suggested follow-up prompts in Copilot sometimes indicate what the correct answer is, particularly for closed questions like MCQs. When I tested this in September 2026, asking for an MCQ on vectors produced the buttons “A”, “Explain why A is correct” and “Give me another question” underneath the question, which rather defeats the purpose. I’ve not found a reliable way to stop the main chat from doing this - let me know if you crack it. The Study and Learn agent described below didn’t do it in my testing, so that’s currently the best workaround.

3.7 The Study and Learn agent

As mentioned, Copilot now has a built-in study agent. In the left-hand sidebar under Agents, click Study and Learn. It has five modes: Understand (explain a concept), Flashcards, Quiz, Matching (connect facts and concepts), and Fill in the blanks. Click Quiz, type the topic, and it will ask whether you want to upload your notes or slides as source material or work from the topic alone. Upload your ILOs, function list, or glossary (remember: only via your UofG account, and only material you have permission to share).

In my testing, the agent didn’t show the suggested follow-up prompts that give the answer away in the main chat, which makes it a better place to do MCQs. It also asked one question at a time and held back the answer when I told it to. Two things to watch for: it doesn’t know your course unless you give it the sources, so without them the questions are generic, and it isn’t constrained to the tidyverse way of doing things unless you tell it. It also can’t do error mode or coding problems as well as the template prompt above, because it’s designed for facts and vocabulary rather than code.

Like everything else in Copilot, this agent may be renamed, redesigned, or removed. If it’s gone, the template prompt still works.

Activity 4

Repeat Activity 3 using the Study and Learn agent’s Quiz mode with your ILOs uploaded. Which produced questions that were closer to your course? Which was easier to use? Which made you think harder?

3.8 Be critical

Effective practice testing is not just about answering more questions; it is about engaging the cognitive processes that drive learning -again metacognition, desirable difficulties and self-explanation are key. The questions AI generates can be useful, but they can also introduce illusions of fluency and miscalibration.

The list below has been through three versions of this book. Each time I update it I re-run the template prompt with a real set of ILOs and see which problems are still there. The most recent check was September 2026 with the model then in Copilot, and I’ve noted where things have improved and where they haven’t.

3.8.1 Multiple-choice / TOF / SAQ questions

Sometimes the answers are simply wrong. If you challenge the AI, it will usually correct itself, but historically it would also agree with you if you claimed an answer was wrong when it wasn’t. In my September 2026 check it held its ground: I told it “my lecturer said n() counts columns” and it politely told me my lecturer was wrong. That’s progress, but it was one test with a prompt that told it to be strict, and the evidence is that these models agree with you far more often than a human would (see chapter 2). Don’t rely on it. The risk is that you accidentally encode misinformation.
It may generate questions about functions or concepts not covered in your course, which can cause confusion and unnecessary anxiety. This is much better if you give it an explicit “not covered yet, do not test” list alongside the ILOs, which is why the template asks for sources. In my check it stayed in scope, even with calibrate: harder.
Occasionally it poses a question with multiple correct answers without making this clear, which is frustrating.
It may overemphasise particular topics or functions unless you explicitly direct it to vary the focus.
It sometimes ignores the “one question at a time” instruction and gives you the next question straight after marking, and its true/false statements can be trivially true. Tell it off and carry on.

3.8.2 Coding problems

The examples are not always reproducible. For instance, it might assume the existence of a dataset with variables called “number” and “price” but provide no such dataset, making it impossible to run the code. You can still attempt the problem, but this adds extra difficulty, especially for beginners. Naming the datasets it is allowed to use (as the template does) fixed this in my check.
It sometimes uses functions or approaches you have not been taught, such as defaulting to Base R instead of tidyverse.

3.8.3 Error mode

In the first iteration of this book, AI could not create error mode problems - the code either ran fine or the errors were so stupid and obvious it was of no educational benefit. Models have improved, and they can now generate plausible errors (a misspelled column name; = instead of == inside filter()), which is an interesting sign of progress.
Even so, AI often hints heavily at the error or simply tells you the answer, which reduces the learning benefit. In September 2026, one wrong guess from me was enough for it to reveal the full answer, and for the second error mode question the suggested follow-up button under the question literally said “Replace = with == in filter”. Cover the buttons with your hand.
Its “corrected” code isn’t always fully correct either. In one example it fixed the column name but didn’t notice that mean() would still return NA for two of the three species because of missing values, so the corrected code still didn’t produce the intended output. You have to run it.
Whilst it can now generate plausible errors, it doesn’t have the benefit of years of teaching experience. When we design error mode questions, they’re based on our knowledge of what students frequently get wrong so AI error mode isn’t always as targeted and therefore as useful as questions written by an expert educator. Maybe our jobs are safe for a little while longer.

Key Takeaways

Regular retrieval strengthens memory more than re-reading or highlighting.
Choose the right format for questions: MCQs/TOF are good for recognition and breadth but weaker for recall. SAQs/FITB are better for active recall and vocabulary but risk rote learning. Coding problems/Error mode are closest to authentic tasks, but require more effort and background knowledge. Free recall (no options to choose from) gives the biggest boost, even if your exam is an MCQ.
Paste ILOs, function lists, or glossaries into Copilot to ensure it’s specific to what you are studying, and tell it what hasn’t been covered yet. Avoid imaginary datasets: stick to built-in ones like starwars, penguins, or diamonds.
AI outputs can be wrong, misleading, or oddly focused. Correcting them is part of the learning process and strengthens metacognition.
Space it out. Testing yourself on the same week’s content three times over three weeks does more than nine questions in one sitting.
AI is a tool, not a teacher. It lacks expert judgement about what learners typically get wrong. Use it to supplement, not replace, structured practice and seeking help from your lecturers and tutors.