7  Coding with AI

Intended Learning Outcomes

7.1 Here we go again

So far we’ve been slowly building up your use of AI but we’ve very consciously stayed away from allowing or encouraging you to use AI to write your code. But it’s now time, so for this chapter, we’re going to do something a bit different.

Rather than giving you new exercises or content, your task is seemingly very simple: redo the backwards engineering report from Chapter 6, but this time, AI must write all of the code.

The rationale behind this is that having worked through the report yourself, you will be intimately familiar with the dataset and the requirements of the assignment. This means that your ability to critically evaluate and check its output will be stronger than with a new dataset. This should (in theory) allow you to:

  • Check whether it’s actually done it right
  • Understand how to engineer your prompts to get it closer to the solution if it’s struggling
  • Evaluate the suitability of alternative approaches to your original solution.

However, you will also do this as a peer coding exercise designed to expose why GenAI is not the friend of reproducibility: two people can ask for “the same thing” and still get meaningfully different code, explanations, and levels of correctness, purely because of differences in prompt wording, assumptions, and constraints.

7.2 Starting advice

A few hints, tip, and rules to get you on your way:

  • Although the dataset for the backwards report is publicly available data, treat it as if it were strictly private and confidential. Even if the AI has the capability, don’t upload the dataset. Instead run str() and summary() and describe the dataset to the AI so it knows the variables and the types of data.
  • You can upload the rendered html report if you want although describing each bit manually will be more educational in the long-run (and more likely to generalise to when you don’t have anything to backwards engineer from).
  • Check everything it produces. Just because it doesn’t throw an error, doesn’t mean it’s right. Check the values it produces against your original solution.
  • Think about the specificity of your prompts. If you want it to produce solutions closer to our solution, tell it to use the tidyverse. If you’d like to see what other approaches it might take, don’t give it a steer on packages (be mindful you might need to install some new packages).
TipActivity
  1. Each person should create a separate Qmd file from your original backwards report Qmd to avoid any confusion.
  2. For each step of the report, agree on what you want to achieve with the AI, e.g., “at this step, we’ll load in the data, rename and clean the columns and set the factors”. To make this more meaningful, don’t make each step too simple (load in the data), but also don’t assign too many tasks to each step (code the whole report).
  3. Then, separately prompt your AI until you get it to do the agreed task(s).
  4. Next compare both of your prompts and the output it produced.
  5. For each stage, makes notes on the following:
  • Did your prompts differ from your partner? For example, was one of you more or less specific or provided longer prompts?
  • What did the AI get right with ease? What did the AI get right with a bit of back-and-forth? What errors did the AI make?
  • How does the code compare to your original solution? Is there anything you think it did better? Is there anything you think you did better? Did you learn any new functions or approaches?
  • If you didn’t know the dataset and hadn’t worked through the report yourself, is there anything you think you may have missed that the AI got wrong?
  • If you hadn’t worked through the report yourself and had just used AI for the first attempt, is there anything you learned during the process that you think you would have missed by using AI? This could be a specific function or it could be about self-efficacy or autonomy.
  1. Get as far as you can through the report by the end of the Tuesday workshop and then submit your log on Moodle.
  2. Additionally, submit your AI produced Qmd file and rendered html file on the Week 7 channel on Teams so that other students can see the varying outputs.