Within three years, artificial intelligence technology fundamentally changed the evaluation environment. During this time, universities have taken a variety of approaches, ranging from banning the use of generator AI entirely to allowing AI by default, in some circumstances.
However, some university teachers and students report that they remain confused and unsure about the “proper use” of AI and what is important. This comes with concerns that AI is driving an increase in fraud.
Also, if AI is used in student assessments, there are broader questions about the value of a university degree today.
In a new journal article, we consider current approaches and evaluations to AI, asking:
Read more: Researchers created chatbots to help teach university law classes, but AI continues to ruin
Why “value of evaluation” is important
Universities are responding to the emergence of generator AIs with a variety of policies aimed at clarifying what is permitted and what is not allowed.
For example, the University of Leeds in the UK has set up a “traffic light” framework when AI tools can be used in assessments. Red means AI, orange allows for use, Green encourages it.
For example, the “red” light in a traditional essay shows students that it must be written without AI support. Amber's notable essay probably allows the use of AI in “idea generation” but does not write elements. “Green” lights allow students to use AI in their chosen way.
To ensure students comply with these rules, many institutions, such as the University of Melbourne, require students to declare their use of AI in a statement attached to assessments submitted to them.
The purpose of these and similar cases is to maintain “value validity.” This refers to whether the rating measures what we think we are measuring. Do you assess students' actual abilities and learning? Or how well do you use AI? Or how much did you pay to use it?
However, they argue that setting clear rules alone is not sufficient to maintain the validity of the assessment.
Our paper
The new peer-reviewed paper presents a conceptual discussion of how universities and schools can approach AI in their assessments.
We start by distinguishing between two approaches to AI and evaluation.
-
Discussional changes: Change only the evaluation instructions or rules. To work, they rely on students to understand directions and voluntarily follow.
-
Structural changes: Change the task itself. These constrain or enable behavior by design rather than by directives.
For example, telling students that “AI can be used to edit takeaway essays” is a debateful change. Changing the evaluation task to include a sequence of writing tasks within a class where development is observed over time is a structural change.
Telling students not to use AI tools when writing computer code is controversial. Developing live, evaluated conversations about the choices students made is structural.
It depends on changing rules
Our paper argues that most previous university responses (including traffic light frameworks and student declarations) are debate. They have changed rules regarding what is permitted or not. They have not changed the rating itself.
It is recommended that only structural changes can ensure that AI use protects the validity of the world, which means that rule breakdowns are increasingly irrelevant.
Therefore, you need to change the task
In the age of generator AI, structural changes are needed if we want assessments to be valid and fair.
Structural change means designing assessments that incorporate validity into the tasks themselves that are not outsourced to rules or student compliance.
This doesn't look the same in all areas and isn't easy. In some cases, students may need to be evaluated in a very different way than they did in the past. But you can't avoid the challenge just by telling your students what to do and hoping for the best.
If the assessment is to retain its function as a meaningful claim about student competence, it must be rethinked at the design level.
