Some thoughts for now on LLMs in university courses.
Do I know what I’m writing about?
<tldr>Maybe. I know about learning and tech.</tldr>
Some background. I’m a prof. I’ve taught programming and other tech stuff for a long time.
Many years ago, I decided for moral reasons to stop publishing, learn how brains learn, then devote the rest of my time to making learning products, workflows, and software needed to create and run good courses. Yes, I can do the publishing thing, e.g., one of my early papers was the fifth most cited paper in my field. No, your reasonable suspicion is wrong; it was a single-author paper.
I claim to know enough about learning to have useful opinions. I kinda know what knowledge is, how to model it, what experiences build it efficiently, and how to make such experiences. If you want, let’s have coffee, I’ll show you what I’ve made, and bore you about it.
I’m a software guy, too. I’m not an expert on LLMs, like my friend Mark Isken. I played with perceptrons (they are too a thing!) and neural nets in the ’90s, and am able to understand lay tech literature on LLMs. I’ve done a little work on expert systems, made forward-chaining production systems for computer-integrated manufacturing, and more recently made behavior graphs for autonomous agents. None of this is LLMs, though it is LLM adjacent.
Learning
<tldr>Learning cannot be made effortless, though it can be made more difficult than it should be.</tldr>
I’m not an AI fanboy nor a sabot-throwing curmudgeon. AI can harm learning. AI can help, too.
<rantlet>Some campus leaders are all-in on AI. It can do no wrong. That obviously makes no sense. Recommendation: nod and smile, then do what you want.</rantlet>
To grok, you need to know about learning. The context in this post is skill learning, where students learn to make artifacts. Programming, writing, analytics, graphic design, like that. Cognitive stuff, not physical, like hitting a three-pointer. That’s a sports thing, apparently.
You can think of knowledge as a state of head meat, modelable in principle by a mess of schemas, mental models, principles, facts, emotions, and – this is important – the connections between them. The more tightly bound schemas etc. are to each other and to world contexts, the more knowledgeable someone is, and the more skilled in their use.
Learning is change in knowledge through experiences, e.g., reading adds new facts. Experiences like writing code strengthen schemas and their connections. Formative feedback reinforces useful connections, and weakens connections that don’t match application realities.
Experiences are the only way knowledge gets into head meat. Experiences like doing tasks. Watching someone else do a task helps, too, especially when you’re new to a schema/mental model set. Just looking at an artifact someone else made isn’t as useful for a novice.
You: OK, where’s the AI?
Me: Patience. To work out the effect of one thing on another, you need to know the another first.
The more connections new info has to existing knowledge, the better.
<skippable example>
Say you’re making a web tech course teaching HTML and CSS. You could break the course into two parts: all the HTML stuff, then all the CSS stuff. Let’s call it the big-chunk course design method
That would be a mistake. Memory is about connections. Content without context is harder to remember.
In reality, HTML and CSS work together. For example, CSS uses the id and class element attributes in HTML to apply styles.
If you use the big-chunk design, you’d teach about HTML ids and classes, but students couldn’t hook them into their existing knowledge well until they got to CSS. They don’t see what those attributes are for.
Use a sliced-course design instead. Take a task, like making a webpage that can be created with simple HTML and simple CSS. Teach just that HTML and CSS together, the minimum needed for the task. Students have a task context, they learn how HTML and CSS hook up to get the task done, and they remember what classes and ids are. You’ve shown the connections.
Then make another example task using a few more HTML tags and a few more CSS properties. They hook into students’ memories for the first task, as well as hooking into each other, and task goals. Keep adding one task after another. Each one is a yummy course slice.
Notice both course designs cover the same content. However, students learn more from the sliced design. Their final knowledge state contains deeper connections, so they will be more skillful.
</skippable example>
You: Can we get to the AI now?
Me: Yes.
You: About gaaahing time.
AI and learning experiences
The question “Is AI good for learning?” isn’t useful. Better: “How can we use AI to help learning, and how can we prevent harm?”
Learning is all about experiences. Let’s watch a student’s experience.
You: Hey! This is more learning stuff!
Me: Yeah. Sorry, not sorry.
Sara learns
<tldr>Give students prompts to copy and paste. Prompts can ask AIs to explain, give examples, and ask questions.</tldr>
Sara reads a lesson from a Python textbook, a good one with a sliced design. The lesson is early in the course. It’s computing the tip for a meal. Sara already knows about tips, so memory hooks are there for task goals.
She learns about the input-processing-output pattern: ask the user for the meal cost, processing to work out the tip and total, output to show them. A few examples give her a schema for a simple program.
Sara reads her first Python statement: how to get data from a user. A few more examples, and she starts abstracting, that is, making inferences from cases. She does a few multiple-choice questions about input statements. The MCQs are not about testing; they’re for learning.
Next, Sara learns about Pythonic calculations, the second part of the IPO pattern. More examples, more abstraction. She sees how computation statements use data from input statements, making more connections. Then more MCQs.
Sara reads about output statements, following the usual explain-examples-MCQs pattern for learning-content creation.
Then she does an exercise. A near-transfer task (similar to the one in the textbook) about computing Aussie rules football scores. Sara submits her work and gets formative feedback. She adjusts her abstractions if needed. She also gets a positive emotional hit, evidence she can learn this Python stuff.
AIing
What can AI do here? The most obvious is do the exercise for her. She pastes the task into ChatGeppetto, copies the solution code, and submits it.
No effort, no learning. She’s unable to use her learning for the next task, since there’s nothing to use. Learning is a chain. An experience, then another, then another, then another. A good course builds experiences on each other.
OK, that’s easy to judge. Sara foot-gunned herself. Bad use of AI. Throw the sabots.
What if Sara’s smarter than that? Let’s go back to that pattern: explain-examples-MCQs. AI could help with all of them.
- Don’t understand an explanation? Ask an AI.
- Need more examples? Ask an AI.
- Ask for some more MCQs.
Yay! All praise AI! Burn the sabots!
But wait, there are issues. Keep those sabots handy. Or footy, really.
AIs explaining
First, good explanations are context-specific, so they build on memory hooks. We’re using a sliced textbook. Python statements are introduced a few at a time. A good explanation would limit itself to that context. Geppetto wouldn’t know to do that. It might (probably will) add stuff Sara can’t use. It might confuse her even more.
Context is also cultural, in a general sense. Tutors and authors have mental models of students in their own head meat. They know what info students can connect to and build on. Like what metaphors to use. Tip: using a phone book as a metaphor for a set of large files doesn’t work so well anymore.
A second issue about explanations is what questions to ask. That matters a lot when using an LLM. Give them context. Maybe paste the textbook’s content into Google Jiminy and ask it to explain it another way. (Are all LLMs Pinocchio themed?)
So, for explanations, LLMs would help some of the time, make things worse occasionally.
Recommendation: teachers and textbook authors give explanation prompts students can copy and paste as needed. The teacher-authors would check the prompts’ quality ahead of time.
OK, yes, sabot lobber, you’re right, the same prompt gives different results each time. I’ll get to that. Keep your sabots on your feet.
AIs exampling
Examples are the second part of the explain-examples-MCQs pattern. LLMs should be able to help, but, again, they need the right prompts so they don’t include programming constructs that aren’t good for that point in the course. More copy-paste prompts, maybe.
MCQs? Same deal. You can ask Geppetto to quiz you, but you have to ask the right way to get instructionally useful questions.
A problem with exercises is students get stuck. That’s why I’m online a bunch of hours every week so students can get help.
Giving help has its own set of rules. Ask students questions until you understand their misunderstanding, if it’s not obvious (even when it is). Tailor your help right there on the spot. Be upbeat about students’ effort. I cheer them. Yes, really. I’ll pump my fist, shouting “Sara, Sara, Sara!”
One thing I avoid: giving them the answer. That vitiates the whole thing.
Using LLMs is tricky here. Students paste code, ask Jiminy why it doesn’t work. J. explains, maybe in a way they understand, maybe not, and gives them the answer.
That’s the downside of help with exercises.
The upside is less frustration. Some frustration is normal, even desirable. Taking a programming course without being frustrated… ain’t gonna happen. Still, there’s a point where frustration is a Bad Thing.
AIs questioning
This is about asking an AI for questions you can use to test yourself.
Same problem as with explanations. Questions should match the particular learning context. That constrains what should be asked, and what distractors are reasonable for MCQs.
I haven’t experimented with telling an AI to make questions fitting within strict parameters. Not sure how it would go.
Sara doubts
<tldr>Students should doubt AI responses.</tldr>
Sara should know explanations she gets from AI might be useful, might not be, and might confuse her further. They can be wrong, too. I had that recently when asking a programming question.
An important thing: if Sara doesn’t understand an AI explanation, she should not infer that she is the problem. That’s especially an issue with depressed people. Attributing failure to themselves rather than tools or circumstances is one of the characteristic cognitive distortions of depression.
Even when using a prompt from her textbook, Sara should be ready for poor explanations. Prompts tested by authors might not work later. That’s LLMs for you. Sometimes you get an LLeMon response.
Tell Sara that if she still doesn’t get it after asking the Sauronic eye, don’t stress. If the explanation generated for the pasted-in prompt is ick, don’t worry about it. The tech has its limits.
She should ask her teacher. They’ll be better at explaining things. Hopefully.
It might help Sara to know something about learning, how it hurts sometimes, that AIs don’t get context, how having an AI to do the work means she won’t learn. That one should be obvious.
That’s all, folks
So:
- Don’t be an AI fanboy or a sabot caster. Neither make sense. Though, when in doubt, sabots are fun to throw, especially when filled with chocolate pudding, or another similarly-colored substance.
- AI can help learners, and it can harm them.
- You can give learners copy-paste prompts you’ve tested.
- Give learners pinches of salt to take LLMs with.