[R] Amazing AI

[Boris Steipe]

Exactly. But not just "error prone", rather: eloquently and confidently incorrect. And that in itself is a problem. When I evaluate students' work, I implicitly do so from a mental model of the student - aptitude, ability, experience, language skills etc. That's useful for summative assessment, since it helps efficiency - but that won't work anymore. I see a need to assess much more carefully, require fine-grained referencing, check every single fact ... and that won't scale. And then there is also the spectre of having to decide when this crosses the line to "concoction" - i.e. an actual academic offence ...

Best,
Boris



> On 2022-12-19, at 03:58, Milan Glacier <news using milanglacier.com> wrote:
> 
> [You don't often get email from news using milanglacier.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> On 12/18/22 19:01, Boris Steipe wrote:
>> Technically not a help question. But crucial to be aware of, especially for those of us in academia, or otherwise teaching R. I am not aware of a suitable alternate forum. If this does not interest you, please simply ignore - I already know that this may be somewhat OT.
>> 
>> Thanks.
>> ------------------------------------------------------
>> 
>> You very likely have heard of ChatGPT, the conversation interface on top of the GPT-3 large language model and that it can generate code. I thought it doesn't do R - I was wrong. Here is a little experiment:
>> Note that the strategy is quite different (e.g using %in%, not duplicated() ), the interpretation of "last variable" is technically correct but not what I had in mind (ChatGPT got that right though).
>> 
>> 
>> Changing my prompts slightly resulted it going for a dplyr solution instead, complete with %>% idioms etc ... again, syntactically correct but not giving me the fully correct results.
>> 
>> ------------------------------------------------------
>> 
>> Bottom line: The AI's ability to translate natural language instructions into code is astounding. Errors the AI makes are subtle and probably not easy to fix if you don't already know what you are doing. But the way that this can be "confidently incorrect" and plausible makes it nearly impossible to detect unless you actually run the code (you may have noticed that when you read the code).
>> 
>> Will our students use it? Absolutely.
>> 
>> Will they successfully cheat with it? That depends on the assignment. We probably need to _encourage_ them to use it rather than sanction - but require them to attribute the AI, document prompts, and identify their own, additional contributions.
>> 
>> Will it help them learn? When you are aware of the issues, it may be quite useful. It may be especially useful to teach them to specify their code carefully and completely, and to ask questions in the right way. Test cases are crucial.
>> 
>> How will it affect what we do as instructors? I don't know. Really.
>> 
>> And the future? I am not pleased to extrapolate to a job market in which they compete with knowledge workers who work 24/7 without benefits, vacation pay, or even a salary. They'll need to rethink the value of their investment in an academic education. We'll need to rethink what we do to provide value above and beyond what AI's can do. (Nb. all of the arguments I hear about why humans will always be better etc. are easily debunked, but that's even more OT :-)
>> 
>> --------------------------------------------------------
>> 
>> If you have thoughts to share how your institution is thinking about academic integrity in this situation, or creative ideas how to integrate this into teaching, I'd love to hear from you.
> 
> *NEVER* let the AI misleading the students! ChatGPT gives you seemingly
> sound but actually *wrong* code!
> 
> ChatGPT never understands the formal abstraction behind the code, it
> just understands the shallow text pattern (and the syntax rules) in the
> code. And it often gives you the code that seemingly correct but indeed
> wrongly output. If it is used with code completion, then it is okay
> (just like github copilot), since the coder need to modify the code
> after getting the completion. But if you want to use ChatGPT for
> students to query information / writing code, it is error proning!