Here’s a cautionary tale about some of the limitations of ChatGPT right now. If you ask it to generate a list of suggested articles or books you should go and read on a particular topic, you’re likely to get a list of made-up titles.
It would be my dream of a research sidekick. You’d want to get a summary of latest research on a topic, so you’d ask an AI tool such as ChatGPT to go and trawl the thousands of journals available and produce a list of hot articles. You could then use this as the basis of your initial reading, and then hit the reference lists to find more related articles.
So I tried it with ChatGPT last night. I asked it for some current articles on the teaching of primary science.
And here’s the list it suggested:
I had to ask it to add journal titles to the list. It would not provide any direct URL’s.
The reason why became clear when I tried to look for these articles. Google had no record of them. The EBSCO academic library had no record. I went direct to the journals themselves and there was no trace of them.
The articles didn’t exist.
Tools such as ChatGPT are aggregation tools, from what I can see. They take in large amounts of related content and remix them / mash them together into new content. This is fine if you’re asking it to create a story, an essay or a poem. It works great for a scheme of work or lesson plan. Create a summary of an article, no problem. It’s creating new from old.
So when it creates a reference list, it’s doing the same thing. A title from here, an author from there, a date and a journal name from somewhere else. And it mashes them together into a new entity. One that does not exist outside of ChatGPT.
Hopefully in the future, it’ll be trained not to do that for references, but for now, it’s an issue.
The video explains more;
If you’ve found a way around this, let me know. Or if you know of better tools for this kind of thing, share it in the comments.
