AI is transforming our lives, offering tools and capabilities that were barely imaginable just a few years ago. While we marvel at the new opportunities AI presents, we must also learn to question AI in ways we’re not used to.
For decades, we’ve tended to rely on the answers generated by computer programs, without much scrutiny. Applications developed and tested by humans often address fundamentally binary questions. When things go wrong, we blame a bug, something in the code (which was most likely written by a human) was incorrect, and once fixed we go back to trusting the computer program again. This belief is so strong that we sometimes follow computer advice even when it defies common sense.. Think about how often you hear of a driver blindly following instructions provided by their satnav and ending up in a river or worse.
With AI, especially Large Language Models (LLMs) like ChatGPT or Google Gemini, things are a little different. These models can make mistakes, get things wrong, or hallucinate in ways similar to humans.
Some research suggests that the hallucination rate for consumer grade LLM’s can be as much as 15% to 20%. We need to get better at questioning the responses they provide in much the same way as we might consider the advice of a friend or colleague.
When you log into Google Gemini or ChatGPT, you’ll see disclaimers about the accuracy of their responses prominently displayed at the bottom of the page. Think about other computer systems you use—do they display similar prominent warnings? They don’t; any provisos are usually in fine print (that you will never read). This highlights a stark difference between LLMs and other programs.
LLM’s often get things wrong, and sometimes they hallucinate, a term which has come to describe the AI making up information to fill in gaps in its knowledge. Unfortunately, LLM’s rarely know they’ve hallucinated, they ‘speak’ in the same loud and confident tone that all computers do, which means we’re unlikely we’ll question their accuracy.
I recently asked an LLM to list New Zealand companies that created AI-based products or services. The list included companies that I had heard of, as well as ones I hadn’t. The LLM included one company in the list that had used AI to assess insurance claims, speeding up the assessment process. However, when I went to search for the company I couldn’t find it, no search results and no website. I was reasonably sure the LLM had made it up and so I asked it if it hallucinated.
Somewhat sheepishly the LLM admitted that it had, (AI’s are good at recognizing errors once you point them out) telling me that it felt as though there should have been a smart insurance company using AI to assess claims.
Had I not questioned the results it would have been very easy to believe the list to be factually correct. I may have gone as far as putting that detail into a report or presentation to be shared more widely across my organization or even shared it with a client – a less than ideal outcome.
As LLM’s improve, instances of hallucinations will likey decrease. But for now, we need to treat LLM’s in the same way as we treat humans, question their responses and be cautious of their accuracy. We need to accept that sometimes they will be wrong even though they sound right. We need to do this whilst at the same time recognizing that AI and LLM’s are becoming more prevalent, embedded in more of the tools we use daily, even in applications like FME, where LLM can be used to write SQL statements based on data structures.
It’s an odd conceit, that the more human-like an AI appears, the more human-like our attitude towards its answer should be. In a way, this brings us closer to them.
About Locus
Locus, your trusted partner for data integration and transformation, helps you break down data silos and streamline the flow of critical information across your business. As the Platinum FME Reseller in Australia and New Zealand, we offer comprehensive support to the FME community, including technical expertise, professional services and training.
About FME
FME is a data integration platform that enables cohesive data transformation and exchange between diverse systems. Widely used across multiple industries, FME empowers organisations to unlock the value of their data and automate its flow across the enterprise driving innovation and informed decision making.
