AI in finance: Navigating the critical risks before realising the rewards

AI is transforming the finance industry, offering exciting opportunities and serious risks. As a follow up to the recent webinar on Gen AI, we are providing two articles: the first on the biggest risks of AI and the second on industry-specific use cases.

In this article, we delve into some of the risks of AI.

Risk 1: Hallucinations — how can AI hallucinate so confidently?

There is a widely reported case where a judge caught a lawyer providing case study references that did not exist. There are other examples in Australia where hallucinations exist, including the case of LJ Hooker.

So why does this happen? AI hallucinates because AI does not actually reason, instead, it predicts based on patterns. It generates text by predicting what words statistically follow others, not by fact-checking or understanding the physical world around us. So when it does not “know” something or has incomplete or conflicting data in its training, it guesses in a way that sounds plausible but can be totally false.

This fundamental nature leads to “hallucinations”: fabrications of data, cases or analysis. The now-infamous legal case, involving invented precedents, was not an anomaly. Precisely because AI sounds so confident, it erodes trust when the response is incorrect.

Given this is how large language models (LLMs) are set up, there is no 100% guaranteed way to remove hallucinations.

What can we do to reduce hallucinations?

The most basic way to reduce hallucinations is a review of the output by another person. Depending on the uses, AI is acting as a junior intern. For increased accuracy, a second AI should be used to check the AI output.

Risk 2: Data security — how do we create our own secured environment?

In addition to robust regulatory requirements for financial services, customers and the community expect our confidential data to be secured.

One practical application of AI is to create custom GPTs based on your own datasets. But how do we ensure the confidentiality of our information is protected, our privacy is respected and copyright laws are met?

There are a number of options available, depending on the use cases, your risk appetite, your client requirements and any global headquarters’ requirements that may apply.

The first step is to get a paid version of an AI tool, even for personal use. This provides you with the option to turn off training of their LLMs. This is the first action we recommend everyone takes — review your data security.

There are many Teams versions of general AI tools. We have seen many successful use cases built upon this approach, even with a small technology team in place.

Risk 3: How do we protect ourselves against deepfakes?

Beyond numerical hallucinations, AI’s ability to generate hyper-realistic text, images and voice — “deepfakes” — poses a systemic threat. Can you trust that analyst report, that CEO statement or that transaction record? The lack of reliable, industry-standard watermarking or provenance tracking creates fertile ground for fraud and market manipulation.

Our call to action is to review your controls for instances where deepfakes can impact your business. This could mean reversing some of the technology you have been using. For example, many banks have reversed the use of voice as a technology for phone banking.

Risk 4: What is AI’s broader impact on the environment and society?

The computational power driving advanced AI, particularly LLMs, is significant. The US exploring nuclear power for AI data centres underscores the scale. In Australia, amidst energy transition pressures and cost volatility, this presents a critical strategic question: do the projected efficiency gains from AI offset its significant, ongoing environmental footprint and operational cost?

The other environmental footprint consideration is the use of cooling for the generation of power. This has been discussed in recent articles in the AFR.

Other areas being debated include the potential for a universal income if the productivity gains come to fruition, and taxing the companies which gain from AI for fairer distribution of benefits.

The lack of visibility of these issues in industry and the media is likely to change as the bigger questions around the impact on society come to light.

AI is a technology where many aspects are being grappled with, and it is certainly one where governments should collaborate to achieve a better outcome.