The promise and the pitfall

AI tools have quickly found themselves embedded into industries across the globe and legal services are no different. From drafting clauses to summarising case law, they promise speed and efficiency at a fraction of the cost of traditional research methods. In a world always hungry for more, AI promises to help teams deliver more with less, the appeal is obvious.

But there's a growing problem that I'm sure we've all seen in our encounters with AI: it's eagerness to please and the confidence in doing so.

AI can generate text that looks convincing, sound authoritative and even cite case law, except when those references turn out to be entirely fabricated. It's worth remembering that AI isn't actually 'thinking' or 'researching' – it's essentially sophisticated predictive text that generates what seems like the most likely next word based on patterns in its training data.

This isn't a fringe concern. Courts have already sanctioned lawyers for submitting filings supported by non-existent cases. For businesses, the reputational and regulatory risks are just as real.

This article looks at why AI can't replace human judgement, what responsible use should look like, and how in-house teams can protect themselves while still taking advantage of the technology.

Hallucinations in black and white

The risk with AI is that it's designed to please and as such will often take your side in matters, and it will do whatever it takes, even if that means fabricating matters. Entire cases, citations and precedents can appear in an output, dressed up with convincing detail and written in the language of authority.

In recent months, UK courts have issued stern warnings over lawyers submitting filings containing AI-generated fictitious case-law. In one High Court matter concerning the Qatar National Bank, a claimant's filing cited 45 cases and 18 of which turned out to be entirely fabricated. In a separate judicial review involving Haringey Law Centre, a barrister relied on five phantom cases that never even existed.

These aren't isolated incidents either and judges have emphasised that any failure to verify sources before submission could be contempt of court or even perverting the course of justice. Beyond the immediate legal consequences, these cases also raise serious questions about professional conduct obligations. Lawyers have fundamental duties to act with reasonable skill and care that extend to supervising and verifying any tools they use, including AI. Reckless use also risks bringing the profession into disrepute – a specific professional conduct concern that can lead to disciplinary action.

Despite these serious risks, the allure for legal teams under pressure is understandable, AI can produce text that reads convincingly authoritative, and in the right hands it's a very powerful tool.

These cases underscore a critical lesson: it's not to avoid AI but instead to still do your due diligence and verify AI-generated material, otherwise they risk exposing themselves and their organisation to ethical, professional and reputational risk.

Responsible use in practice

AI can still play a role in legal work, but as support not as the team leader. Using it unchecked as a shortcut to finished advice is where the risks appear.

The practical safeguards are straightforward:

• Verify everything: citations, case references and quotations must always be checked against original sources.
• Use AI to help you draft: outputs should be red-lined, annotated and reworked before being shared.
• Protect confidentiality: ensure AI tools don't compromise client privilege or confidentiality. Be cautious about what information you input into AI systems. Remember that anything you input into AI systems is likely used to train the software and could appear in future outputs for other users, making confidential information effectively public.
• Be transparent internally: make clear what has and hasn't been human-verified, so colleagues understand the limits of what they're reading. Operating incognito will only catch you out at the worst possible moment.

When used appropriately, AI can be genuinely helpful. I've seen teams use it effectively for initial contract clause drafting, summarising lengthy disclosure bundles or documents, and generating first-draft research notes. The key is treating these as starting points, not finished products.

If you use AI to help fill in the gaps and speed up your processes i.e. for 20% of the work instead of 100%, you'll free up time without eroding standards. It can support first drafts, help summarise large volumes of information, or test alternative phrasings.

Building the right framework

The best way to enforce these practices is through an internal policy. It doesn't need to be lengthy, but it should set out where AI is encouraged (drafting, summarising, brainstorming), where it's prohibited (legal opinions, final advice, client-facing documents and court submissions), and what checks are mandatory before anything is relied upon.

An effective AI policy should include:

• Clear boundaries: specify which tasks are appropriate for AI assistance and which require human-only work.
• Approval processes: determine who can authorise AI use for different types of work.
• Review procedures: establish how AI-generated content must be checked and by whom.
• Confidentiality and security protocols: ensure client information is protected when using AI tools, including data handling procedures, encryption requirements, and restrictions on what information can be inputted into AI systems.
• Documentation requirements: record when and how AI has been used in particular matters.
• Regular review: update the policy as technology and best practices evolve.

It is critical that you strike a balance of guided usage, rather than defaulting to outright prohibition on one hand, or unmonitored free-for-all on the other. Both extremes create risk: either missing out on efficiency gains or exposing the business to reputational damage. A clear framework allows teams to explore AI's benefits while keeping control of how it is applied.

The reality is simple: people will use AI whatever you say. Without clear guidelines, that use goes underground, only to surface at the worst possible moment. A transparent framework avoids this, encouraging responsible experimentation while keeping control of how and where AI is applied.

Legal teams should also prepare for courts requiring disclosure of AI use. Some jurisdictions already require lawyers to tell courts when AI was used in preparing submissions, a trend likely to spread as judges seek transparency about how cases are being prepared.

Handled this way, AI becomes part of the toolkit without undermining accountability. It gives teams confidence to use the technology, while showing boards, clients and regulators that standards of diligence are being maintained.

In the end, the message is very simple AI can support good legal work, but it cannot replace sound legal judgement.