The AI tool Consult built by the UK government sped up the analysis of over 50,000 responses to the Independent Water Commission review in 2 hours – and helped to speed up the government’s decision to abolish Ofwat.

A joint announcement by the Department for Science, Innovation and Technology and the Department for Environment, Food & Rural Affairs says the tool developed to analyse the responses to the government-commissioned review of the water sector. was found to be at least as accurate and reliable as humans.

The simple task of sorting over 50,000 responses into key themes made the Independent Water Commission (IWC) analysis more efficient and effective. The AI tool categorised responses into themes in around 2 hours, costing £240 and experts only needed 22 hours to check the results

It meant policy experts could focus on using themes and categorised responses to inform recommendations for their independent report, rather than sorting tens of thousands of individual responses.

Alongside the AI-assisted thematic analysis, the team also completed detailed manual reviews of responses from stakeholders to ensure their perspectives were thoroughly considered.

The work of ‘Consult’ was compared to 2 groups of experts. It agreed with one or both of the groups almost 83% of the time, while the 2 well-practiced human groups only agreed with each other 55% of the time.

“Consult’ is part of ‘Humphrey’, a bundle of AI tools designed to speed up the work of civil servants and cut back time spent on admin and money spent on contractors.

The Government-developed “Humphrey” AI tool was used successfully for the first time earlier this year to support the analysis of the Scottish government’s live consultation on non-surgical cosmetics.

Going forward, the Consult technology will be used to analyse other consultations’ responses. It forms part of the government’s plan to make better use of technology across public services, in a bid to target the £45 billion in productivity savings that it offers

Digital Government Minister Ian Murray said:

“This shows the huge potential for technology and AI to deliver better and more efficient public services for the public and provide better value for the taxpayer.

“By taking on the basic admin, Consult is giving staff time to focus on what matters – taking action to fix public services. In the process, it could save the taxpayer hundreds of thousands of pounds.”

Another tool in the ‘Humphrey’ suite, called ‘Redbox’, helped 5,330 officials at its peak work more efficiently – with the technology helping them to summarise long documents, draft briefing notes and more.

Since it was introduced, major tech companies have started to provide tools that give officials a secure way to use large language models that are integrated into IT systems they are already using, for example, Microsoft Copilot. Often, these come as part of existing software deals between the government and technology companies.

A recent trial of Microsoft Copilot found that the technology could save officials 2 weeks every year. As a result, engineers in the team are developing new tools, such as those identified by the Prime Minister as ‘AI Exemplars’ which aim to speed up planning decisions to help build homes.

As a result, development on Redbox will not continue, though it has now been open-sourced. The engineers that built the tool have gone on to use their knowledge to build other technology in the ‘Humphrey’ suite and also shared information that was used to build GOV.UK Chat, the generative AI chatbot that will soon be trialled in the GOV.UK App.

The evaluation of Consult on the Independent Water Commission call for views shows that it secured an F1 score (a common measure of alignment for AI tools) of 0.79 and 0.82. This is higher than the F1 score between human reviewers (0.74), and shows an increase from 0.76 when the technology was used on the Scottish government consultation, which received fewer responses (2,000).

According to the government, the Consult tool could ultimately help save 75,000 days of manual analysis which is currently slowing down policy action across government every year and save £20 million in staffing costs.

Click here to download the evaluation of Consult : Independent Water Commission Call for Evidence