Cabinet Office taps data analytics firm for ‘spring clean’ of millions of files

Written by Sam Trendall on 21 February 2023 in News

Department extends existing supplier engagement in £500k deal

Credit: tookapic/Pixabay

The Cabinet Office has signed a £500,000 deal with to expand the department’s use of automated data analytics to support “spring clean” exercises in which millions of files will be sifted – and, where applicable, deleted.

Freshly released commercial documents indicate that, two years ago, the department “decommissioned a legacy IT system called Apollo… [which] contained significant volumes of unstructured and semi-structured data” – amounting to about 4TB, across 11 million individual files held in 170 differing formats.

Initial analysis suggested “there were significant volumes of ROT – redundant, outdated, trivial (ROT) – [information] as well as historically valuable information that must be retained by the Cabinet Office for long-term preservation”. 

On 1 February 2021, the department signed a contract with Belfast-based data analytics firm Automated Intelligence. In a £124,000-a-year deal that ran until 31 January 2023, the firm’s AI.Datalift technology has been used by the Cabinet Office as the foundation for developing an “automated decision-making tool… [that] was piloted on the legacy Apollo data set”.  

“The tool has proven to deliver accurate and reliable disposal decisions, aiding digital archivists to greatly reduce the volume of ROT held by the Cabinet Office for this data set,” according to newly published procurement documents.

Related content

Senior executives on the department’s People and Operations Committee last year approved an expansion of the automated analysis tool for use on other data sets held by the government's central agency.

The Cabinet Office has thus awarded a new two-year deal to Automated Intelligence, running directly on from the conclusion of the previous deal on 31 January. This contract will be worth £236,500 a year plus VAT – almost double the value of the prior engagement.

The contract outlines that the firm is expected to provide “data analytics tool that is programmable so that data can be classified as either ROT or as valuable… [and] that is able to action a deletion on data so designated”.

The system developed using the company’s technology will now be used to support annual “spring clean” projects in which millions more files will be analysed.

These exercises – led by the Cabinet Office’s Digital Knowledge and Information Management (DKIM) team – are designed to sort through digital documents and categorise them for either deletion or retention.

This process has created an accumulation of “legacy information awaiting a disposal decision”.

“DKIM currently holds 4.9million digital files in its ‘holding pens’ which is made up of information collated from across Cabinet Office through the annual DKIM-led spring clean process, an annual muster of records identified for disposal and retention according to Cabinet Office policy,” the text of the new contract says.

Automation technology will be used to further analyse files that are set to be retained, in order to identify and remove any remaining redundant, outdated, and trivial data.

Having completed sifting of this legacy data, the system will be used to “analyse information collected through the DKIM spring clean in future years, currently estimated to be 350,000 documents per annum”.

The deal with Automated Intelligence will see the company work with the Cabinet Office to “explore the use of the tool to support business units to identify information in scope for future spring clean activity – [which] is currently a resource intensive, manual process”, the contract says.


About the author

Sam Trendall is editor of PublicTechnology. He can be reached on

Share this page




Please login to post a comment or register for a free account.

Related Articles

DWP, Home Office, MoJ and Defra launch £1bn tender for shared services tech providers
2 June 2023

Departments look to sign joint deal with a software provider and a system integrator

ONS seeks new data source on UK firms’ overseas owners
24 May 2023

Statistics agency looks to establish a single unified partnership

‘Very beneficial – or very dangerous’: MPs warn of urgent need for AI regulation
5 May 2023

Conservative and Labour MPs call for more focus from the top of government

CCS defends Digital Marketplace closure and pledges to ‘reinstate transparency’
3 May 2023

Suppliers and former officials have lamented the decision but procurement agency claims outgoing platform could no longer ‘accommodate growing demands’