Adelaide's Digital Archivists Are Quietly Solving a Problem That's Stumping Cities From Rotterdam to Osaka

South Australia's cultural institutions are working through a backlog of tens of thousands of duplicated digital images clogging public-facing archives, and the methods being trialled at Lot Fourteen are drawing interest from counterpart organisations in Rotterdam, Osaka, and Vancouver. The problem is unglamorous but expensive: duplicate image files inflate storage costs, corrupt search results, and undermine the credibility of digital collections that governments have spent years and millions of dollars building.

The issue has sharpened in 2026 partly because of scale. State and territory governments across Australia accelerated digitisation programs during the pandemic years, bulk-scanning physical collections without the staff or software to catch redundant files on ingestion. The consequences are landing now, as institutions hit storage ceilings and prepare public-access upgrades. For Adelaide, the timing cuts close: the State Library of South Australia on North Terrace is mid-way through a redevelopment of its digital discovery portal, and the History Trust of South Australia, headquartered at the Migration Museum on Kintore Avenue, is merging several photographic collections into a single searchable platform.

What Adelaide Is Doing Differently

The approach being developed at Lot Fourteen, the former Royal Adelaide Hospital site on North Terrace repurposed as a technology and innovation precinct, centres on perceptual hashing, a technique that fingerprints images by visual content rather than file metadata. Unlike simple file-size or filename matching, perceptual hashing catches duplicates that have been rescanned, recoloured, or saved in different formats. The Australian Institute for Machine Learning, based at Lot Fourteen, has been involved in adapting open-source hashing tools for archival use, though the specific contractual arrangements with state cultural bodies have not been publicly disclosed.

This contrasts with how Rotterdam's city archive and Osaka's municipal digital library have approached the same problem. Rotterdam's Stadsarchief undertook a manual curation drive in 2024, dedicating staff time to reviewing flagged files individually, a method archivists there have described publicly as unsustainable at volume. Osaka's approach leaned on vendor-supplied deduplication software bundled with a storage contract, which critics in the Japanese library science community have noted creates long-term vendor dependency. Vancouver Public Library launched an open-tender process in early 2025 for a bespoke deduplication layer, with results yet to be published.

Adelaide's hybrid, algorithmic detection followed by human curatorial sign-off, costs more upfront than a pure software solution but is designed to preserve the contextual judgement that archivists argue a machine cannot reliably make. Two images that are visually near-identical may have entirely different provenance, captions, or copyright status.

The Numbers Behind the Mess

The scale of the problem in South Australian collections is not trivial. The State Library's digital repository held more than 1.2 million image files as of its last published annual report. Industry benchmarks from the Digital Preservation Coalition, a UK-headquartered international body, suggest duplication rates in rapidly grown public archives typically run between eight and fifteen percent, which, at the lower end of that range, would point to roughly 96,000 potentially redundant files in the State Library collection alone. Storage costs for cultural institutions on AWS or Azure infrastructure currently run at around AU$23 to AU$28 per terabyte per month for standard access tiers, according to published cloud pricing schedules, meaning even modest deduplication delivers measurable savings.

The History Trust's Kintore Avenue site is contending with a specific complication: three separately donated photographic collections were ingested using different metadata schemas between 2019 and 2023, meaning duplicates don't always share consistent tags or identifiers. Staff there are understood to be cross-referencing records manually before any automated pass is applied, a painstaking process.

For other institutions watching from elsewhere, the Adelaide case will be instructive precisely because the city is mid-project, not post-completion. The State Library's upgraded portal is scheduled to go live in the second half of 2026. If the deduplication layer holds under public search load, it will give credibility to the hybrid model at a moment when cities like Rotterdam are reconsidering their purely manual approach. If it stumbles, it will add weight to the argument that the vendor-bundled route, for all its drawbacks, at least delivers a defined outcome on a fixed timeline.