How Adelaide's Digital Archive Crisis Got Here: The Long Road to Duplicate Image Replacement

South Australian cultural institutions and government agencies are sitting on hundreds of thousands of duplicate digital image files, the product of more than a decade of rushed digitisation drives, incompatible legacy systems, and storage budgets that never kept pace with the volume of material being scanned. The problem is now acute enough that at least three major Adelaide-based organisations have begun formal duplicate-image replacement programs in the 2025–26 financial year.

The timing matters. The state government's investment in the Lot Fourteen innovation precinct on North Terrace — home to the Australian Space Agency, the SmartSat Cooperative Research Centre, and a growing cluster of defence and data firms — has put digital asset management under a spotlight it has never had before. When sophisticated technology tenants share infrastructure or collaborate on projects with legacy public institutions, the state of those institutions' data holdings becomes impossible to ignore.

How the Duplication Problem Accumulated

The roots go back to the early 2010s, when State Records SA and cultural bodies including History Trust of South Australia launched successive mass digitisation projects under federal and state grant funding. The goal was preservation — moving fragile physical collections into digital formats before further deterioration. The urgency was real. But the governance around file naming conventions, metadata standards, and deduplication protocols was not uniform across agencies. Different programs used different scanning contractors, different resolution standards, and different database software. Files were saved in multiple locations as a precaution against data loss, then never reconciled.

By 2019, the State Library of South Australia on North Terrace held an estimated several million individual digital image files across its collections infrastructure. Overlapping copies — created during migration from one storage system to another, or when items were rescanned after earlier scans were deemed inadequate — had become embedded in catalogues without clear flags identifying which version was authoritative. The problem was not unique to Adelaide, but the scale of the state's digitisation ambitions made it particularly pronounced here.

The arrival of AUKUS planning work from 2021 onward added a new layer. Defence contractors arriving in Adelaide — many now clustered around the Osborne Naval Shipyard in the city's northwest and the Edinburgh Parks precinct in the north — brought corporate data hygiene expectations that collided with the messier reality of state government digital archives. Procurement processes began surfacing questions about image provenance and file integrity that agencies struggled to answer cleanly.

What a 'Replacement' Program Actually Involves

Duplicate image replacement is not simply deleting spare copies. Done properly, it requires identifying which version of a file is the canonical record — highest resolution, most complete metadata, correct provenance tags — designating that as the master, updating all linked database references to point to the master, and then systematically retiring secondary copies while maintaining audit trails for accountability. For institutions covered by the State Records Act 1997, that last step requires formal disposal authorisation.

The process is resource-intensive. Industry estimates for large-scale deduplication projects in Australian public sector contexts typically run into hundreds of thousands of dollars when staff time, software licensing, and database remediation are included. Several Adelaide institutions explored open-source deduplication tools before concluding that their collections required commercial solutions with stronger metadata-handling capabilities.

The South Australian government's Digital Strategy, updated in late 2024, nominates data quality and asset management as priority areas for the 2025–2028 period. That policy framework has given agencies a formal basis to request capital funding for remediation work that previously fell into a gap between operational and infrastructure budgets.

For organisations now beginning replacement programs, the practical path forward involves a phased approach: audit first, establish master-record criteria second, automate identification of duplicates third, and execute retirement in batches with human sign-off at each stage. Institutions that have already moved significant collections onto cloud platforms — several have migrated to Australian data centre infrastructure since 2023 — find the process more tractable than those still relying on on-premises servers in ageing facilities. The window to get ahead of the problem, before AI-driven cataloguing tools make inconsistent metadata even more consequential, is narrowing fast.