How Adelaide's Digital Archive Crisis Reached Breaking Point: The Duplicate Image Problem Explained

South Australia's public sector and its growing constellation of tech-focused organisations are sitting on a sprawling mess of duplicated digital imagery — a problem decades in the making that has quietly ballooned into a storage, compliance and cost headache that can no longer be ignored.

The issue, broadly described as the duplicate image problem, refers to the accumulation of identical or near-identical digital files stored across multiple systems, servers and cloud platforms without any coordinated culling process. It sounds mundane. The financial and operational consequences are not.

How the Problem Grew

The roots run back to the early 2000s, when South Australian government agencies began digitalising records in earnest. Each department procured its own storage infrastructure, its own scanning workflows and its own file-naming conventions. Nobody was talking to anyone else. By the time the state government began consolidating digital operations under the Department for Industry, Innovation and Science frameworks around 2015, the duplication was already entrenched across entities from the State Library of South Australia on North Terrace to SA Health facilities in the outer suburbs.

The acceleration came with the Lot Fourteen precinct on North Terrace, which opened progressively from 2019 onward on the former Royal Adelaide Hospital site. As start-ups, defence technology firms and the Australian Space Agency's local operation moved in, they plugged into shared digital infrastructure and brought their own image libraries with them. Onboarding processes rarely screened for duplicate assets. The precinct now hosts more than 100 organisations, each generating visual content at scale.

The AUKUS submarine program added another layer. Local defence contractors in the Osborne Naval Shipyard corridor and around the Techport precinct north of Port Adelaide have been expanding their digital documentation requirements sharply since 2023, producing engineering diagrams, site photography and technical imagery that flows across multiple classified and unclassified repositories. Duplication in that context is not just wasteful — it creates version-control and security classification risks that programme auditors flag as unacceptable.

The Cost of Doing Nothing

Cloud storage is not free. Enterprise-grade cloud contracts for South Australian government entities are typically priced by the terabyte per month, and industry estimates — consistent with published AWS and Azure pricing structures in the Australian market as of mid-2026 — put per-terabyte monthly costs in the range of $25 to $45 for hot-access storage. When duplicated image sets run into the hundreds of terabytes across a large agency, the redundant spend compounds every billing cycle.

The State Records Act 1997 creates an additional pressure. Organisations subject to the Act are required to maintain accurate, non-redundant records that can be retrieved and authenticated. Legal and records management professionals have long argued that uncontrolled duplication undermines that obligation, because it becomes impossible to definitively establish which version of an image is the authoritative record. A 2024 review by the State Records office — whose findings were summarised in the agency's publicly released annual report — identified digital asset management as a priority area requiring policy clarification across agencies.

The hydrogen jobs plan, centred on the Whyalla steelworks site and associated manufacturing corridors, is generating its own documentation intensity as contractors photograph progress on infrastructure builds and upload files to multiple project management platforms simultaneously. Program managers have acknowledged internally that deduplication protocols were not written into early contractor briefs.

Practical solutions exist. Deduplication software — tools such as Rclone, or enterprise platforms marketed by vendors including Veritas and Commvault — can scan repositories, flag exact and perceptual duplicates, and either delete or quarantine them. Several Lot Fourteen tenants are understood to have piloted such tools in 2025. The State Library has been running structured digitisation workflows on the Kaurna Country heritage collections that include hash-based duplicate checks at the point of ingest since at least 2022.

The next step, according to records management practitioners, is for the state government to mandate deduplication standards in its digital procurement specifications — before the next wave of AUKUS contracts and Olympic Dam expansion documentation pushes the problem further beyond easy reach.