How Adelaide's Digital Archives Ended Up Full of Duplicate Images — and What Happens Next

South Australian cultural institutions and government agencies are sitting on enormous backlogs of duplicate digital images — copies stacked on copies across ageing servers — after years of rushed digitisation projects outpaced the policies meant to govern them. The problem is not new, but pressure to rationalise public sector IT spending in the 2025–26 budget cycle has forced the issue onto agency agendas that had long deferred it.

The reasons this is landing hard right now are straightforward. Adelaide's digital infrastructure footprint expanded dramatically over roughly a decade, driven by overlapping programs: the Lot Fourteen innovation precinct on North Terrace brought dozens of start-ups and government-backed ventures onto shared or adjacent systems; AUKUS-related defence contractors establishing Adelaide offices imported their own document management standards, frequently incompatible with state systems; and the hydrogen jobs plan rollout generated a secondary wave of project documentation, tender records and promotional imagery that multiple agencies stored independently. Each program created its own siloed archive. Nobody reconciled them.

How the Backlog Built Up

The State Library of South Australia on North Terrace, one of the country's oldest continuous collecting institutions, completed a major digitisation push of its photographic holdings between roughly 2018 and 2023. The History Trust of South Australia, headquartered at the Migration Museum on Kintore Avenue, ran a parallel program. So did the Art Gallery of South Australia, also on North Terrace. Each institution used different metadata schemas, different file-naming conventions and, critically, different thresholds for what counted as a "master" image versus a working copy. When state government ICT consolidation efforts attempted to migrate holdings onto common platforms, duplicates multiplied rather than collapsed.

The issue is not purely administrative. Storage costs money. Cloud storage contracts negotiated by the Department for Innovation and Skills — which oversees parts of the Lot Fourteen precinct — have grown substantially as redundant file counts ballooned. Industry estimates for enterprise cloud image storage in Australia put costs at roughly $20 to $40 per terabyte per month depending on the vendor tier, and institutions dealing in high-resolution archival scans accumulate terabytes quickly. A single digitised glass-plate negative at archival resolution can run to several hundred megabytes. Multiply that across collections numbering in the tens of thousands and the arithmetic becomes uncomfortable fast.

Interstate migration into Adelaide — the city has recorded consecutive years of net positive internal migration — has added an unexpected dimension. New residents arriving from Sydney and Melbourne have engaged with state digital collections at higher rates than local baseline figures, according to patterns libraries have observed in their access logs. That uptick in demand exposed search and retrieval failures that duplicate records cause: searches returning the same image four or five times, or worse, returning degraded working copies rather than the authoritative master file.

What Remediation Actually Looks Like

Fixing the problem involves three distinct phases that institutions have described in planning documents and public tender notices over the past eighteen months. First, automated deduplication tools scan holdings and flag probable matches using perceptual hashing — a technique that identifies visually identical or near-identical images even when file names differ. Second, human review confirms matches and designates a single master. Third, metadata is standardised and migrated to a canonical record, with superseded copies deleted or archived to cold storage.

The Australian Institute for the Conservation of Cultural Material has produced guidance on deduplication workflows relevant to collecting institutions, and several Adelaide organisations have drawn on that framework. The process is slow. Institutions doing it properly reckon on reviewing hundreds of images per staff day, meaning collections in the tens of thousands represent months of dedicated work.

For agencies tied into the broader Lot Fourteen ecosystem — where the Australian Space Agency and various defence-adjacent tech firms share physical and increasingly digital infrastructure — the stakes are higher than archival tidiness. Duplicate or mis-catalogued imagery in project documentation can create compliance headaches under federal record-keeping obligations. The practical advice circulating among Adelaide's ICT community is blunt: start the deduplication audit before the next system migration, not after. The cost of fixing it downstream is almost always higher than the cost of doing it now.