How Washington's Digital Archives Ended Up Flooded With Duplicate Images — And What's Being Done About It

Washington's public digital archives are carrying a significant redundancy problem. Across multiple District and federal repositories, the same photographs, scanned documents, and agency graphics appear dozens — sometimes hundreds — of times under different file names, folder structures, or metadata tags, a byproduct of years of poorly coordinated system migrations and a near-total absence of deduplication standards.

The problem matters now because the Trump administration's restructuring of the federal workforce, driven in part by the Department of Government Efficiency, has accelerated the consolidation of agency IT systems without providing the cleanup protocols that consolidation demands. When teams are reduced and databases are merged under deadline, duplicates multiply. The District government, meanwhile, is operating under its own budget pressures, with Mayor Muriel Bowser's administration managing a fiscal 2026 budget that already cut several technology modernization line items to offset reduced federal transfers.

How the Problem Built Up Over Decades

The roots go back further than the current administration. The DC Office of the Chief Technology Officer, headquartered at 200 I Street SE, has overseen at least four major platform migrations since 2005. Each migration — from legacy servers to cloud-hybrid systems — created opportunities for file duplication. Standard practice during those transitions was to copy entire directories rather than perform selective transfers, meaning every version of a photograph or scanned permit application moved with the archive, not just the most recent or highest-quality copy.

The National Archives and Records Administration, based in College Park, Maryland, faces a parallel challenge at federal scale. NARA has publicly documented the growth of its electronic records holdings, which by 2023 exceeded 1.5 billion logical data objects. Even a duplication rate of one or two percent across a collection that size represents tens of millions of redundant files consuming storage capacity and degrading search results for researchers using the Archives' online catalog.

Within the District itself, the DC Public Library system's Washingtoniana Division — which maintains digitized collections of city photographs, newspapers, and maps at the Martin Luther King Jr. Memorial Library on G Street NW — has grappled with the same issue at a more manageable scale. Librarians there have noted that donations of digitized family and neighborhood collections, particularly from Anacostia and NoMa community groups, often arrive with extensive internal duplication because donors scan from multiple sources without cross-referencing.

The DOGE Factor and What Comes Next

The current federal restructuring has added a new layer of urgency. As agencies shed staff and consolidate IT functions, the institutional knowledge needed to clean up legacy data is often lost with departing employees. A records management specialist who knew which of three versions of a 2019 agency photograph was the authoritative file takes that knowledge when they leave. What remains is a storage bill and a search index that returns the same image three times.

Commercial deduplication software has existed for years — tools capable of scanning large file directories and flagging bit-for-bit or perceptual duplicates. The challenge for public institutions is procurement. A mid-tier enterprise deduplication license can run between $15,000 and $80,000 annually depending on repository size, a real constraint for agencies working under continuing resolutions or Bowser administration budget freezes.

The practical path forward involves three steps that archivists and records managers broadly agree on: conducting a full audit of existing holdings before any further migration, adopting a single metadata standard across all ingestion pipelines, and assigning at least one dedicated records hygiene role per major repository. For District residents who depend on public records access — whether journalists filing Freedom of Information requests, Anacostia homeowners researching property histories, or developers pulling permit records near the NoMa Metro station — a cleaner archive means faster, more reliable results.

The Fourth of July holiday has paused most federal offices today, but the duplicate files are still there, sitting in servers in College Park and on I Street SE, waiting for someone to decide they're worth cleaning up.