DC's Archives and Museums Are Battling a Digital Duplicate Crisis, Here's What Officials and Experts Are Saying

Washington's cultural institutions are sitting on a problem that has quietly metastasized over the past decade: their digital collections are riddled with duplicate images, sometimes tens of thousands of near-identical files, straining storage budgets and making public access harder than it needs to be. The issue has moved from back-office headache to policy conversation, with library administrators, archivists, and technology specialists now publicly disagreeing over how aggressively to act.

The stakes are real. Digital storage is not free. The National Archives and Records Administration, headquartered on Pennsylvania Avenue NW, manages more than 100 petabytes of digital holdings across its facilities. Archivists and records managers familiar with large federal collections say duplicate image files, generated by bulk scanning projects, donor uploads, and repeated digitization of the same physical materials, can account for anywhere from 10 to 30 percent of raw storage load in poorly managed repositories, though precise figures for any specific institution require audits that most have not publicly released.

Why the Conversation Is Happening Now

Timing matters. The Trump administration's restructuring of the federal workforce, accelerated by the Department of Government Efficiency's cost-reduction push, has put every line item at agencies like the Smithsonian Institution and the Library of Congress under renewed scrutiny. Budget officers are asking questions that would have seemed bureaucratic a few years ago: What exactly is in storage, and does all of it need to be there? Duplicate image files, which offer no preservation value and impose real costs, have become a symbol of a broader inefficiency argument.

At the DC Public Library, which operates the Martin Luther King Jr. Memorial Library on G Street NW as its flagship branch, staff have been working since 2024 on a collection hygiene project that includes deduplication of digitized photographs from the Washingtoniana Division. Library administrators have acknowledged the project publicly in budget presentations to the DC Council, though specific cost savings have not yet been reported. Mayor Muriel Bowser's office has pointed to digital infrastructure improvements at city agencies as part of a broader modernization effort, though no dedicated funding line for image deduplication has been announced.

The Smithsonian's National Museum of African American History and Culture, on Constitution Avenue NW, completed a major digitization push in 2023 and 2024 tied to its community archiving partnerships. Technology staff there have discussed in professional forums the challenge of receiving duplicate submissions from community contributors, a common problem when multiple donors digitize the same family photograph independently. The institution has not publicly quantified how many duplicate files resulted from those drives.

What the Experts Are Recommending

Digital preservation specialists who work with federal and municipal institutions broadly agree on a few practical steps. First, they argue that deduplication should happen at the point of ingest, not as a cleanup operation years later, catching duplicates before they enter a collection system is vastly cheaper than hunting them down afterward. Second, they caution against automated deletion without human review, particularly for historical photographs where two apparently identical images may carry different metadata, provenance records, or resolution levels that make one more archivally valuable than the other.

The American Library Association's digital preservation guidelines, updated in 2024, recommend that institutions establish hash-based comparison protocols, a technical process that generates a unique digital fingerprint for each file, before any removal decisions are made. Several smaller DC institutions, including the Historical Society of Washington DC on Massachusetts Avenue NW, have begun piloting such protocols on portions of their photographic collections.

For residents and researchers, the practical consequence of unresolved duplication is slower search results and, in some cases, misleading catalog entries that make a collection appear larger and more varied than it actually is. Anyone using the DC Public Library's digital portal or the Smithsonian's online collections tool has almost certainly encountered the problem without knowing it, two search results that look distinct but link to files that are, byte for byte, identical.

The next concrete checkpoint comes in September, when the DC Council's Committee on Libraries and Technology is scheduled to receive an update on the Martin Luther King Jr. Memorial Library's digital collections project. Advocates for archival transparency say that session will be the clearest public signal yet of how seriously the city intends to treat the problem.