The Record - September 1998

From The Archivist: Records Everywhere, But How Are They Going to Survive?

By John W. Carlin
Archivist of the United States

Editor's Note:
The following is a slightly expanded version of an article that appeared in the Washington Post, August 23, 1998.

Your house contains all kinds of family records that you may imagine you are going to leave to your grandchildren, and theirs, and so on into posterity forever. Not likely.

Archivist of the United States, John W. Carlin

Sunlight is imperceptibly fading the family photographs on your piano. High temperature and humidity levels in your attic and basement are destroying the photo negatives, love letters, and newspaper clippings you have stored there. You do not see the effects until one day the picture images look like ghosts, letters break where they have been folded and refolded, and newspaper clippings crumble in your hand.

Technological obsolescence is even more threatening. Do you still own a phonograph that will play your 78 rpm recordings? Do you own a turntable at all? Does the old tape recorder on which your mother recorded your first words still work?

The computer age has raised the preservation problem to a mind-boggling new level. How much computer data did you lose when your four-year-old system crashed last week? Will your new system read your old diskettes? And how many files have you inadvertently erased? Even if your system lasted more than ten years, your electronic records probably would not.

What is happening to your records—and the technology needed to access them—is happening to the records of your country, your culture, and much of the world. At risk are enormously important records that individuals need to document their rights and entitlements, that governments need to conduct public business, that citizens of a democracy need to hold officials accountable, and that scholars need to write accurate history.

The archival facilities of the National Archives and Records Administration collectively hold literally thousands of audio and visual records, many of them in formats already obsolete.

For example, one cannot just go buy in a store the Sony Model 800 machines on which former President Nixon recorded the famous White House tapes that came to light during the Watergate investigation. In fact, the National Archives has found only eight machines that will play the original tapes, including Nagra TRVR recorders for which it seems possible to find spare parts only in Europe.

Similarly, it is hard to find workable equipment that will play the original Dictabelt and tape recordings of White House conversations made by Presidents Eisenhower and Johnson, let alone records electronically dictated by thousands of other government officials decades ago. And a home movie-maker made the original Zapruder film of the assassination of President John F. Kennedy with a camera that is now obsolete.

Because originals are so important, we keep a kind of museum of equipment that will work or can be modified to work. But the tapes, disks, and films themselves deteriorate. We try to slow such deterioration by keeping the Zapruder film in cold storage, for example, or baking old audio tapes until they come unstuck from each other long enough for us to re-record them. Unless some new technology some day gives us better options, long-term preservation requires that we convert such audio-visual records to modern formats to make them publicly accessible.

That is not as simple as it sounds for two reasons. One is that even some of our second-generation equipment has become obsolete. But more important we are dealing with a tremendous amount of material to re-format. We have 68 employees meeting many preservation needs in labs in our Washington area facilities, and other employees and contractors help preserve materials throughout the more than thirty archival facilities, records centers, and Presidential libraries that we operate across the nation. But just to re-record our collection of ABC radio news coverage of the Cold War era, featuring such famous broadcasters as Walter Winchell and Drew Pearson, means dealing with 27,000 disks. NARA’s archival facilities also hold millions of feet of motion pictures, such as Eva Braun’s original color "home movies" of Adolph Hitler and the film about Pearl Harbor that famed Hollywood director John Ford made after the attack, and millions of photographs, more than half of which are a half-century old. Many photographic negatives are on deteriorating acetate film. They can deteriorate even faster than the acidic paper with which we must deal to preserve millions of textual records. Among pictorial documents in jeopardy are images from the New Deal, the Great Depression, World War II, and the postwar years. Our audio and visual records contain historical sights and sounds that bring the 20th century alive.

Preserving such audio-visual material is simple, however, compared to the problems posed by the even newer information technologies of our time. As the government’s record keeper, NARA faces increasingly enormous quantities of records generated with computer technology, such as word-processing documents, automated databases, and electronic mail. As with paper records, NARA must work with federal agencies in appraising the value of such records and we must approve decisions on how long each body of records should be kept, including which ones warrant preservation in the National Archives. Electronic records pose the biggest challenge to record keeping ever. Over the past quarter-century, the National Archives has taken in approximately 90,000 files of electronic records, which were a minute fraction of all such records generated even in that time. We estimate, however, that today the Treasury Department alone is generating annually, in e-mail only, 960,000 files of electronic records that we are likely to need to preserve. That is more than ten times as much from just one agency, in one year, as the National Archives has received from the entire government in a quarter-century.

NARA preserves and provides access to paper dispatches that generals sent and received in conducting the Civil War more than a hundred years ago. But a hundred years from now, will electronic records remain accessible with which to study command decisions in the Gulf War? Or, thinking of current headlines, how much of the record material accumulating in the computer systems of the various independent prosecutors will survive deletion, deterioration, or the discarding of outmoded machines and programs that can read it? Before, only a small part of the historical record has been on highly fragile media; now, almost every record will be. And given the rate of software and hardware obsolescence, maintaining original systems on which to read such records is neither technically nor economically feasible.

Preventing Gaps

The National Archives was created in 1934 in part because fires in government offices over the years had destroyed so many documents, including the census records of 1890. To this day, American historian Charles Hosmer has written, whenever we do research, we hit this blank. More recently, a participant in a Getty Center conference, Danny Hillis, was quoted as saying something eerily similar: Historians will look back on this era and see a period of very little information. A ‘digital gap’ will span from the beginning of the wide-spread use of the computer until the time we eventually solve this problem. But we may be facing starvation in the midst of plenty. Deanna Marcum recently pointed out, The amount of information we create on our computers is growing exponentially, leaving us with a quantity of data that humans have never coped with before. The National Archives confronts both faces of this problem: on the one hand, the possible loss of very valuable historical information, and on the other, the problem of sifting out the valuable records from a rapidly increasing quantity being created.

"But we may be facing starvation in the midst of plenty."

It isn’t just historians who will suffer from inaccessibility to the information that we are all increasingly generating and keeping in electronic form. Potential losers include veterans who need to document benefit entitlements, individuals using court records to secure legal rights, film producers searching for material for documentaries, researchers writing books and articles on all kinds of subjects, and government officials who need to find data, track activities, and review policies.

Much has been done and can be done to contain this modern equivalent of a records fire. Industry has developed more durable film with more stable color dyes, for example. We can preserve records of all kinds longer if we place them in archival-quality containers, store them where temperatures and humidity levels can be kept relatively constant, and protect them from direct sunlight, pollutants, contaminants, and rodent and insect infestation. We can delay film deterioration through cold storage, and we can copy photographic images from flammable nitrate and unstable acetate-based film to more protective polyester stock. Similarly we can transfer sound recordings to more durable media, and we can microfilm textual material, making multiple copies for use as well as for preservation.

These are well-established and generally cost-effective techniques. The volume of records to which we can give extended life in these ways is limited not so much by inadequate know-how as by dollar limits for equipment, training, and personnel.

"Much has been done and can be done to contain this modern equivalent to a records fire."

We have avoided the problems of obsolete and deteriorating media in the electronic records we preserve in the National Archives: Whenever any of the digital media in our custody show signs of deterioration, or whenever they reach 10 years of age, we recopy the records to new media. This recopying has several benefits: it keeps the records intact and offsets obsolescence; the newer media always store much more information in less space, so we save on storage costs; and the new media can be read more quickly, so that we can provide faster access to the records. Nevertheless, preserving and providing long-term access to electronic records is vastly more difficult than for other records. For most types of electronic records being created today, there are no known technical solutions that would enable us to guarantee that, in the future, we can provide access to those records in the same form that a user would see today. Other problems include: How do we identify and sort out e-mail of on-going value from that of only transitory use? How do we authenticate electronic word-processing documents and protect their integrity from tampering? On what terms do we arrange for access to automated databases? Must we periodically migrate electronic information to new systems as old ones become obsolete and digital media fail? Can standards be developed so that archives will have to dealwith only a few formats rather than many? Or can new computer systems be made capable of emulating older systems, or capable of understanding descriptive data encapsulated with old records, so that the new systems can render such data readable? It may even be possible to develop a new storage medium both affordable and durable a computer-age equivalent of stone tablets or at least microfilm, but we may not be able to buy or repair the equipment needed to read such a medium, or the way information is written on this medium could become obsolete and thus inaccessible.

Working Together on Electronic Records Challenges

Ensuring that we have reliable electronic records that maintain their integrity over time depends a lot on research-and-development projects in which NARA and many others are deeply involved. An interagency Electronic Records Work Group that I organized will make recommendations soon to guide decisions about retaining or disposing of certain kinds of electronic records. We are working with the Department of Defense on electronic record-keeping systems that may be applicable elsewhere in the government, with the State Department on electronic cable files, and with several agencies on electronic records declassification. And internally, NARA is trying to squeeze more out of its resources to put into the management of electronic records while also preserving them in an electronic records center that will need much more than its current annual budget of approximately $2 million. The effort is not inexpensive NARA is partnering with other agencies, businesses, and universities to meet the challenges in part because millions of dollars must be invested in the effort but we can succeed, particularly if Americans care.

Among the records we safeguard are our national Charters of Freedom the Declaration of Independence, the U.S. Constitution, and the Bill of Rights which hundreds of American citizens, school kids, and foreign visitors come to view every day in the rotunda of the National Archives Building in Washington. Nearly 50 years ago, we put these precious documents into helium-filled glass cases, lowered each night into a bombproof vault, which made them, in the words of President Truman at the time, as safe from destruction as anything that the wit of modern man can devise. Because of flaws in the outer shell of the encasement, we will soon start re-encasing them to be sure.

We want to preserve them forever. But we also must save so much else that citizens of a democracy need and historians want to study. It will be worse than sad if the marvelous technologies that are giving us a new information age outrun our ability to keep a record of it. We hope to learn how to use new technologies to keep records better and even cheaper. For the sake of both democracy and history, we must.