Unreadable archives in the years to come (LA Times)

[Steve E.]

I think this is of interest.....Our interest in mechanical reproduction dovetails with the concerns about digital archiving found here. Thanks to my friend Stephen Klein for forwarding this to me.

--Steve


Unable to Repeat the Past
Storing information is easier than ever, but it's also
never been so easy to lose it -- forever. We could end
up with a modern history gap.
By Charles Piller
LA Times Staff Writer

September 13, 2006

Carter G. Walker remembers the day her memories
vanished.

After sending an e-mail to her aunt, the Montana
freelance writer stepped away from the computer to
make a grilled-cheese sandwich. She returned a few
minutes later to a black screen. Data recovery experts
did what they could, but the hard drive was beyond
saving - as were the precious moments Walker had
entrusted to it.

"All my pregnancy pictures are gone. The video from my
first daughter's first couple of days is gone," Walker
said. "It was like a piece of my brain was cut out."

Walker's digital amnesia has become a frustratingly
common part of life. Computers make storing personal
letters, family pictures and home movies more
convenient than ever. But those captured moments can
disappear with a few errant mouse clicks - or for no
apparent reason at all.

It's not just household memories at risk. Professional
archivists, those charged with preserving the details
of society, tell a grim joke: Billions of digitized
snapshots, Hollywood movies and government annals,
they say, "will last forever, or five years, whichever
comes first."

Socrates described memory as "a block of wax . the
mother of the muses. But when the image is effaced, or
cannot be taken, then we forget and do not know."

Digital storage methods, although vastly more
capacious than the paper they are rapidly replacing,
have proved the softest wax. Heat and humidity can
destroy computer disks and tapes in as little as a
year. Computers can break down and software often
becomes unusable in a few years. A storage format can
quickly become obsolete, making the information it
holds effectively inaccessible.

No one has compiled an inventory of lost records, but
archivists regularly stumble upon worrisome examples.
Reports detailing the military's spraying of the
defoliant Agent Orange in Vietnam, needed for research
and medical care, were obliterated. Census data from
the 1960s through 1980s disappeared. A multitude of
electronic voting records vanished without a trace.

Records considered at risk by the National Archives
include diagrams and maps needed to secure the nuclear
stockpile and policy documents used to inform partners
in the war on terror. Much like global warming, the
archive problem emerged suddenly, its effects remain
murky and the brunt of its effect will be felt by
future generations. The era we are living in could
become a gap in history.

"If we don't solve the problem, our time will not
become part of the past," said Kenneth Thibodaux, who
directs electronic records preservation for the
National Archives. "It will largely vanish."

Humans have long imprinted collective memories on
available objects, inscribing stone slabs, marking
paper, etching paraffin cylinders and finally encoding
computer disks. Chinese astronomers of the Shang
Dynasty etched the words "three flames ate the sun"
onto an ox scapula to pass on their celestial
observations.

Thirty-two centuries later, that "oracle bone"
confirmed for today's scientists an ancient eclipse,
which allowed them to recalibrate their understanding
of how the sun affects the Earth's spin.

Suppose those early stargazers had scratched out their
findings in secret code on a mud flat. In effect,
that's what NASA did when it used digital tape to
store spaceflight data from the 1960s and 1970s. The
observations could have helped unravel today's climate
change and deforestation mysteries, but by the 1990s
most of the tape had degraded beyond recovery.

Federal practices haven't improved much since then.
Leading archivists said that the records of George W.
Bush's presidency would probably be far less complete
than those of Abraham Lincoln's.

In Lincoln's day, scribes vigilantly penned events and
actions momentous or minute. Trusted records were
viewed as essential to legitimize government and
preserve citizens' rights. The bureaucracy generated a
fairly complete record of what the government did,
including voluminous chronicles of the Civil War.

Future historians will have a harder time with Iraq
war records, created in several digital formats, some
of which are already obsolete, said David Bearman,
president of Archives & Museum Informatics, a
consulting firm in Toronto.

In 20 years, pushed aside by waves of cheaper
technology, "those records will be very difficult, if
not impossible, to retrieve," he said.

Digital files are also remarkably easy to destroy, by
accident or design.

Just after the U.S. invasion of Panama in 1989, Air
Force historian Eduard Mark was assigned to write a
history of the campaign. When he found the right
records, the officer in charge was seconds away from a
single keystroke that would have purged every daily
"situation report" prepared for the Joint Chiefs of
Staff, data crucial to understanding the conflict.

Soon after, Mark had an epiphany.

"I spend much of my life burrowing around in archives.
Curiously enough, I had never noticed that the offices
I worked in were not generating much archival
material" or systematic records of any kind, he said.

Historically, the Pentagon created vast paper trails
memorializing orders for paper clips, D-day battle
plans and heated policy debates. In the 1980s,
computers replaced typing pools and file clerks.
Carbon copies were gradually replaced by perishable
e-mails, cryptic PowerPoint slides and transient
websites that can be deleted instantly.

It's more than a loss to history.

"If officials leave no paper trail," Mark said, "how
can they be held responsible for their actions?"

At the same time, though, more information than ever
is being created and stored.

UC Berkeley scientists estimated in 2003 the world's
annual output of digital content stored on magnetic
and optical media such as hard drives and compact
discs, not counting films, TV shows or websites. Their
upper estimate was equivalent to 500,000 times the
print holdings of the Library of Congress.

Yet a few generations from now, this period may be the
most obscure since the advent of the printing press,
partly because of the structure of digital files.

As a book, "War and Peace" is a literal representation
of Leo Tolstoy's words. Properly stored, it would be
readable for hundreds of years. On a CD, "War and
Peace" is an encoded string of 0s and 1s. Without the
right descrambling hardware and software, the disk is
best used as a coaster for a cold drink. More and
more, documents are produced only in digital form.

"We are capable of producing perfect copies, which
confer a kind of immortality on the things we create,"
said Rand Corp. archives expert Jeff Rothenberg. Yet
those copies require software "to make them real."

What can be done when old devices and software are
eclipsed? Electrical engineer Charles Mayn, 63, has
spent his career answering that question.

He runs the preservation lab of the National Archives
- a museum of archaic wire recorders, Dictaphones and
wax cylinder players - where movies and audio files
are transferred from obsolete to contemporary media.

Mayn's toughest challenge was 11,000 hours of audio
recorded in Germany after World War II. It contained
thousands of unique interviews of war-crime defendants
and witnesses, such as assistants to the Nazi doctor
Josef Mengele, who conducted horrific experiments on
death-camp inmates.

"Mengele was wanting to find out what happens to
pilots if they fly too high, the air's too thin, they
come down too fast," Mayn said, referring to one
recorded interrogation. "So the technician helped with
experiments on prisoners in pressure chambers."

The interviews, which contain crucial details
otherwise lost to history, were recorded with a
"Recordgraph," on 50-foot long, one-inch wide, nested
plastic belts. The device cut grooves into the plastic
much like those on an old vinyl record.

Not a single working Recordgraph machine could be
found to play the interviews.

So Mayn built two from scratch.

Over a decade, the interviews were moved to
quarter-inch audiotape. Kept cool and dry, tape can
last 50 years. But soon after the job was finished in
the mid-1990s, the last factory making quarter-inch
tape closed its doors and players are no longer made.

Today, everything the Archives rerecords is going
digital. The old media are dead.

Mayn said that like the Recordgraph and quarter-inch
tape, he's among the last of his breed. No one could
build a replacement DVD player from scratch, because
there's no reasonable way to resurrect the software
once it is lost.

"Someone a few centuries out who found a [Recordgraph
belt], can kind of figure it out - put a needle on it
and get sound back," he said. "If they find a CD,
there's just nothing there."

The National Archives building in Washington is
inscribed, "What is past is prologue" - a fitting
aphorism for the agency that conserves the nation's
heritage.

The agency is spending $308 million on an electronic
system regarded as the first step to solve the digital
archive problem. Yet a chief method the agency uses,
translating information onto more contemporary media,
is like a child's game of telephone. Every transfer
loses shades of meaning.

The difficulty and cost of the process prompted WGBH,
Boston's public broadcasting television station, to
hedge its bets. It purchased 6-foot-tall, 1960s-era
video recorders and shrink-wrapped them in cold
storage to ensure a way to play back a unique
collection of Boston Symphony concerts from 1955 and
an interview series hosted by Eleanor Roosevelt,
featuring such luminaries as then-Sen. John F.
Kennedy.

Transferring data gets more difficult over time. New
material emerges at an ever-greater rate. Technical
descriptions that allow old documents or images to be
viewed on new devices must be appended to each file.
Such descriptions gain complexity with each migration
and soon outgrow the original documents.

The limits to the Sisyphean migration strategy have
stimulated several new approaches.

The Los Alamos National Laboratory in New Mexico
operates a website that converts academic papers in
physics and other fields into several digital formats,
increasing the likelihood that the information will be
readable as software standards evolve.

Scientists are also working on universal translators -
software designed to operate on any computer and
translate any software to the latest standard - and
"emulators" to mimic old digital files for use on
modern devices.

But those methods are also imperfect, another reason
that the records of modern society could become like
the artifacts of a primitive culture - fascinating,
but mysterious and full of gaps.

Jason Lanier, a computer scientist who coined the term
"virtual reality," describes what's at stake this way:

"If you let forgetting and remembering happen
arbitrarily, you're losing part of yourself."



George Mallman

[cuttercollector]

Thanks. Good article!
I was at the AES and listened to the historical committee and also spoke with a person speaking about archiving. It is interesting the subject comes up everywhere. The committee was concerned about archiving doccuments in electronic form by services that "guarantee" not to loose the bits you send them.
I was talking with the archivist speaker about what marvelous things have been done to clean up hundred or even 50 year old disc recordings which are still in a viable playable format (which is good!) but might be, after they are cleaned up, subject to the same issues as were stated in the article.
Wouldn't it be ironic if the best way to preserve Caruso was to not only keep the original discs but to clean him up digitally with the best "magic" we have today and cut another state of the art mechanical disc (normal record) to preserve it!
Part of the issue is getting rid of orginal material after some intermediate step precieved as "better" for the time. What if all we had were tape copies made with restoration techniques of the 1950s instead of the Caruso records?

[tsullivan]

There is a short movie produced by RCA Victor called "Command Performance", made in 1938, which goes through the entire process of making records. Its available on DVD from a number of ebay sellers. It starts with the making the wax recording blanks they used in those days, to the recording session and cutting the wax master, and producing the metal master and stampers, and finally the stamping of the records. The movie also showed the underground storage vault, where the metal masters of every RCA performance is stored. They went for a walk down the halls of this vault, randomly pulling out metal masters and playing a short version of the music supposedly on the master. One was a Caruso number, another a John Phillp Sousa number. Since that film was made in 1938, and those recording were already old then, I have always wondered if RCA (and other record companies) still have all those metal masters from every performance they ever recorded. It seems to me that these metal parts will survive longer and be playable longer than any other type of media, especially magnetic tape.

CD's, you say? HA! I've already got a handful of CDR's that I burned on my computer perhaps 4 or 5 years ago that are now unreadable.

Tom

[W.B.]

Most of those metal parts as in that film (the RCA logo during the opening credits, however, points to a circa 1942 or '44 re-release) disappeared when the Camden, NJ building that housed them was demolished some time in the 1970's. A few escaped the wrecking ball when collectors took whatever they could. But RCA had a devil's time assembling some compilations of recordings from within those vaults when it came time to put out Red Seal LP reissues during that decade and later.

[Aussie_cutter]

Steve, thanks, that is a thought provoking article.

Recently I had to dispose of some obsolete CD-R's. I was surprised how some of them had become very brittle. Others were starting to "flake" and lose the dye. I guess they were only 10-15 years old.

H

[Serif]

http://en.wikipedia.org/wiki/Apollo_11_missing_tapes

NASA erased the slow-scan video masters of the first alleged Moon walk footage. The reels were bulk-erased for important telemetry recordings during the Cold War. *l*






- A. Paul Ling

[opcode66]

Don't wait for your storage device to fail. Simple solution.

I don't have any hard drives that are running that are older than two years. I have a maintenance schedule. After two years I use cloning software to clone the OS and data to a new (and consequently larger) hard drive.

This minimizes the possibility of a catastrophic and irreparable disc failure and the subsequent loss of data to nearly zero. I had a drive crash on me and lost everything once about 14 years ago. It has never happened again. Ever.

My maintenance schedule includes external drives. Most people don't realize that they put the cheapest drive possible inside those fancy enclosures. They are more prone to failure than a lot of other drives because they are so cheap. Backing up to an external drive that is 4 or 5 years old is a really bad idea seeing as though that backup drive is close to the end of its life.

I haven't lost a single file in almost 15 years now with regular changeouts of my magnetic media. The other nice thing about my approach is that I have a working drive shelved. If my current drive did die unexpectedly I would be able to put the old drive in and boot the computer. It would be old, but the computer would work. For someone who makes their livelihood from a working computer, this is crucial.