>> From the Library of
Congress in Washington, D.C.
[ Silence ]
>> Good morning everybody.
I'm Beacher Wiggins and again
I am delighted to be able
to introduce the speaker for
today's LC's Digital Future and You.
We happen to have with
us Dr. Kari Kraus
from the University of Maryland.
Roberta Shaffer, the Associate
Librarian for Library Services,
knows of her work and
has been involved in some
of what she had been doing and
wanted Dr. Kraus to come and talk
to us-- staff here at
the Library of Congress.
So, we're delighted that we
were able to finally arrange
and pull this program off.
Kari Kraus is an Associate
Professor at the College
of Information Studies and
the Department of English
at the University of Maryland.
She's also affiliated and
an affiliated staff member
with the Human-Computer
Interaction Lab at Maryland.
Her research and teaching interests
focus on digital humanities,
digital preservation, game studies
and design and long-term thinking.
She has written for, "The New York
Times" and "The Huffington Post"
and her academic work has
appeared in venues such as
"Digital Humanities Quarterly,"
"The Journal of Visual Culture,"
"The International Journal
of Learning and Media,"
and "The Cambridge Companion
to Textual Scholarship."
Her book project, "Hopeful
Monsters: Computing, Counterfactuals
and the Long Now of Things" is
under contract with MIT Press.
Today she'll talk to us about
a new project being undertaken
at the University of Maryland
related to extracting data
from historic recordings.
We're happy to have Dr. Kraus
and I give you Dr. Kraus.
>> Kari Kraus: Thank
you [inaudible] Hi.
Thank you for having me and
a special thanks to Judith
and Angela for inviting me.
I wanted to start by
putting this project
in the larger context of my work.
This is a very unusual
path my research has taken.
As stated in my-- in the
introduction I work primarily
in the areas of digital
preservation, game design
and research and long-term thinking.
As mentioned I have a book
under contract to MIT Press
and I've presented previously at
the Library of Congress on some
of my digital preservation work,
particularly the Preserving
Virtual Worlds project,
which some of you might know about.
It was funded by the NDIIPP
program at the Library of Congress.
And that project was about exploring
how to preserve video games.
We adopted a case set approach.
We had video games from the
1960s through the early 2000s
and we looked at various
preservation strategies and methods
like emulation, migration,
reimplementation and so forth
and then explored metadata standards
and packaging requirements
for those games.
So, I do have a background
in digital preservation
that the current project intersects
with, but it's very different,
as I just mentioned, from
anything I've ever done before.
It's a collaboration with
colleagues at the University
of Maryland including Min Wu
in the Department of Computer
and Electrical Engineering
and Doug Oard in the iSchool.
And it's building on
pioneering work done by Min Wu,
which I'll describe and talk about.
The graduate students on the
project are Adi Hajj-Ahmad also
in the ECE program and Hui
Su in the same program.
And then I want to give a shout out
to our library collaborators at UMD,
Chuck Howell who's Head of
Special Collections in Mass Media
and Culture, Eric Cartier a
digital reformatting specialist,
and Laura Schnitker,
Ethnomusicologist
and Sound Archivist
also with the Mass Media
and Culture special collections.
We're quickly finding that we
couldn't do this work without them
for all kinds of reasons.
That slide is cut off
just a little bit.
The story I'm going to tell is
one of the historical entanglement
of sound and power transmission
or power transmission systems.
This entanglement has a long history
going back to Edison's studio
in New Jersey at the
turn of the century,
the turn of the 20th century.
Actually about 25 years before
the turn of the century.
And so we have actually the first
sound device that could both record
and reproduce sounds, the cylinder
phonograph invented by Edison
and the first electrical
power system also invented
by Edison occurring roughly
in tandem with each other.
Again around the 1870s,
1880s, 1890s.
So, we have these two inventions or
technologies happening or occurring
at the same place at the same time
in the same lab by
the same individual.
The image you see there
is actually from Etsy.
I don't know if there's any
Etsy fans here, but it's a horn
from an antique phonograph
that's been retrofitted
with light bulbs to
turn it into a lamp.
So, I thought it was a good
image that actually embodied
that historical connection.
Now, I want to give a brief
overview first of this project.
This involves something called
ENF, electric network frequency,
and I'll be primarily
referring to it as ENF
in the course of the presentation.
These are ubiquitous
environmental signals or signatures
or you might call them fingerprints.
They get embedded automatically
in audio and video recordings
at the time they're created.
So, audio engineers
know this as power hum
and they're always
trying to get rid of it.
It's seen as noise
rather than signal.
But fortunately, it's almost
always still present in a variety
of historical recordings even when
an effort is made to filter it out.
It's subliminal of course, and
inaudible, but still there.
And so we're able to exploit that in
really novel and interesting ways.
These ENF signatures or traces arise
from small pseudo-random
fluctuations of the ENF
in the alternating current of
the power grid and I'll explain
that a little bit more
in just a minute.
One thing you should know is
that we're talking about traces
in alternating current,
not direct current.
And this is going to be a theme that
I come back to a little bit later.
So, the power grid in the--
or the power system in the U.S.
the electricity is distributed
from power stations to the end users
at a value of generally 60 hertz.
Now, if it's going to fluctuate
or deviate slightly from 60 hertz
and that's what helps us, but that
is typical of all North American
or almost all North
American power grids.
The rest of the world is different.
The rest of the world the
alternating current is
at around 50 hertz, so
definitely in Europe
and other parts of
the world as well.
So, the U.S. is a little bit
different in that respect.
This is a map of the power grid
system in the United States.
And what you see is that
this gives you a sense
of how much geographical
territory any one power system
or power grid has control over.
So, the eastern two thirds of the
U.S. are on a single power grid.
The west coast is on
another power grid.
And then, of course, the state of
Texas has its very own power grid.
What this means is that any
recording made at the same time
as any other recording in
the same power grid area
or region is going to
have identical ENF.
They're going to identical
ENF traces.
So, that's visualized here in this
figure where you have ENF extracted
from a-- most likely from a wall
outlet in three states, Maryland,
New Jersey and Massachusetts,
so all on the eastern grid.
And so one-- the Maryland--
they're closely overlapping as
you see and that's the point.
But, the Maryland, the Maryland
ENF signature is represented
by the red line.
The New Jersey by the green line
and the Massachusetts
traces by the blue line.
These have been normalized somewhat.
So, a recording made at 8:00
a.m. in New York City is going
to have the same power
signature, the same ENF signature
as a recording made at 8:00 a.m. the
same time in say Washington, D.C.
[ Silence ]
Here you power signatures by
country, so moving briefly outside
of the U.S. I'm going to be talking
almost entirely about the U.S.
and really the eastern power
grid system throughout this talk.
But, this is a quick look at
the more global landscape.
So, you see India, some ENF
traces from India's power line,
from China's and from-- and
then another one from the U.S.
And each actually is taken from--
the ENF is taken from
a different source.
So, in India from,
directly from main's power
so most likely from a wall outlet.
And then from a photo diode in the
U.S., so a sensor that detects light
and converts that light
to current or voltage.
And then, China the ENF
signature's extracted from a video.
So, each country has its
own unique signatures
and Min's actually been doing
some research that will allow her
without any reference ENF to try to
geo-locate with fairly high levels
of accuracy an ENF
signature extracted
from a recording of unknown origin.
So, if we don't know where
the recording originates,
it's not only that we can date
and timestamp, but we can start
to geo-locate it to
some extent as well.
That's very new exploratory
research.
I tweeted that I was going to give
this presentation and a friend
who saw the blurb about it said,
"Whoa that would be a good plot
point in a paranoid sci-fi film."
And indeed it would.
There is a kind of, I think
dystopian dimension to it.
But, the applications
I'm going to be talking
about are much more benign.
Okay, and so what we see here is--
okay, so one thing to know is that
the way in which ENF has been used
so far has been almost
entirely in a few test cases
in a legal environment,
in a court of law.
And of those cases almost all
of them have been in Europe,
in Europe not the U.S. And
so far scientists have--
they've been working with ENF
signatures from recordings
for a little over a decade,
from say the late 90s,
1999 or thereabouts forward.
And the way in which
they've been using it is
that they've been extracting
or actually creating a database
of ENF signals or traces since,
depending on the power grid,
but since the early 2000s in Europe
and a little bit later in the U.S.,
around maybe 2004, 2005 in the U.S.
And they've been using a probe.
They-- generally the way it's done
is that you insert a small probe
into a wall outlet and the
output of that probe is a signal
that can be linked
up to your computer
and then recorded continuously
around the clock.
So, we've got ENF reference
databases for some power grids
that have recorded the ENF
signatures or signals continuously,
24 hours a day, seven days
a week, 365 days a year
since the late 90s and early 2000s.
Now, you can-- what that
allows researchers to do,
particularly forensic scientists,
is that they can take a recording
whose date or timestamp is unknown
and try to match the ENF signatures
against those in the database.
So, you're looking for
alignments of signatures
or alignments of those traces.
And it's been used to
authenticate recordings.
So, if you've got a defendant who
says that a recording was made
at such and such a time you
can with this technique verify
that it was indeed made at that date
and time or disprove that it was.
And you can also detect splicings,
say an audio clip has been inserted
between you know two
other audio clips.
That would be visible
in a spectrogram
as a discontinuity
in the ENF signature.
So, what you see in this
slide are two spectrograms
on the left and in the middle.
One is pulled from a video
and the other is pulled from,
directly probably from
the power mains.
And they're-- they
occur at the same time
and so they have very
similar patterns.
When you're looking at a spectrogram
like this the and this is--
I'm not someone who's well
versed in the physics of sound,
so this has been a
crash course for me.
But, you're looking for especially
the ENF signatures that we're trying
to identify, bands
of energy that show
up as very strong or bright colors.
It's the intensity of the color that
signals the strength of the signal.
And so you see that in
the image on the left
and in the middle as
that bright red band.
And generally you've got in terms
of the X and Y axes, it's often time
on the X axis in seconds or minutes
generally for what we're doing
and then hertz on the Y axis.
So, the measurement of frequency
and that sometimes reverse.
Sometimes you'll see
the hertz frequency
on the X axis and time
on the Y axis.
Alright, so I mentioned use
cases so far have been working
with these ENF reference databases.
Remember they only go
back to about 1999.
So, the scientists are
working almost entirely
with relatively contemporaneous
recordings.
And we're interested in seeing if
we can move the techniques backward
in time by as much as say 50 years
or more with approaching the problem
in different sorts of ways.
So, if we don't have a time machine
that will allow us to go back
to the 70s or 80s or earlier to
stick a probe into a wall outlet
to collect ENF then
what can we do instead?
What we're attempting
to do-- there's--
we're actually pursuing
two sorts of strategies.
One is and I'll mention it first
because it's one that we've kind
of put on the back burner for
now, but very much still want
to explore when we get a chance.
That second tech-- that
second sort of strategy again
that we're holding off on for now
involves doing a tremendous amount
of research to understand what kinds
of archival holdings might exist
at power stations and power plants
around the U.S. Do they have--
even if it's data that's been stored
on paper or in some other form,
there might be useful information in
the form of waveforms and so forth
that would allow us to have,
you know sort of construct a
reference database out of that data.
The other technique that we're
currently exploring fairly intensely
is to create a reference database
by drawing on trusted metadata
in very large audio-visual
collections.
So, if we've got really
good metadata
on say live broadcasts
going back a number
of decades then we can use the
ENF signatures in those recordings
to help us date recordings
of unknown provenance.
So, a big part of where
we are right now is trying
to understand the audio-visual
universe if you will.
And this has been, it's been
surprisingly hard to wrap our heads
around for a whole host of
reasons that I'm sure are familiar
to really probably almost everyone
here and that is just the fact
that so many of these
collections are--
they don't speak to one another.
There's no federated you
know sort of finding aids
or catalog records for them.
So, each collection is housed
at its own institution.
It often doesn't speak
in any coherent way
to other similar collections
within that institution,
much less outside the institution.
There are often multiple competing
finding aids and cataloging records
that one must cross
reference and consult.
And then there's the metadata
quality varies considerably
and so we often get really good
metadata on say live broadcasts.
But, for let's say films,
motion picture films,
we don't know when
those were created.
We know when they were circulated or
when they were viewed by the public,
but not when they were
created necessarily.
So, this has been-- I mean this
has really opened our eyes.
And so we're really
exploring a number of--
we're taking a kind of really
incremental approach right now
to try to tease out a
number of fundamental
and basic research issues to
build off of in the future.
So, one great collection that
we are able to draw on right
at the University of Maryland,
which has an amazing broadcast
collection is the NPR Collection.
And I know Library of
Congress has that too,
although my understanding is there's
a weird bifurcation where Library
of Congress has like
maybe the cultural program
and [inaudible] has the
news and information program
and they're switched around.
And I think we have at Maryland
the news and information as well.
The good thing is we
have the original media,
the original source media.
So, we're in the process of
digitizing that right now
and it's being undertaken by
Eric Cartier who I mentioned
at the beginning or the
outset of my presentation.
But, we also of course, have the--
we have the original
reel-to-reel tapes
and we have playback equipment.
And so one thing we're exploring
is best practices for preserving
or retaining that ENF signature
in the digitization process
and I'll say more about that.
So, we need big audio-visual
collections
with really good metadata.
The NPR Collection
has great metadata.
Right here at the LOC you've got the
National Broadcasting Corporation
Radio Collection, which
actually Bryan Cornell who's here
in the audience introduced me to.
And that's a tremendous
collection on which to draw.
And I won't go through all
of the statistics right now.
I will mention I guess the
source media, which is the--
which are those lacquer discs
from the 1930s to the 1980s.
My understanding is that the data
on the lacquer discs was later
transferred onto polyester tape
and then digitized from
the polyester tape.
Brian do you know or?
>> That's right.
>> Kari Kraus: Yeah, okay.
And so that presents its
own interesting issues,
which I'll get to.
We're also actively looking
for interesting case studies.
So, I mentioned how we're
trying to get reference audio
from large audio-visual
collections with trusted metadata,
but we're also of course, in search
of interesting audio-visual
materials that lack good metadata
that might be good candidates for
trying to date and timestamp them.
And so one just local example
right at the University
of Maryland is the WMUC
radio collection, which was--
which has recently been digitized by
Laura Schnitker, an audio archivist
and goes back to the 1960s.
It covers-- there's like 1500
cassettes and reels and discs dating
from like 1960 to the late 1990s.
And a lot of that has really good
metadata, but there are a number
of instances where metadata's
lacking and they have a--
they think they know
approximate dates and so forth.
So, we want to work with some of
those, because this is an example
of the kind of problem you'd
find at other universities
around the country almost all
of which have local
campus radio stations.
But, we're also in search of
really high profile examples.
And so we immediately
started taking a look
at the Kennedy White
House recordings.
There is the wonderful Miller
Center, the Public Affairs
at the University of Virginia.
Their presidential recordings
program has made available
about 5000 hours worth of
presidential recordings covering
about six administrations from
the 1940s to the early 1970s,
and that includes the Kennedy tapes.
These are all secret recordings.
The Kennedy tapes, the Johnson
tapes and the Nixon tapes and that's
about when the secret recordings
ended for obvious reasons.
So, there's this tremendous
resource and a number
of the Kennedy recordings,
secret recordings of meetings
with high level officials
are undated.
And so we immediately zeroed
in and honed on some of those.
Those exist on two different
types of source media,
a reel-to-reel cassette
tapes and Dictabelts.
I don't know how many of you are
familiar with Dictabelt technology.
I wasn't until I started
this project.
They are-- there are
approximately 73 Dictabelts totaling
about 12 hours of conversations
and there's
at least 280 separate
conversations on those belts.
The manufacturing company
called these Dictabelt records,
but as you'll see from the images
they're really in many ways closer,
they seem like sort
of a closer descendant
to the cylinder phonograph.
And you've got this sort of plastic
film-like substrate that's wrapped
around the cylinder and
that's the recording medium.
And I have to say I've
been a little bit confused
by the technical descriptions
I've seen.
I'd be curious at the
end if anyone is familiar
with Dictabelt recordings.
But, the descriptions refer to it
as the sound getting
recorded as grooves.
But, I've also seen
separate descriptions that,
which implies a kind
of intaglio process,
but I've seen other
descriptions that make it sound
like they're embossed in
almost in sort of relief.
So, I couldn't-- I did
some searches for images
and I couldn't get
any close ups of the--
that film-like plastic stuff
to actually see what
the sound looks like,
although the manufacturing
company made a big deal
about promoting this
technology as visible sound.
So, we worked with some,
with the Dictabelt records.
There are about eight of them that
are undated or that lack timestamps.
One we kind of zeroed
in for various--
primarily because it's one of
the longer Dictabelt recordings.
It's Dictabelt 5A.1, which
records a telephone conversation
between Kennedy, this
is right in the middle
of the Cuban missile crisis.
We know that much.
So, we know it's about October
1962 by the content of the audio.
So, it's Kennedy in
conversation on the phone.
The Dictabelt recordings
are all phone conversations
with various high level officials
at a kind of critical moment
in the Cuban missile crisis.
So, when we first started
we thought awesome.
We've got this undated recording.
It's really high profile.
It'll-- it could potentially change
the way you know scholars think
about this sequence of events.
All we need is some reference audio.
We'll just get a bunch of
reference audio from October 1962
and we'll have nailed this.
And no it didn't work
that way at all.
We're finding it's
so incredibly hard.
It's almost we start
to see and we're--
I would just love to
see more data on this.
You start to get a
feel fairly quickly
for which decades are really rich in
surviving audio and visual materials
and which are more
impoverished and why.
And so we-- I-- we started doing
a bunch of searches at a lot
of collections including at LOC
and we-- it was just maddening.
We were finding tons of fragments of
sound, but we really we're looking
for a lot of live broadcasts that--
whose coverage would give us you
know multiple hours in a day.
And we're just turning
up empty handed.
So, what I did instead with
dictation belt 5A, which is not--
there's no-- to the best of my
knowledge it has not been dated.
But, I started doing extensive
research to try to really narrow
down when I thought it was produced.
And there is a lot--
by cross-referencing a lot of
data including White House diaries
and timelines on the Cuban missile
crisis and listening to the tape
and looking at log books from the
White House I feel pretty certain,
pretty darn confident
that this was dated
from August 31, 1962 in the evening.
And even given telephone
records I strongly suspect
at 6:42 p.m. on the eastern grid.
So, then we thought great, now we
can just find audio from the evening
of October 31, 1962,
another dead end.
I found out that the CBS Evening
News was broadcast at six, four--
sorry at 6:45 p.m. at this time.
I believe it was still 15 minutes
and it hadn't moved to 30 minutes.
This was with Walter Cronkite.
So, I thought awesome, there's got
to be a copy of the CBS Evening News
from October 31, 1962, no.
As far as I can tell,
that does not exist.
I contacted the archivists
or I guess they're probably
records managers at CBS.
They looked for me.
Their vaults did not contain
this and they frankly admitted
that they've lost a ton of
broadcasts from this period.
So, once again we came up short.
Working with Bryan I have
just recently located,
which we're in the process of
having digitized a piece of audio
from October 31, 1962 at
precisely the right time.
So, it exists.
It's part of this NBC radio
collection on these lacquer discs
and so we will see what
information it yields.
But, as you can see this has
forced us to move back and start--
instead of starting really big
we feel like we really need
to explore lower level problems and
then kind of scale up from there.
So, I'm going to talk about
some of those and we're going
to go through most of these.
The first one is going to
be a visual demonstration.
One thing we want to do is
simply do a visual demonstration
of how the technique can work.
So, can we find multiple
recordings of the same event
and then align their ENF signatures
and as again a kind of proof
of concept or can we find
recordings of two different things,
but on the same power
grid at the same time.
You can imagine like you
know multiple recordings
of say a sports broadcast
so we're starting there.
Now, everything has it issues.
The first thing we did was we went
to the Vanderbilt television
news archive
because they've been
recording the nightly news
since about 1968 every single
evening or at least on the weekdays.
So, we ordered one copy of the
CBS Evening News from August 9,
1974 and one copy of the NBC Evening
News, which were both broadcast
at the same time, in
1974, on the same grid.
Originate-- the recordings
originate from the same grid.
Anybody know the significance
of that date by the way?
Yeah.
>> Richard Nixon.
>> Kari Kraus: That's right, yeah,
the Richard Nixon resignation.
And so, but then what we
saw is you can actually--
you can see the ENF signatures.
Those are going to appear way down.
You see the hertz measurements
on the Y axis.
And so remember that our
power grid system has-- is--
the electricity is this
alternating current at 60 hertz.
So, you see the ENF signature
on-- at the 60 hertz line.
It's much brighter and more
visible in the NBC image
on the right than on the left.
And then you also see
it, we often can work
at other harmonics,
at higher harmonics.
So, if we don't get
good ENF signatures
at the 60 hertz level we can move
up to different harmonics often.
And so you see the every--
the ENF traces at 120 hertz,
180 hertz, 240 hertz as well.
But, they don't align well between
the two broadcasts and here's why.
I mean if we had thought about this
from the beginning we would have--
we wouldn't have been
able to figure--
we would have been able to
anticipate this problem,
which is that these news broadcasts
are live really in name only.
You do have the anchor presenting
live, but that's intercut
with all kinds of prerecorded
segments or live segments
from other power grids or
locations from our perspective or--
and then interrupted by commercials.
So, you see here an approximate
timeline showing the news show break
down by clip type and you can see
that the live segments
are very, very few.
Those are in red.
So, you've got the CBS broadcast
on the left and NBC on the right.
And the live elements are very few.
They-- some are like 10 seconds in
a 30 minute broadcast or 23 seconds.
And then those live
segments almost never match.
They do in a few instances so we
could with some effort find segments
to a line, but we decided
that we would move on.
We had better luck with
two concurrent Apollo 11
audio recordings.
So, these-- one of these was
taken from the intercom loop used
by the flight director in
the mission control room
to direct others on the-- involved
with the Apollo 11 landing.
And then the-- so that's
one recording.
That's the recording that you see
in red, the ENF signature in red.
And then the other
one is taken from--
it was a recording that was sent
out or broadcast to the public
by a public affairs
officer in Houston.
So, you hear the astronauts' voices
on both recordings and it was--
we knew that these at the same time.
And we did-- you know
we did an alignment
of the two ENF signatures
from those recordings.
And so there is-- they
aligned really well.
They're-- where you do see
deviations is generally
because some content, some semantic
content in a recording ends up being
at the same hertz value as
the ENF signature itself.
So, that creates some misalignments,
but they align very well.
So, we're continuing
doing some of that.
But, that-- so we were
able to demonstrate
that if we do have two
recordings produced on,
you know on the same power
grid at the same time
that we can indeed
align their signatures.
Now, another interesting problem
is differentiating different ENF
signatures and the same recording.
And so as audio-visual
specialists you know
that very often the case the
recordings we have are copies
of copies of copies.
So, it's the case that the content
or the data has been migrated
or transferred from one physical
carrier to another and then
at some point may be digitized.
And so that creates multiple ENF
signatures in the recordings.
So, we need techniques
to differentiate them
and then we also need techniques to
boost and enhance those signatures.
So, I'm going to talk a
little bit about that.
This is a nice example.
In this case we have a
spectrogram of a Kennedy recording,
not one of the Dictabelt recordings,
but one of the audio
cassette recordings.
And you see, you clearly
see two ENF signatures.
One is brighter or more intense
than the other, the one on the left.
And then we see that one of
the signatures actually extends
over a longer period of
time than the other one.
In this case we've got time on
the Y axis and then hertz on the,
the frequency measurement
on the X axis.
So, what happened in this case, the
first problem is how do you know
which is the original ENF
signature and how do you know
which one is the recaptured
ENF signature as part
of the digitization process?
How do you know which is which?
And in this case we see
that one signature runs
for a longer period of
time than the other.
And so we were able to tell
by listening to the recording
that the original recording
stops at one point.
There's just you know silence while
the digitization recording keeps on.
And so we were able to determine
that the signature on the left is
from the original audio recording
in the signature on the right is
from the digitization process.
Interestingly in almost
every case we, in fact,
I don't know that there's
any exceptions at all.
The original ENF signature from the
analog media tends to be stronger
and more robust than
the digitized signature.
[ Silence ]
And one problem that you run
into is that often there's--
when there's more than one
signature they overlap completely
as in the image on the right.
So, we know that there are
actually two signatures there,
but one is covering or
overlapping the other.
So, what do you do?
So, we've developed some techniques
for handling situations like this.
One is there's techniques that
allow you to subtract away one
of the signatures, but I'm going
to talk about some other ways too.
This is a "March on
Washington" recording from 1963.
It's a "Voice of America"
broadcast recording from 1963.
And what you see is one really good
ENF signature at the 63 hertz mark
or thereabouts, that really
bright signature or band.
But, there are two
other faint signatures
so we've actually got three
ENF signatures showing
up in this one recording.
One is very faint at
the 60 hertz level.
And then the other is pretty darn
faint at also around 62 or 64 hertz.
So, what we did in
this case is the--
what you'll notice is that the
63 hertz is that's not normally
where you'd expect to see an
ENF band, an ENF frequency band
because remember our power grid
is 63, sorry 60 hertz, not 63.
We intentionally recalibrated the
playback equipment so that it--
there was some speed deviation
either, we've done both higher
and lower, as a way to prevent
the overlap of ENF signatures
from the original recording
and then the one
in the digitization process just
slightly deviating the speed
so you get one at this
higher hertz mark.
And then we-- simultaneously we
are collecting when we do the--
when we're experimenting with when
we do the digitization process we're
simultaneously collecting
reference ENF from a wall outlet.
So, that it makes it-- so that
we can take that reference ENF
from the wall outlet that reflects
the same time as the ENF we're going
to get from the digitization
process.
And that allows us to easily
distinguish the digitization ENF
from the ENF in the
original recording.
So, two techniques there,
one is the speed deviation
and one is collecting at the same
as digitization the reference ENF.
Now, that third ENF signature
we speculate is that the Voice
of America recording was itself at
least a second generation recording
that had been copied off
some other physical carrier
and so you're getting
the remnants of that.
So, and this is actually
just the same recording
at different harmonic levels
so at the 126 harmonic level.
So, we are doing-- we are going
to conduct what I call
the Alvin Lucier,
"I Am Sitting in a Room" experiment.
If any of you are familiar with
the sound artist Alvin Lucier,
this is a famous piece of
his where he records this--
he speaks a very short
piece of text.
It lasts like-- it's-- it
takes like a minute to recite
and it's self-referentially
describing him sitting in a room
and recording the sound
of his own voice.
So, he records a sound in
a room with good resonance
and then he plays that
recording back
into the same room and rerecords it.
And then takes that recording, plays
it into the room and records that.
So, you get these-- this sort
of transmission chain effect.
It's on a-- in some ways it's like
the children's game of telephone
or Chinese whispers where you have
multiple generations of copies.
And we actually-- we took
actually the Alvin Lucier,
one of the recordings to see
if we could out you know get
out a bunch of-- I think.
I want-- I don't even know
how many repetitions there are
in this recording.
At least 15 I would say.
But, the problem-- the-- we did
get some good-- some okay results.
The problem is each recording
in and of itself is so brief
that it doesn't provide
very good data.
We generally like around 10
minutes of recording to work
with although we have methods for
dealing with shorter recordings.
But, we want to do some lab
experiments where we take a--
we create a recording in some
kind of acoustic chamber,
make a recording and then play it
back in the Alvin Lucier style,
record that, play that
version back, record that
and see how many ENF
traces we can get.
Given that we know we'll encounter
real world examples as you just saw
where there are more
than two ENF traces.
[ Clicking sounds ]
We're also just experimenting with--
we're trying to understand
the quality of the ENF
and the deviation of, the deviation
or sorry the distortion problems
we see with ENF on a wide variety
of physical character--
carriers or media types.
So, we just want to see what
we can extract ENF from.
And so, so far we've experimented
with compact cassette tapes,
with reel-to-reel audio tapes,
with Dictabelt recordings and we're
about to find out about
the lacquer discs.
And so in-- and so far in every case
we've gotten good ENF signatures
on analog-- on these
different analog media formats.
And we want to just experiment
with as many as we can.
So, I just listed a few examples
here from wire recordings,
from motion picture optical
sound, from electric phonographs
to the earliest phonographs
would have been purely mechanical
in their processes.
But, electrical-- electric
models came out pretty early.
And so as a kind of limit case
can we get any ENF from that
and I'll talk-- say something
about that in just a second.
Also we were interested in what ENF
can do not just in terms of date
and timestamp in recordings,
but what can it tells us
about the technology more generally
or the composition
techniques of artists.
So, I just choose optical
sound, which is one I would love
to experiment with, which is
these pictures of waveforms
that are actually printed
on film stock.
And this was used in
the 1930s and 40s with--
and you see actually on the
right hand the waveforms on--
as a representation of
waveforms on the film stock.
The most famous example is
probably Disney's, "Fantasia,"
which used this optical
sound technique,
optical sound track technique.
And it has that-- and if
you've seen the movie there's
that interesting intermission
episode where the conductor
of the orchestra speaks
to the audience
and gives them this mini lesson
in how optical sound works.
And so there's an animation of the
soundtrack and he says very sort
of poetically every beautiful
sound also creates an equally
beautiful picture.
So, it's this very self-referential
moment where he's talking
about the soundtrack underlying
the film and the techniques.
So, you know what the-- what's
interesting about this technique is
that editors who used it
could visually edit the film.
They wouldn't have to listen to it.
They would play with
the visual patterns on--
printed on the film stock and
could splice and so forth.
And so we'll get different
ENF signatures
for the different audio segments.
And I think that there's you
know one, again one potential
of this technology is, what can we
learn about different techniques
by getting-- by extracting a lot
of these embedded signatures?
We also want to explore whether
there are any traces that come
from recordings that were made
on grids with direct current.
This goes back-- I said one thing is
we're interested in is a limit case.
How far back in time can we go
to extract meaningful signatures?
And in the early history of
power transmission there was this
so-called battle of the currents
between direct current
and alternating current.
With alternating the current's
actually changing directions
and the direct current
it's just going one way.
And so Edison who invented the
first power transmission system used
direct current.
He had a little system.
His-- he called it his
electrical light system in Menlo--
in the Menlo Park laboratory
and then also on Pearl Street
in New York City in
the late 19th century
and that used direct current.
One problem with direct
current is that the generator
or the power plant has
to be located very close
to the facilities that
it's powering.
And one advantage of
alternating current is
that you can have much longer
transmission lines connecting the
power plant to the end user.
So, we had this battle of the
currents and Edison fought very hard
for all kinds of reasons,
for ego reasons,
for monetary reasons
for direct current.
And there was-- some of you might
know the tale of Topsy the elephant
who was a victim of this sort of--
this whole battle of the current.
This was a circus elephant
who gained notoriety
for killing one of his trainers.
And it turns out that
that trainer was actually
like burning cigar butts
into the elephant's trunk,
which is just horrible and so no
wonder the elephant acted out.
But, it was decided that
he needed to be put down.
And so Edison volunteered
to, this is a macabre story
but electrocute the elephant
and he filmed the entire thing using
early motion picture technology,
also developed by Edison
and then used that as--
in part of his campaign for
advocating for direct current
because he used alternating
current to electrocute the elephant.
Alternating current had been
legal as an execution method
for humans since-- for about
three years at this time.
And so he wanted to associate
in the publics' mind the danger
of alternating current as a way to
encourage the use of direct current.
But, it didn't pan out and we
all use alternating currents now.
But still, direct current
grids persisted
in places up until like the 1960s.
And we're actually seeing a new
rise of DC current because so many
of our electrical devices-- I mean
all of our electrical devices,
our iPhones, our PCs, our flat
screen TVs, etc. use direct current.
So, we have a little
converter that's--
you know that we have to use that
converts the alternating current
from our power mains
to direct current.
Right now that accounts
for up to a fifth
of our total power consumption,
but there have been predictions.
I'm actually citing a
piece by Peter Fairley in,
"The MIT Technology Review"
that within about 20 years
up to 50 percent of our power
consumption will be you know using
direct current.
So, there's an emergent
rise of micro--
of DC micro grids consequently.
So, we think that by exploring
the limit case of direct current
with very early audio
recordings it also speaks
to audio forensic scientists working
with more contemporary recordings
because they'll also have a need
to work potentially with direct--
with recordings that have traces and
we wouldn't call them ENF traces.
They're some other kind of trace
potentially from direct current.
And then finally I'll
conclude with a-- the--
something that we are proposing
in a new grant proposal,
which is a public time
coding service.
So, one thing we know is that a ton
of audio and video with audio exists
out there not only at institutions,
at collecting institutions,
but also private individuals.
And so if we could crowd
source audio-visual
that would help address a lot
of the problems I've mentioned
in the course of this presentation.
And it could form the
backbone for a system
where an end user theoretically
could upload a piece of audio
or video and sometimes
even video without audio.
We've been able to actually
detect ENF in videos without audio
when there's indoor lighting.
The fluctuations in the
indoor lighting also allow us
to extract ENF.
So, if we could crowd source
trusted metadata in audio recordings
and then also use that as
a service to the public
and to collecting institutions where
they could upload a file and see
if there is any match
in the database,
an ENF match in the database.
That would then give you back
several pieces of information.
Whether there is a match, the time
at which the recording then was
made, the approximate location,
the source of the reference
ENF trace and the duration
of any continuities,
sorry any discontinuities.
I was reading through recently the
state of recorded sound preservation
in the United States, that white
paper commissioned by the Library
of Congress and published by
CLIR, the Council of Library
and Information Resources.
And they note the need,
they advocate for the need
for a national recorded
sound database, quote,
create a unified database of
sound recordings held by libraries
and archives as well as
by individual collectors
to address problems of discovery.
So, could something like the public
time coding service also be the
basis, a kind of foundation for such
a national sort of census of sound?
And I'll just say one
problem that we have too
that we're addressing is the short--
the storage capacity of different
media, analog media formats.
So, I said that we like to
work with at least 10 minutes
of recording time, but a lot
of early media formats are--
contain less data than that.
So, we're developing techniques to
work with shorter segments of ENF.
And then a lot of barriers
as I've covered throughout
this presentation,
one of them is copyright
as you can imagine.
And you know we're very eager to
work with the NBC Radio Collection,
but we know too that that'll entail
getting copyright permissions
from NBC itself.
I mean I point out the irony
that we're entirely indifferent
to the semantic content.
We couldn't be less interested.
The semantic content to us is noise
and the noise to us is signal.
And so it's-- we're
trying to extract the stuff
that audio engineers are
always trying to get rid of.
And so if ever there was a
case for fair use here I would
like to think that this could be it.
Can you help us?
We're looking for interesting
case scenarios,
interesting recordings to work with.
We're looking for funding sources.
We're looking just for you know
for help maybe building the
public time coding service.
And so I will leave you with a
couple of relevant publications.
I actually accidently duplicated
two of the publications.
But, we have a short-- our
first short article is coming
out as a team as part of the
iConference proceedings later this,
actually in March 2014
and then a recent piece
by Min and her students.
The lead author is actually Hui Su
one of our graduate students looks
at some of the recapturing problems
that we've been trying to address.
And so thank you very much.
[ Applause ]
>> And will you entertain questions?
>> Kari Kraus: Yes, absolutely.
Hi, yeah in the back?
>> The database that mentioned
and currently exists--
>> Kari Kraus: Yeah.
>> Who maintains it?
>> Kari Kraus: No one
does right now.
So, this is-- we're actually
writing an NSF proposal right now.
And so we-- we're proposing
this is one potential component
of the project.
>> So, the recordings that you said
that exist from like the late 1990s.
>> Kari Kraus: Right.
Oh, that database got it.
Not the public--
>> Yeah, yeah, yeah.
>> Kari Kraus: Yeah.
Okay, so this is a
really great question.
I want to-- so initially
with the first one.
The first one was developed on
the Romanian power grid in Europe
by a scholar named
Caitlin-Catalin Grigoras.
And for a long time this scholar
was the keeper of the ENF database.
It was not an institutional really;
at least that's not
my understanding.
Now, in the-- I want to
say in Europe the FBI
or the police department in
England maintains a database.
In the U.S. it seems to be--
these databases seem to be
associated with universities.
I believe Virginia Tech is the
keeper or the custodian of one
such database presumably
for the eastern grid
in the U.S. I'm not even sure
that there is a database
for the Texas grid yet.
As of around 2011 there was not
a database for the Texas grid
to the best of my knowledge.
But, it's not hard to do.
I mean with the storage capacity
you just need one of these probes
and the storage capacity makes
it relatively viable to do this.
And theoretically you just need
one database for each grid,
although obviously
redundancy is a good idea.
So, if we've got you know--
if we've got individuals
at different institutions
keeping their own databases
that would be really useful.
At Maryland we don't have
our own reference database,
but we periodically as
I mentioned gather ENF.
So, we're digitizing recordings.
We'll gather ENF to--
as a reference.
But, yeah and we're
just-- I mean in terms of--
I have very little
sense of other parts
of the globe what's being done.
I did show some of those-- some
of the spectrograms from India
and China, but I don't know
if there extensive databases.
It would be-- if the LOC would like
to do that that would be awesome.
[ Silence ]
>> If I [inaudible] the
sort of unique time matching
of fluctuations is there anything
more general that you can learn
about patterns of fluctuations from
different time periods, you know?
For example, maybe in the 1960s
because they did a certain kind
of testing at a certain time
every day there would be a
signature fluctuation?
But, that testing was stopped.
So, you know that anything that had
that kind of fluctuation will be
from a certain era or maybe
fluctuations had become less
pronounced because they figured
out a way to control in the way
that they didn't and you can
timestamp from a before and after.
>> Kari Kraus: Wow.
Wow, that's really interesting.
I think we don't know enough yet.
I mean if we're using--
there's a way
in which you can use
the recordings to kind
of reverse engineer
the power grid, right.
And you can actually learn a lot
about the power grid technology
through the intermediate
technology of the recordings.
It may be that when we start
to explore the power grid
archives that that will emerge.
Now, I do know that Min, my
colleague Min Wu, she is working
on methods and I mentioned
this very, very briefly,
that will allow her to take
a recording of unknown origin
and basically approximate
which grid across,
you know across different parts
of the globe it belongs to.
I think there's probably
that potential
and I think that's a really,
really interesting idea
because it would allow
you to at least narrow
down some period of time.
Even if you couldn't get precision
of-- you know a precise timestamp
or even a precise day, if you
could say this is definitely
between you know 1922
and 1924, right.
That's still interesting, right,
depending on the situation.
So, I think that's
a really good idea.
I think we have to learn more.
Yeah or yeah?
>> If you knew the
date, but not the time--
>> Kari Kraus: Uh huh.
>> Of the recording and you have--
and there's a definitive matrix
number and an engineer's log
and you have a recording from that
matrix does that begin to give you--
>> Kari Kraus: Yes.
>> The information you need?
>> Kari Kraus: Yeah.
There's all kinds of ways you
can start to narrow things down.
So, sometimes we do have--
I mean even just the spreadsheet
I've been working with or looking
at quite a bit of the
radio collection
at the University of Maryland.
Often they have approximate
dates for things.
And so if you've got an
approximate date you can begin
to assemble reference recordings
around that time period
to help you better.
Or if you just know
the day then you need--
you know you'd need a much
smaller amount of audio
to then give it a precise timestamp.
So, there are ways to start
narrowing things down.
But, I think you know your
question really speaks to that
and I think we just have to
advance a lot more in the research.
But, I think that seems like a
really fascinating area to explore.
Bryan?
>> Yeah I was wondering--
I mean audio engineers use power
conditioners to try and get rid of--
>> Kari Kraus: Yes.
>> Is there some way that if the--
on such recordings it
would leave a trace anyway
that you can think about?
>> Kari Kraus: It-- well so
we've been wondering about this.
You know with the-- like with the
Voice of America recording of,
"The March on Washington."
Their digitization is really--
it's all about getting you
know very clean recording.
And so that's-- I think that's
one reason you're getting a weaker
ENF signature.
But, I'm not-- I mean I'm
not entirely sure you're ever
eliminating that ENF signature.
It seems-- we found that with
lossy file formats, you know with--
even with MP3 we can often extract
ENF and it doesn't even have to be
that the recording
equipment was plugged
into a wall outlet necessarily.
That's when you get the best ENF.
But, if the recording equipment
is operated on batteries,
operating on batteries, but it's
in an indoor environment as part
of you know a large-- there's
an electromagnetic field.
It's still-- there's
going to be coupling
between that electromagnetic field
and the recorder and
the microphones.
So, you're still liable even in
a condition like that to get ENF.
So, I-- but that would
be I think a really--
another interesting lab experiment
to do is to take a recording of--
and this would-- if
anybody has a record--
such a recording we would
love to work with it.
Where the audio engineer you know
tried to block all power hum, right.
Could we-- is there
anything that's still there.
I would love to do a test on that.
Yeah?
>> A suggestion for that.
You mentioned the Apollo
11 recording--
>> Kari Kraus: Yes.
>> That you use.
Did you use both the CAPCOM
communications and the recordings
of that were done in the capsule?
>> Kari Kraus: So--
>> They're different.
>> Kari Kraus: They're
different, right.
You know I want-- so this is where
I wish I had my colleague Doug Oard
with me because he's
doing another NSF project
on the Apollo 11 recordings.
So, I know very little about them.
I do know there's at least
three sources of audio
that they're trying to synchronize.
So, there's one actually
application domain for this is
if you've got a complex set
of events unfolding in time
and you want to create
a really tight,
a really tight timeline
synchronizing the evidence
around that event you can imagine
taking multiple recordings unfolding
over a period of hours
and aligning them
by finding their overlapping
segments
and doing really complex alignments.
So, that's some of what we were
doing with the Apollo 11 recordings
with those alignments, but yeah
that's a really good question.
I know there are at least
three sources of audio
from that, from the landing itself.
So, yeah.
>> You mentioned you were at the
courts in Europe using ENF data.
How relevant do you think
that that could become
as evidence in a court of law?
>> Kari Kraus: Well, you
know it already has been used
in a number of cases.
I want to say if you Google for--
and this is why I mentioned like
the police department or the FBI
in England and it's keeping one
of the ENF reference databases.
There was a BBC story a year
or two or a few years ago
on how ENF is being
used in legal contexts.
So, so a lot of it has to do
with standards of evidence.
So the forensic community works
with very high standards of evidence
that can be-- that will make this
admissible within a court of law.
And there are-- there's one
for example master's thesis
from I think 2011 that
actually goes into great detail
about the configuration of
the ENF database that's needed
to make this kind of evidence
admissible in a court of law.
So, with our work we're not
meeting that threshold of evidence.
We're still working with
very high probabilities
and when we take two recordings and
can align their signatures the idea
that they would be a random match,
the idea that they would be a
false match is very, very low.
So, we're still working with very
high probabilities but we're not
at that staying threshold
of forensic evidence
that you are seeing
with scientists working
with the really contemporary
recordings.
In fact today often they don't like
working with analog media at all.
They just, they don't see it
as sufficiently, the signatures
as sufficiently trustworthy I guess.
But, but part of our research
program is developing new techniques
to address and right problems that
we encounter with the analog media.
>> I'm not sure if this has
any relation but when you talk
about [inaudible comments] I
remember a teacher years ago said
that receiving a signal,
one of the best antenna
if you can isolate it is plug into
just one of the two plugs of a house
and you have like the whole state
of Maryland in your antenna.
Of course, I suppose
[inaudible] danger
of electrocution, fire
or whatever it is.
Is there any relation
between that and--
have you ever heard of this kind?
>> Kari Kraus: No, no and all I
really know is there are in a number
of studies just, just
as there is research
on configuring the databases, the
ENF databases according to standards
that would be admissible
in a court of law,
so too there are schematic designs
for the probes, the ENF probes
to insert into wall outlets.
So, but beyond that
I know very little
about the data collection from,
from a direct power main source.
Other than that-- I mean
that's-- you want that.
That's the best kind of
ENF reference to work with.
Yeah.
>> So, for the people that are
doing [inaudible audience comments]
>> Kari Kraus: Great question.
Yes.
>> Do we need to do
something different.
Can you get what you
need from our datasets?
>> Kari Kraus: This is something--
we want to do a paper on this
and we want to think
about-- so what--
some practices are already helpful.
So, at University of Maryland,
for example, Eric Cartier,
his preservation master from-- for
the NPR recordings retains the tones
and silences in the
original recording
and then the various derivatives get
rid of that in the access copies.
But that is very, very helpful.
In fact, we get stronger
ENF in those--
and when there's no content you know
to interfere with the ENF signal.
So, so that's already in play.
You know, one, one thing is
that audiovisual archivists
or digitization experts are already
creating multiple files as part
of the digitization process.
I mean there's the-- I forget all
the terminology preservation master,
the access copies, the production.
I think there's, what's the
term that I'm thinking of?
>> Service copy.
>> Kari Kraus: Service copy,
you know could we have
an ENF trace copy.
You know it's always
about resources right.
That's the problem and, and ideally
you know you have, you have--
I'm mean I'm going to pitch
out the Utopian vision.
The Utopian vision is that you've
actually got two playback devices.
So, one for your preservation
master that you're going
to get all your derivative
copies from and then one
for your ENF trace copy,
where you're actually using
that speed deviation technique so
you actually are reinstrumenting
or recalibrating that
playback device differently
from the other one.
But you could still get, you could
still hypothetically have an ENF
trace copy that is produced
from the same playback equipment
as the other copies.
You know you wouldn't
have that offset effect
that would be really useful, but
nonetheless it would still be,
I think there could be other,
you know other techniques.
So the other thing
you can do actually is
and this is something I
mentioned is if archivists,
if audiovisual archivists were-- had
a probe and they were collecting ENF
as they were digitizing and
they had a database of that,
right then you've got, you've got
ENF reference that allows people
who work with ENF to say okay,
this is the digitization ENF.
And I know that because there's
also the reference audio.
Maybe the metadata for both
is sort of packaged together.
And then you can instantly say I can
disregard that if you're interested
in the original ENF signature
from the analog media.
So I can, I really
want to do a paper
where we're exploring that issue.
What might this mean
for audiovisual practice
and particularly digitization
and but also media,
I mean reformatting isn't just
you know from analog to digital.
It's also often from analog
carrier to analog carrier.
As we see with the NBC collection,
which went from the lacquer
discs onto polyester tape.
So, there's implications
for that as well.
>> I have a basic question.
I think that what you're
saying that you could also pin
down where the recordings
took place,
but that's only within
the grid, right.
You couldn't go to a--
>> Kari Kraus: This is-- right.
Right, so you could sort of
think of the time capabilities
that ENF gives us is-- are
of much finer granularity
than the geo location
abilities, which is really just
about narrowing it to a grid.
And as we saw a single grid covers
a lot of territory that's useful
for the date and time stamping.
That's the you know the larger
the grid the better for the date
and time stamping because
then you've got--
you've got your universe of
reference audio is much larger.
But for the geo location side
of things it works against you.
But, and then-- but there's--
that's also a research area.
It's--
>> So is there likely
that there's something--
>> Kari Kraus: There
may very well be.
There's that-- I mean Min,
Min has suggested as much.
So there may very well be.
>> Very good.
>> Kari Kraus: Thank you.
Thank you so much.
[applause]
>> This has been a presentation
of the Library of Congress.
Visit us at loc.gov.