I recently came across an interesting article in the New York Times discussing the Holocaust, the increasing ignorance amongst members of my generation about certain key facts, and the looming issue wherein concentration camp survivors are dying off due to old age, making it impossible to continue to hear their stories firsthand. I myself was fortunate enough to hear from a local area survivor, Helen Sperling when I was in high school, and was always struck by the intimacy of being in the same room as someone who had lived through an indisputably horrific experience. My most vivid memory of Helen’s story was how her best friend rapidly came to perceive Helen as “dirty” due to her Jewishness (an event summarized here at some level of detail).

The Times article piqued my interest, when at the end of the article, it mentioned that the Illinois Holocaust Museum and Educational Center had started presenting holograms to try to capture the experience of interacting with and hearing speak a Holocaust survivor, as I did in my teenage years. This led me to take a look at the following video:

What struck me in particular about the Shoah Foundation’s New Dimensions in Testimony program was the use of a geodesic dome (seen in some of the photos of their website) for recording the survivors’ testimonies in 3D. The use of an NLP model for answering questions was similarly intriguing, although didn’t catch my imagination as much. And now I must apologize in advance if this is too meandering, but in my mind this harkened back to an experience in my first lab studying psycholinguistics.

When I was working in the LAB Lab as a research assistant we were studying speech processing in real-world contexts using high density EEG headsets (see here). We were using a methodology to localize brain activity based off electrical activity on the scalp using a forward model to describe how known electrical activity should look on the scalp and an inverse model to infer where novel data would be localized to given the forward model. For a very short period, we were able to try out a piece of equipment that could capture the exact position of electrodes on an individual’s skull, thus allowing one to better estimate the forward model for an individual participant. This piece of hardware was essentially a much smaller and more limited version of the geodesic dome used to record the Holocaust survivors (see below).

The mothership

This got me to thinking, what if a much larger dome, similar to the dome used to record the Shoah interviewees were used to simultaneously record both EEG signals and an individual’s motion. Motion and position data could be used to both constrain a head model and also provide a means of potentially correcting head motion artifacts in the EEG data. There is some precedent to perform motion correction using camera tracking via a technique known as REEGMAS, and there has been recent work at Carnegie Mellon indicating that motion can be tracked effectively with a large number of cameras on a dome. If these feats of engineering were combined and elaborated upon, one could envision an experiment in which several individuals wearing EEG headsets could move about naturally within a confined space and communicate as they would normally in a real-world context. Overall, the addition of comprehensive and immersive audiovisual data could provide great insight into how we use speech and communicate.

If I put my mad scientist cap on, this also makes me wonder if in the future we might better be able to constrain NLP models based on the understanding we could gain from such a setup to be substantially more accurate and better convey a communicator’s thoughts, even after they have moved on from life. Or perhaps I’ve just watched the season finale of Black Mirror series 4 too many times.