In the 1940s, Maria spent a part of her childhood in Barcelona, Spain. Her memories of her father are still vivid.

When Maria was six years old, she would go to a neighbor's house in the same building whenever she wanted to see her father. This was because she could see the prison below through the balcony railings and catch a glimpse of her father through the small window of the cell. Her father was imprisoned there for opposing the government at that time.

There are no photos of this memory on the balcony, but now Maria can have something similar: a "fake photo." According to the Barcelona-based Domestic Data Streamers (DDS) design studio, this is a "memory-based reconstruction," which is to reconstruct the scene that a real photo might have captured in the memory.

Fake photos are blurry and distorted, but they can still instantly transport people back to that moment in memory.

"It's easy to tell whether the memory reconstruction is accurate, because it's a heartfelt response," said Pau Garcia, the founder of DDS studio. "This happens every time. People's reaction is, 'Ah! Yes! That's it!'"Currently, the "Synthetic Memories" project operated by DDS Studio has already transformed the memories of dozens of people into images in this way. The studio has used image generation models, such as OpenAI's DALL-E, to concretize people's memories.

Advertisement

Since 2022, the studio has been funded by the United Nations and Google, and has been working with immigrant and refugee communities around the world to transform scenes that have never been photographed into images, or to recreate long-lost old photos for those who have been displaced from their homes.

Now, DDS Studio is taking over a building next to the Barcelona Design Museum to record people's memories of the city with synthetic images. Garcia said that anyone can participate and contribute memories to the growing archive.

Synthetic memories may not only be a social or cultural effort. In the summer of 2024, the studio will begin to cooperate with researchers to study whether its technology can be used for the treatment of dementia.Unforgettable Graffiti

The idea for the project originated from an experience Garcia had in 2014 when he was in Greece working with an organization that was resettling Syrian refugee families.

A woman told him that she was not afraid of being a refugee, but she was afraid for her descendants to become refugees because they might forget their family history: where they shopped, what clothes they wore, and how they wore these clothes.

Garcia had volunteers graffiti the walls of the buildings where these families lived, depicting the woman's memories. "The graffiti was not good, but the idea of synthesized memories was born," he said.

A few years later, when Garcia saw what image generation models could do, he thought of those graffiti: "This was the first thing that came to my mind."Garcia and his team use a very simple process. They sit down with the interviewee and ask them to recall specific scenes or events. A prompt engineer then writes prompts for the model based on the memories, which generates images.

His team has established a vocabulary of prompt terms that have been verified and are suitable for evoking memories from different historical periods and locations.

However, Garcia says that prompts often need to be adjusted repeatedly: "You show the generated image to the interviewee, and they might say, 'Oh, the chair was on the other side' or 'It was at night, not during the day.' Then you have to keep modifying the prompt until it matches the interviewee's memory."

So far, the DDS studio has used this technology to preserve the memories of various immigrant communities, including Korean, Bolivian, and Argentine families living in Sao Paulo, Brazil.

At the same time, it has also collaborated with a nursing home in Barcelona to study how this memory reconstruction technology can help the elderly.The team collaborated with researchers in Barcelona to conduct a small-scale trial involving 12 subjects, applying this method to reminiscence therapy. This is a treatment method for dementia, aimed at stimulating cognitive abilities by showing someone images from the past.

Reminiscence therapy originated in the 1960s and has many supporters, but researchers are divided on its effectiveness and implementation methods.

Garcia said that this pilot project allowed the team to refine the process and ensure that participants could sign informed consent forms.

The researchers plan to conduct a larger clinical study in the summer of 2024 with colleagues from the University of Toronto in Canada, comparing the therapy using image generation models with other treatment methods.

One thing they found in the pilot was that if the images were printed out, the elderly would establish a better connection with the images.Garcia said: "When they see the picture on the screen, they cannot establish the same emotional connection. But when they can see it with their own eyes, the memory becomes more important."

Blurry is better

Researchers also found that older versions of image generation models work better than the newer ones. At the beginning of the project, they used two models launched in 2022: DALL-E 2 and Stable Diffusion. The latter is a free image generation model released by Stability AI.

The images generated by these models have flaws, including strange faces and distorted bodies. But when they switched to the latest version of Midjourney (another model that can create more detailed images), the results were not as popular.Garcia said: "If what you create is very realistic, people will pay attention to details that do not exist. On the contrary, if it is blurry, the concept of memory reconstruction will be better understood.

Memory is a bit like a dream, they are not photos, without meticulous details. You may not remember whether the chair is red or green, only remember that there is a chair."

Since then, the team has started using the old model again. Garcia said: "For us, flaws are a feature.

Sometimes, some things may exist (in memory), or may not exist. This is the quantum state in the picture, which fits the characteristics of memory very well."

Independent filmmaker Sam Lawton, who did not participate in this work, is excited about this project. What makes him particularly happy is that the team will study the cognitive impact of these images in a rigorous clinical study.Loudon once used image generation models to recreate his own memories. In a film titled "Expanded Childhood," which he shot in 2023, he used DALL-E to extend old family photos beyond the scenes, blurring the boundaries of real childhood scenes with surreal ones.

Loudon said, "The impact of these generated images on the human brain is what prompted me to make this film. I couldn't conduct comprehensive research, so I turned to the most natural way of storytelling for me."

Loudon's work discusses many issues: What impact does long-term exposure to images generated or altered by artificial intelligence have on us?

Can such images help reconstruct traumatic memories? Or do they create a sense of false reality, leading to confusion and cognitive dissonance?

Loudon showed his father the photos in "Expanded Childhood" and added his comments in the film: "I feel there are some issues, but I don't know what they are. Is it that I don't remember?"Garcia is aware of the dangers of confusing subjective memories with the true photographic record. He says that the reconstruction work based on memory completed by his team is not a "document of recorded facts."

In fact, he points out that this is another reason for insisting on using the old version of the image generation model to generate less realistic images.

Garcia said, "It is very important for people to be able to clearly distinguish between what is a synthetic memory and what is a real photo. This is a simple way to distinguish."

But Garcia is now worried that the companies behind these models may take down the old models. Most users look forward to larger and better models. For synthetic memories, the less detail, the better.

He said, "I am really afraid that OpenAI will shut down DALL-E 2, and we will have to use DALL-E 3."Please provide the article you would like to have translated into English.