JVCKENWOOD Demonstrates Brainwave-Activated AI-Driven Music Composition and Video Creation at CEATEC 2024

In CEATEC 2024, we visited the JVCKENWOOD boothconsisting of three themed areas around “Connecting people, time, and space to create the future.” This is in line with the company’s philosophy of “bringing happiness and peace of mind to people around the world” through cutting-edge technology to enrich lives. JVCKENWOOD is shown video, audio, and wireless solutions designed to create new value for the future. First place, written by the company “Impression – A place of entertainment,” * featured technical exhibits that engage all five senses—sight, hearing, touch, smell, and taste—providing an immersive, multi-sensory experience.

AI Creates Music and Video Based on Your Subconscious Emotional Response

in JVCKENWOOD booth’s “Impression Entertainment Area,”* we had a new demo that combines brain wave detection, human senses, and AI to create personalized music and video. Participants are exposed to stimuli, such as pre-programmed videos, designed to evoke emotional responses. The brainwave headset then receives the emotions generated, and AI uses this data to record music and project accompanying images in real time. The result, described as “the music and video you created,” is played back. AI also differentiates the participant’s passive reactions, automatically generating music and images that reflect their emotional state, providing a unique interactive experience.

How Does the Technology Behind AI-Generated Music and Video Production Work?

Suguru Goto, Associate Professor in the Department of Music and Visual Arts at Tokyo University of the Arts, present a demo developed by a research team with the support of JVCKENWOOD. Suguru Goto, who is also a composer and storyteller, demonstrated the innovative use of computer communication technology to create art in the JVCKENWOOD space.

How Did the Demo Work?

The team used the The Emotiv Insight EEG headset is designed to help researchers and engineers create new solutions for brain-computer interaction in a variety of fields. Having personally tested several Emotiv products in their early days, I can attest to the robustness of brainwave-powered hardware tools, which have become trusted solutions for neuroscience-based interactive development.

When we got to the demo, the headset was already limited to the participants’ brainwaves. According to Suguru Goto, the system only needs a few minutes to balance each new user. The process began with a short meditation session, followed by the participant watching several pre-prepared videos designed to evoke specific emotional responses.. An Emotiv Insight headset recorded brain waves during these activities while AI-driven software interpreted the data.

Suguru Goto explained that brain waves are analyzed using machine learning algorithms, which classify the data into 28 emotional states. These regions are plotted on a Cartesian plane, with the X-axis (horizontal) from discomfort (unpleasant) to happiness, and the Y-axis (vertical) from wakefulness (arousal) to sleep. He also showed me a tablet showing a graphic representation of how the software further divides these 28 emotional states into 64 episodeswhich provides a more detailed view of the participant’s emotional responses.

The 28 emotional states, divided into 64 categories, are arranged on a Cartesian plane, with the X-axis (horizontal) from discomfort (unpleasant) to happiness and the Y-axis (vertical) from alertness (arousal) to sleep .

After the participant completes the video sessions, the system automatically created music and video based on his emotional responses. The demonstration highlighted how brainwave data, combined with artificial intelligence, can create action-based personalized audio-visual content, giving a glimpse into the power of brain-computer interaction technology in the creative fields.

For the demo at CEATEC, i sound was played through JVCKENWOOD wooden speakersonce The AI-generated video was shown on the big screen.

Concept art for the project, share with me in private video, it involves showing emotional videos on a monitor facing the participant, while AI-generated visuals are projected onto the floor of a darkened room where the person is sitting.

Beyond AI-based Art: Unlock Unlimited Possibilities for Human Life

Anyway this is an interesting art research project, Professor Suguru Goto revealed that it offers great potential for health-related applications, especially mental health care.

I know that Emotiv’s main purpose is to provide EEG-based hardware and software tools so that researchers can develop solutions that improve the mind and mental health. This includes creating new methods of music neurofeedback therapy.

JVCKENWOOD SP-WS02BT wooden speakers – sound reproduction was good

In addition to video stimuli, other sensory inputs—such as sound, taste, and smell—may also elicit emotional responses in participants. For example, a participant can hold a JVCKENWOOD “Hug Speaker” (see picture), which vmoving in sync with the sound, or eenjoy the smell of wood from speakers made of different types of natural wood. These stimuli can be accompanied by AI-generated video and music created with brainwave data.

“Hug Speaker” – photo courtesy of JVCKENWOOD

Although we didn’t see a demo of the other sensory stimuli in the booth, I’m hopeful that they will work seamlessly with the same demo setup and AI-driven software, creating images and music based on the participant’s unconscious emotional response to this sensory stimulation.

* Editor’s note: “Impression – Entertainment Area” English version of Google based on JVCKENWOOD Japanese Press Release

Filed Audio > Design > Image-Video. Read more about AI (Artificial Intelligence), Brainwave, CEATEC, Ceatec 2024, Eeg, Emotiv and Japan.