object tracking --> audio generation? Synesthesia Machine

Discussion of modular and standalone video generating/processing techniques and associated hardware.

Moderators: Kent, Joe., analogdigital, infradead, lisa, parasitk, plord, daverj, lizlarsen, nickciontea

Post Reply
User avatar
rayjohnson
Common Wiggler
Posts: 75
Joined: Wed Dec 14, 2016 4:08 pm

object tracking --> audio generation? Synesthesia Machine

Post by rayjohnson » Sun Dec 18, 2016 4:05 pm

Hello,

I'm thinking about starting a software project that takes video and generates music from the video.
one of my main ideas is tracking objects and using their detected parameters to relate to musical parameters.
the larger idea is a 'synesthesia machine', a framework of ideas and collection of software modules to patch between the audio and visual domains.

anyone see anything like this idea? i think it's different than most visualizers out there.

Thanks!
I made a fake radio show:

http://www.houseboatmusic.com/vpr/

User avatar
makers
Veteran Wiggler
Posts: 653
Joined: Wed Mar 20, 2013 1:54 pm

Post by makers » Sun Dec 18, 2016 7:37 pm

David Rokeby's "Very Nervous System" immediately comes to mind. It's a different application of object tracking but certainly worth investigating. I haven't followed its development in years but there was a version of the code called SoftVNS for an old version of Max.

http://www.davidrokeby.com/vns.html

User avatar
nathanxl
Pickup Truck Full of Forty Minutes
Posts: 509
Joined: Wed Jun 22, 2011 6:37 am

Post by nathanxl » Mon Dec 19, 2016 8:00 am

The IOS app called PhonoPaper, when used in freemode, converts the signal coming in from your phone camera into audio.
Check it out:
http://www.warmplace.ru/soft/phonopaper/
https://nathanjohnthompson.wordpress.com
"The risk of reverse engineering is that we come to imagine that the only things worth knowing about products are the technical details hidden behind the curtain. I argue that the cultural lives and imaginations of the people behind the curtain are as important, if not more, for understanding how these systems come to exist and function as they do."

User avatar
Annie Socoria
Learning to Wiggle
Posts: 42
Joined: Fri May 13, 2016 8:03 pm
Location: Ottawa, Ontario
Contact:

Post by Annie Socoria » Mon Dec 19, 2016 10:04 pm

I was looking for something along these lines to generate audio from the results of my video feedback sessions. A visualizer in reverse of sorts. Haven't found anything that quite does what I had in mind and I lack the skills at present to make this. Given the complexities of synesthesia it would be interesting to see how a video image could be interpreted as sound.

User avatar
makers
Veteran Wiggler
Posts: 653
Joined: Wed Mar 20, 2013 1:54 pm

Post by makers » Tue Dec 20, 2016 10:34 pm

nathanxl wrote:The IOS app called PhonoPaper, when used in freemode, converts the signal coming in from your phone camera into audio.
Check it out:
http://www.warmplace.ru/soft/phonopaper/
Right, and while your at Warmplace site check out VirtualANS. The original ANS synthesizer is certainly worth investigating in this context as well as it's namesake Scriabin.

https://en.m.wikipedia.org/wiki/Alexander_Scriabin

Post Reply

Return to “Video Synthesis”