Here are two patches I made for sonifying pictures and videos. I've tried to label the controls to make them easier to use.

Pictures (patch attached to this post) - this one makes sound from the RGB pixel value the mouse is hovering over. It was made to be controlled with a Wiimote but you can just use a mouse.

Videos (patch attaced to next post) - this one makes sound from the average HSV values per video frame. Also uses blob detection to sonify the spatial location of objects.

They both output MIDI too so you can send them out of Pd for sound (which is especially good for the video one because it struggles with sound and video rendering at the same time)

You can download reports with details of the patches and how they work, video demos and the GlovePIE script for the Wiimote controller here: http://alexjonesaudio.co.uk/sonification.html

http://www.pdpatchrepo.info/hurleur/Image_Sonification_Pd_Patch.pd

Video Patch

http://www.pdpatchrepo.info/hurleur/Video_Sonification_System.pd

These are fantastic

Hi there, these patches are really interesting, and I also read your paper on your website which was really fascinating, especially that you are applying this work to the study of glacial movements and evolution. I was also hoping to test the video patch with some slowly evolving phenomena, but unfortunately every time I open a video, and the GEM window, pd crashes. Here are some particulars:

-firstly, the photo patch works fine

• the video patch : it is not a general problem with GEM, because I can use the pix_film object that you use here by itself and open a wide variety of formats

• I have tested .avi files, quicktime mov files, and mpeg4 all with the same results

-I am on Windows 7, most recent version of PD/GEM. graphics card is not great but as I said before there are no problems to use the same objects alone, and playback is achieved, I can even manipulate the videos in real time, no crashes...just this patch, as soon as I open the rendering window AND a video (the rendering window -that is, the GEM window- opens fine without anything loaded) PD immediately crashes.

any ideas why that may be or what I could do to prevent it? I am very interested in how you approached this...

J

@aj647 Very interesting patch. If you are still here (because you posted it long ago): Is it possible to find the reports about the patch somewhere? Because the links you posted are not working anymore.

And sadly, after installing the required libraries I have the same problems as @Coalman had, so the video patch is not working for me. But I think I can understand the concept more or less from the patch.
I try to build something similar with ofelia and like to understand different approaches of video sonification and audio visualisation. So I am also interested in other patches that do this kind of things.