Hello everyone!

I wanted to use the Pure Data to create a desktop or picture sonification tool (dynamic conversion of the graphic properties to sound). For now my main question is whether there is a way to get the graphical data from the screen (get something like printscreen) and work with them using GEM. Right now I was only successful with loading an individual images using the pix_image.

I guess that I might need some external script for this, or do you have any other ideas?

@MarSam Hi,

I don't know any external that takes a screenshot, but you could use [shell] to launch a command line that do just that and then load the screenshot in Gem -> sonofication.

Cheers

Thank you @EEight, I am running the PD-extended on the Windows 8.1. I cannot get the shell object running (seems like my PD does not know it, I read elsewhere in the forum that it does not work under the Windows). I found some Windows Powershell scripts for taking screenshots, but that might be rather slow if I want the sonification to be dynamic.

Is it possible to run some C or C# code from PD patch? I found some C# code that can directly determine the color of the pixel under the mouse pointer. I have experience only with Java and PHP but I think this might be better solution.

@MarSam It's possible to code an external in python, lua, c, c++ and other languages, but it's not an easy task. Maybe another approach should be considered here: using pd only for the synthesis; the screenshot / pixels part in another program (processing, c# etc) sending the data via open sound control.

Cheers~

@EEight That's right, maybe I'll try Python. I decided to make demo in PD capable to sonify only pictures. I have one issue though. I am looking for a function that will return me the color of specific pixel under the mouse pointer (in GEM window). Right now I am using [gemmouse] to determine the mouse position in the window, but I don't know how to get info about the pixels on the mouse position.

Have you seen these?

https://github.com/hellocatfood/pixel-waves
http://www.hellocatfood.com/create-jpgs-in-pure-data/

Probably not exactly what you're looking for, but also worth exploring.

Hi MarSam

on OSX you could use camtwist to capture the screen and feed it to gem as a virtual webcam. I don't use Windows at the moment, but a quick research showed me this:

http://filehippo.com/de/download_many_cam_free/

maybe that is at least some hint for a possible direction. I would be very interested in your resutl!

All the best,

johnny

@MarSam Use [pixelinfo]. Take a look at this patch, it does mostly what you asked for (but using .jpg, you can replace with pix_video and many_cam_free as suggested by Johnny Mauser).

Cheers~

Thanks @EEight @LiamG & @Johnny Mauser for very useful suggestions, I will give them a try!

I've progressed a bit, here is what I could do so far. Thank you all for your advices and tips, it really helped me!:

Desktop graphics sonification using ManyCam:
https://www.sugarsync.com/pf/D6163802_05628859_7827373

The patch uses ManyCam (or similar software, tip above by @Johnny Mauser thanks again!). The desktop is fed to the GEM. Mouse position is extracted and it's position used to sonify pixels under mouse pointer.

The patch still needs a lot of work, especially to tune the waveforms for the individual colors (RGB). There are some other things like clipping output etc. that need to be fixed. I need to move on with my research project right now, but I'll post the updated version whenever they'll be ready.

Known glitches:

• Y position is incorrectly reflected to the GEM window, X position is ok. I need to dig into this if this is gemmouse issue or something else.

• Sound clipping, not much time to fix this right know.

• Lots of other small things

• White pixels sound is too loud and clips the output (max amplitude for all color synths, this probably needs to be adjusted or somehow limited maybe by expression that will reduce the output if all amplitudes are max.