Analysis of the microphone signal is a very interesting topic. Feature extraction, voice recognition, music analysis, all these things would be amazing to include into the work we're doing and hopefully one day we will. I think there are still a few issues that have stopped us from fully incorporating it in yet. Mobile devices are only now just to getting to the point where they are capable of handling the processing required to run all of this at once in realtime. Don't forget that it also has to run an app, graphics, animations, logic, musical synthesis, sample playback, etc... Cheap implementations obviously have crude results and the ideal algorithm is always more complicated.
It's just a matter of time before:
a) the processing headroom on mobiles gets to a point where it's less of an issue (although you always have to push the limits )
more work is done on efficient, easily implementable algorithms for analysis
Therefore, in actuality our analysis methods are still fairly simple, we do onset and pitch detection as standard in all the Dimensions. We use bare bones vanilla Pd with some hardware optimisations and a readsf capable of reading AAC files for memory limitations.
@BerengerRecoules said:
I am also interested in the different interactions possible through the phone, what make the sound evolve (options on the touchscreen, accel, GPS ? camera ?), in short how did you use whas was available on the phone ?
It's interesting you should say that because in the early days we worked on utilising the compass information to pan objects (semi)binaurally around the user. It worked really well, however, we wouldn't have been able to support iPod Touches as they don't have a compass. Trying to add a combination of touch and compass interaction proved confusing and we didn't want to alienate a potentially large user base. It was a shame as it was quite fun!
The majority of our time was spent on the sound design. Trying to meld the incoming audio input into an effect that works with a musical piece and is understandable can get quite tricky. You have to also remember that you have no control over the microphone signal. It's basically a very rubbish mic with limited frequency response, feedback problems and the potential to have essentially white noise come through.
Some of the more interesting effects have been sampling the signal to buffers, then processing them with reversed reverbs or pitch changes. It can get quite scary!