@whale-av Hi David,
This is what we've built so far. I would not have been able to figure this out on my own.
I am pretty sure there are many more redundant bits.
having extra eyes on it really helped.
Use your computer as a midi/data controller.
Modulate channel independent control values using your mouse/trackpad.
Send note velocity data from your computer keyboard.
Required Pd externals include cyclone/MouseState and you will also need to download TUIO_Mouse by Martin Kaltenbrunner here: https://sourceforge.net/projects/reactivision/files/TUIO 1.0/TUIO-Clients 1.4/TUIO_Mouse-1.4.zip/download
colibri.pd is the main patch and deals mostly with mouse pointer position, the ki.pd abstraction with note vel. from the keyboard.
Disclaimer: this is an experimental release. use at your own risk.
if you have time to review the code and find anything that could be done more efficiently please bring it up, any feedback would be appreciated.