@lacuna I dunno, maybe this?
matrixMixerMaybe.zip
Look at the toggle sends to see the modulation input names. There are signal rate receives on the other end, but I just used toggles to test. The whole thing probably accrues 2 blocks of latency along the way, but I didn't test.
Edit: I don't understand why, but I'm not getting any delays. Did I create the objects in some magical order?
Edit2: What's the matter with me? The modulation receives are control-rate. But you can easily change them to audio-rate and ditch the toggle matrix I used to test.