@KoaN Thank you for posting what you have discovered. It will surely help someone in the future.
In windows you can start 2 instances of Pd.... one for audio and one for midi.... which can help as although the audio settings do affect the midi lag you can set one of the instances to run with a tiny buffer that would cause audio dropouts..... but use that instance purely for midi.
This can be done in other os's too.
The 2 instances can then communicate control rate messages using [netsend] and [netreceive].
A steep learning curve but you seem to be advancing rapidly.
The instances can be opened with a pre determined setup using command line flags.... http://msp.ucsd.edu/Pd_documentation/resources/chapter3.htm#s3.4.1
Here is an example of a batch file that should achieve this.....
start "C:\Users\David\Desktop\Pd Vanilla\pd-0.55-2.msw\pd-0.55-2\bin\pd.exe -noaudio -midiindevice 2 -midioutdevice 2" start midi_patch.pd
start "C:\Users\David\Desktop\Pd Vanilla\pd-0.55-2.msw\pd-0.55-2\bin\pd.exe -r 44100 -asio -nomidi -audioindev 9 -audiooutdev 9 -inchannels 18 -outchannels 22 -audiobuf 2 -blocksize 64 -callback -nosleep" start audio_patch.pd
exit
The paths will be wrong for your computer, so will need to be changed, and the command line flags will be wrong for your sound card, and probably for the midi devices too. For the sound card and midi devices you should first open your patch and open the Media AudioSettings and Media MidiSettings and write down the device numbers and settings for those found on your system as Ps starts up.
Then create an audio patch (-nomidi) and a midi patch (-noaudio) and change the command flags to match your setup for each.
BUT
I cannot (so far) get the flags to work in a batch file opening 2 instances (works fine for one instance so it is my batch programming problem).
So copy the code above into a simple text editor like notepad, save it as e.g. "pd_twice.bat" and then edit it to match your Pd binary location and the Pd files you want to load to each instance. Once the 2 instances are open set the audio and the midi for the instances to something like this and do not turn on DSP for the midi instance.......
Keep the batch file with the Pd files you are loading in the same folder.

Of course your settings for the audio and the midi instances will be different..... different devices, different delay for audio.... but the midi instance can have the delay set to 1 (or maybe even 0) as it will not be running any audio.
As @jameslo says above...... in Pd the "Delay" setting is the buffer for communication with the sound card.
David.