if that patch is the way you're doing it, then you might be running into the same problem another person was talking about the other day - that pd's blocks are not always in sync.
also, you probably also want to send one block of zero's in to clear the audio buffer before switching, so i think it's generally a good idea to have 2-3 blocks of headroom between the line down time and the block switch.
if THAT doesn't help, then i can unfortunately suggest that you might be asking a bit much of the iOS and iPad to go down to blocksize of 16. try it with a block of 64 and see what happens.
There might be another reason too, of course...that's just all i can think of right now.
OH, unless of course you're running No GUI, in which case, there is a quite qell documented bug of delays not being set properly. Not sure if that has been fixed since i last checked, but if not, it might be affecting [pipe] as well. From memory, i think the easiest workaround was to dynamically turn dsp on and off and on again, but again, i can't be sure.