This patch seems to indicate the same: delay-test.pd
Instead of adding up, the differences roughly stay within the 5 ms margin.
For now my conclusion would be that delay and pipe are ok, but one should not rely on time measurement too much.
The source of the error could be realtime, os or hardware, which is still not determined in my view.