We've switched to my tool to measure latency and availability of our databases at work. This brand new tool, not yet open source, leverages the power of Prometheus histograms to measure those metrics finely.
I had to use DNS names instead of IP addresses to reach the endpoints, because it was easier to use (label = name).
But, the farthest the infrastructure was, the more degraded the measures were. Everything ran locally so what the hell? The level of degradation was not random, it was equal to the light distance to France.
It was DNS. It's always DNS.
The private zones were not deployed on the local resolvers, so we had a 200ms round trip in Australia, to reach resolvers in France, before actually connecting to the database.
The DNS cache didn't help at all.