Check out the smokeping_prober. It generates high resolution histogram data that you can use to compare different paths.

I say application monitoring is the key thing mainly because what we actually want to know is, is the system as a whole functioning correctly? Without positive evidence that the application is succeeding we can’t tell for sure.

So true.

The problem was at the time when Nagios was introduced the only metrics system we had was SNMP. SNMP was kinda ok to get into with the introduction of Cacti. But due to the complication of ASN.1, the split between the actual data and the MIBs, and the lack of good documentation on how to implement it properly, it was not a good solution to extend. Adding application metrics in SNMP? Good luck with that.

Thankfully then came statsd, and then Prometheus, to make it much easier to implement application monitoring, as well as integrate system and network monitoring with application monitoring.

rsyslog/syslog-ng are simple enough.

But logging and monitoring/alerting are different things. Logging is not great for alerts. You want an actual monitoring system for that.

Prometheus is not that complicated, has a huge community of integrations, and is extremely flexible for monitoring everything.