Add alerting #59

New issue

Open

opened 2026-09-27 00:21:33 +02:00 by Marcel · 4 comments

Marcel commented

2026-09-27 00:21:33 +02:00

Owner

e.g. alertmanager

failed systemd units
items that are longer that 4h in the postqueue
disk is at 80%
healthcheck of alerting to status.elbforge.org

e.g. alertmanager - failed systemd units - items that are longer that 4h in the postqueue - disk is at 80% - healthcheck of alerting to status.elbforge.org

Thomas Liske commented

2026-09-27 12:42:09 +02:00

Owner

@m4rc3l wrote in #59 (comment):

disk is at 80%

This is not a sensible threshold in general. A relative threshold could make sense for small filesystems but it might not be useful for large filesystems. More important than the free space (which I would set to at least 90% for most use cases) is the grow rate.

Please also add the the inode usage!

healthcheck of alerting to status.elbforge.org

The status page should contain outages or performance issues… the alerts from above are not (directly) related with such issues 🤔

@m4rc3l wrote in https://elbforge.org/elbforge/elf-infra/issues/59#issue-444: > * disk is at 80% This is not a sensible threshold in general. A relative threshold could make sense for small filesystems but it might not be useful for large filesystems. More important than the free space (which I would set to at least 90% for most use cases) is the grow rate. Please also add the the **inode** usage! > healthcheck of alerting to status.elbforge.org The status page should contain outages or performance issues… the alerts from above are not (directly) related with such issues 🤔

Marcel commented

2026-09-27 12:59:27 +02:00

Author

Owner

The alerting system calls a reverse status check at status.elbforge.org so that we enshure the alerting is running. Like it's done with the route servers at dd-ix

Göran Heinemann commented

2026-09-27 22:55:31 +02:00

Owner

maybe also something along the lines of "more than 20 signups in the last minute" (as soon as signups are live)

Tim Koop commented

2026-10-17 22:44:17 +02:00

Owner

HAProxy:

unhealthy/down backend/frontend
huge spike/rise in requests for 10 minutes
more than usual 5xx responses
response time longer than usual for backend

HAProxy: * unhealthy/down backend/frontend * huge spike/rise in requests for 10 minutes * more than usual 5xx responses * response time longer than usual for backend