svc-adm01: split IXP deployment units #91

2024-12-08T18:33:27+01:00

liske commented

2024-12-08 18:33:27 +01:00

(Migrated from github.com)

Split IXP deployment into independent units:

split build from deployment @liske
deploy the route servers with some safety time gap @liske
use template units for individual host units @liske
add conditionals to disable deployments on demand (systemctl disable doesn't work in our setup) @liske
add some handy CLI script for systemctl status + enganging kill switches (touch /var/lib/arouteserver/kill/$NAME) @liske
implement some exporter config (for alerting) for engaged kill switches in /var/lib/arouteserver/kill/ @MarcelCoding

Split IXP deployment into independent units: - [x] split build from deployment @liske - [x] deploy the route servers with some safety time gap @liske - [x] use template units for individual host units @liske - [x] add conditionals to disable deployments on demand (`systemctl disable` doesn't work in our setup) @liske - [ ] add some handy CLI script for systemctl status + enganging kill switches (`touch /var/lib/arouteserver/kill/$NAME`) @liske - [ ] implement some exporter config (for alerting) for engaged kill switches in `/var/lib/arouteserver/kill/` @MarcelCoding

MarcelCoding commented

2024-12-08 18:37:27 +01:00

(Migrated from github.com)

Should the deployment automatically be disabled if the other one (routeserver or switch) failed? Maybe also with some sanity checks? (e.g. verify that the bird rc unit is running?)

liske commented

2024-12-08 18:45:02 +01:00

(Migrated from github.com)

Should the deployment automatically be disabled if the other one (routeserver or switch) failed? Maybe also with some sanity checks? (e.g. verify that the bird rc unit is running?)

Configuring switches is almost independent from the routers (see below) in this context. The deployment services will not run if the ddix-ixp-build.services has failed or was not run, yet.

(Yes some more sanity check should be added inside the ansible deployment like the use of BGP session culling before disabling interfaces. But that is out of scope for the nix-config.)

> Should the deployment automatically be disabled if the other one (routeserver or switch) failed? Maybe also with some sanity checks? (e.g. verify that the bird rc unit is running?) Configuring switches is almost independent from the routers (see below) in this context. The deployment services will not run if the `ddix-ixp-build.services` has failed or was not run, yet. (Yes some more sanity check should be added inside the ansible deployment like the use of BGP session culling before disabling interfaces. But that is out of scope for the `nix-config`.)

MarcelCoding (Migrated from github.com) reviewed

2024-12-08 18:47:46 +01:00

hosts/svc-adm01/configuration.nix

					
				@ -88,0 +69,4 @@

				          User = "ddix-ix-failed-notification";

				          DynamicUser = true;

				        };

				        script = ''

Maybe move that into a variable and reuse it.

MarcelCoding commented

2024-12-08 18:59:40 +01:00

(Migrated from github.com)

Configuring switches is almost independent from the routers (see below) in this context. The deployment services will not run if the ddix-ixp-build.services has failed or was not run, yet.

I mean that the rs02 should only be deployed if rs01 successes the deployment.

> Configuring switches is almost independent from the routers (see below) in this context. The deployment services will not run if the ddix-ixp-build.services has failed or was not run, yet. I mean that the rs02 should only be deployed if rs01 successes the deployment.

liske commented

2024-12-08 19:06:35 +01:00

(Migrated from github.com)

Configuring switches is almost independent from the routers (see below) in this context. The deployment services will not run if the ddix-ixp-build.services has failed or was not run, yet.

I mean that the rs02 should only be deployed if rs01 successes the deployment.

It doesn't matter if the deployment did succeed of the other RS. We must not stop deploying if one of them failed due to hardware failures etc. The deployment won't fail if there are semantic issues in the peering LAN.

> > Configuring switches is almost independent from the routers (see below) in this context. The deployment services will not run if the ddix-ixp-build.services has failed or was not run, yet. > > I mean that the rs02 should only be deployed if rs01 successes the deployment. It doesn't matter if the deployment did succeed of the other RS. We must not stop deploying if one of them failed due to hardware failures etc. The deployment won't fail if there are semantic issues in the peering LAN.

Sign in to join this conversation.

No reviewers