DHCP broken in a part of the datacenter

After a software upgrade and reboot of the AR-114-9 modules, dhcp relay from the Picos network is broken. The cause is still not 100% clear, we will update this incident when we know more.

We tried setting up dhcp servers within the picos network, but this didn’t solve the issue. We now suspect the picos switch itself, which we will try to restart in the morning of June 6th (between 7:30 and 8:30). This will affect all connections behind the picos switches. We hope the downtime will be less than 10 minutes.

Affected clients are mostly vms on csvmh* machines, a workaround is to configure static addresses on the clients, effectively bypassing dhcp.

update June 6th

The reboot of the picos switches (PicOS is the network operating system of the whitelabel switches we have for 25Gbit/s network) didn’t solve the issue.

We solved it by using broadcast dhcp instead of relayed dhcp