We became aware of the Kernel vulnerability and implemented the kernel vulnerability mitigation
The change was to set
kernel.yama.ptrace_scope = 2
(A sysctl setting). For users of MPI jobs, this setting broke the jobs. When we have a fixed kernel and rebooted the nodes, we can change this back to something that will work with MPI.
Workaround or fix, use mpirun --mca btl_vader_single_copy_mechanism none etc...