We replicate data from our UK based SCP server group to our French storage servers as a disaster recovery function. However that DR function just bit us in the butt. The French datacenter where the replication goes to burned over the weekend. This didn’t immediately cause problems but when the replication queue on the UK side grew beyond a safe limit a number of our filesystems went into read only mode as a safety feature.
We’re working now to clear the replication queue (The remote end is toast literally) and remount the filesystems read/write. Thanks to the eagle eyed customer that spotted this it shouldn’t take long.
Update: All impacted servers should now be operating properly.