Ok we’ve been struggling with another issue that appeared to start at the same time that our Canadian goof happened last month. We didn’t understand it well until this morning.
We have a LOT of people that buy and resell our space. For some reason in the last 30 days IO rates have gone through the roof across the board. We don’t look at the data just the IO numbers and we assumed this was all connected to our hardware issues and our Goof last month.
Turns out what cooked our SSDs and helped lead to this month of slowness is something called Chia coin which until this morning people were using our storage service to “Farm”. I’m working with these clients now to move their Chia farms off of our service. This isn’t the place to mine crypto we’re here to store real data.
For more info on this:
UPDATE: WOW what a difference that’s made. Chia Coin had apparently been the source of a lot of the lets say clobberage that we were seeing. We’re still evicting them and performance is improving with each one.
UPDATE: 2:20PM EST. We’re now confident enough we’ve found the issue to again start adding customers.
We’re going back to 5/22 and working forward in the order requests were received.
UPDATE 4:00PM EST. I think we’ve gotten the bulk of them off of the Canadian servers. The six slow servers in Canada will remain offline until we can get some SSDs in them swapped, as a result we’re re balancing cache AGAIN but this is going fairly smoothly. Speeds should improve as the night goes on.