Note to impacted UK customers

A subset of our UK based customers have been impacted by the recent work. Their service has been running at the speed of an intoxicated snail. (Yes that is accurate). As a result we’re sending the following message to them and applying the discount mentioned. The letters will go out later today and the discounts applied at the next billing cycle.

Lima Labs would like to say:

WE GOOFED and we are sorry, enjoy 50% off for the next 3 cycles on us.

What does this mean?

A global federation of CEPH clusters with overlay caching is a complex beast. It’s resilient but complex.

About a month ago someone (me) made a simple error inside of our storage system that caused a number of disaster recovery processes to start. Data that shouldn’t have left North America under normal conditions was suddenly being served from the EU. The mistake was simple, some misnamed OSD devices and data pools the results were not.

We’ve unwound the mistake but doing so required massive back end work to correct. We had thought through how to recover from a full DR swap but a condition where data was co-mingled just wasn’t in our map.

In the process we discovered some failing hardware in the UK (SSDs which were set to die LONG before their time) and took advantage of the outage to replace them. This added extra time to the work but it needed done.

So what about the 50% off?

Expect a discount of 50% on your NEXT 3 invoices from Lima Labs. It’s our way of apologizing for the snail speed of our UK pop during this work.


When will I see it?
You will see the discount NEXT billing cycle and it should persist for 3.


Are you all fixed?

Data is in the right place, CEPH is still re balancing in Canada and the UK, the caching tier is coming back online this morning. It will take some time to properly fill but the second time you hit your data it should now be much quicker.

1 thought on “Note to impacted UK customers”

  1. I was just one of those customers impacted, I realized the slow speeds as funny as it sound very quickly.

    Thank you for your honesty and for your dedication to your work, a great wave of gratitude and appreciation from Germany, and by the way, nobody’s perfect and nobody needs to be.

    I’m working as an IT professional and IT trainer now since almost 30 years, I know how much work is evolved and how complex IT sometimes is.

    Kind regards,
    Robert

Leave a Reply

Your email address will not be published. Required fields are marked *