0
Fixed

An explanation of last week's outage

Kevin Holmes 8 years ago in Known Issues updated by Jessica Sommerfeld 7 years ago 1

Hello everyone,


I am writing to provide an explanation for the outage we experienced last Wednesday afternoon, beginning at approximately 4:40 p.m.


While we have made significant upgrades to nearly every part of our infrastructure over the last 90 days, and we have seen marked improvement in our overall performance and user experience during that time, the cause of the problem last Wednesday was not connected to any of these recent changes.


Instead, the outage was caused by a hardware failure on our main database. You may find it interesting, or odd, or just plain unlucky given our recent performance issues -- but this particular problem has never occurred before in the history of Procentive. The hardware in question had been upgraded in 2016, as well.


Because the problem was connected to the main database, our systems engineers exercised additional caution in bringing the system back online, which is why it took nearly an hour before Procentive was accessible to customers again.


Our engineers have made the necessary fixes to the system and added extra consistency checks so we can meet our optimal recovery time should this happen again (which should be unlikely).


We apologize for this and other recent performance issues. Please know that we care about you, and your staff. We care about the work you do. And because of this, we will continue to work hard to make Procentive the best it can be now, and far into the future.


If you have any further questions, please let us know.


Thank you,


Kevin Holmes

Director of Product Services