Availability of some store and admin pages impacted by a code release

Incident Report for Roadster

Resolved

A code change was pushed that had an unexpected impact on our client side analytics which caused some store pages and the dealer admin login page to fail.

Our internal monitor caught the issue ahead of time in our staging environment, but it did not prevent the push from proceeding. Once it was in production, our monitors caught it right away and we tested and deployed a fix which was live within 30 min.

We are working on updating our deploy procedure to ensure internal monitors can halt a deployment before it hits production environments to prevent this from happening again in the future.

Posted Aug 31, 2020 - 22:30 UTC