elevated error rates on convex for pro instances

At approximately 6am Pacific we had a high rate of server errors for approximately 30 minutes for a subset of customer backends. This was caused by a 20x increase in traffic that saturated one of our databases and led to resource exhaustion on one of our function execution clusters, which is why the impact spilled beyond just that specific database. We’ve scaled up our infrastructure further and are deploying some changes to load shedding and retries to minimize the impact of such an event in future. While we aim to have no cascading impact on other customers from a load spike we’re also deploying dedicated isolated hardware for customers on our upcoming Business plan.