Service Error Rate Increase
Incident Report for Tipser

On August 31, 2021 21:53 UTC, our monitoring informed us of an unusual 504 error code that originated from the Checkout API.

Upon further investigation, we discovered a timeout configuration error between our CDN and load balancer. This could have led to a situation where the request was still being processed by our API, but the CDN had already returned a 504 error message. These errors were hidden from our monitoring because the request was processed successfully, but the response never reached the customer.

The fix was introduced at 14:50 the following day. It was also including alerting enhancements.

All endpoints were affected but only small portion of requests.

Posted May 13, 2022 - 13:14 CEST

We have noticed increased number of 503 returned by our service
Posted Sep 01, 2021 - 17:00 CEST