tag:tipserstatus.com,2005:/historyTipser Status - Incident History2024-03-28T03:53:08+01:00Tipsertag:tipserstatus.com,2005:Incident/186128242023-09-10T01:30:00+02:002023-09-26T17:44:42+02:00Transient API connectivity issues<p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>01:30</var> CEST</small><br><strong>Resolved</strong> - Our CDN provider had temporary connectivity problems in the north-eastern US region, which resulted in some requests not being handled. The problem lasted several minutes.</p>tag:tipserstatus.com,2005:Incident/147334342022-12-07T04:30:00+01:002023-09-26T17:47:51+02:00API unavailable<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>04:30</var> CET</small><br><strong>Resolved</strong> - Our API became unavailable due to a fault infrastructure deployment.<br />At the time of the incident, all endpoints under api.tipser.com were unavailable.</p>tag:tipserstatus.com,2005:Incident/117462792022-10-09T13:30:00+02:002022-10-09T18:25:36+02:00On October 9th, between 13:38 and 13:43 UTC our CDN provider had trouble connecting to our backend.<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>13:30</var> CEST</small><br><strong>Resolved</strong> - As the result all our APIs were partially unavailable - 27% of requests sent in this period failed.</p>tag:tipserstatus.com,2005:Incident/111530892022-09-15T22:00:00+02:002022-09-21T12:30:45+02:00API outage<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>22:00</var> CEST</small><br><strong>Resolved</strong> - On Thursday, September 15, 2022, between 19:50 and 20:10 UTC our APIs became unavailable, as HTTP clients were unable to verify a TLS certificate securing a connection to our endpoint.<br /><br />It was a transient incident caused by our CDN provider:<br /><br />"On Thursday, September 15, 2022 at 17:44 UTC, Fastly deployed a configuration change to how fallback entries in our TLS configurations are generated. This led to incorrect intermediate certificates being referenced in non-fallback entries. This would have resulted in 5XX errors referencing inability to fetch local issuer certificate. This change was reverted, mitigating the incident.”</p>tag:tipserstatus.com,2005:Incident/108375782022-06-06T14:30:00+02:002022-08-16T15:41:11+02:00APIs outage<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>14:30</var> CEST</small><br><strong>Resolved</strong> - At 12:25 PM UTC our APIs became unreachable. The alerting immediately triggered a response of the SRE team.<br /><br />The cause was quickly determined - a recent change in an infrastructure code caused a removal of API services from our production cluster. It didn't happen on a development environment where the change was tested due to differences in the environments' setup.<br /><br />The fix was developed and tested on our main service. As a result our /v4 and /v5 endpoints became available again at 12:32 PM UTC. At 12:47 PM UTC /v3 and the rest of the services were operational too.<br /><br />To make sure that mistakes like this one won't repeat, following action have been taken:<br />1. Immediately we blocked an ability to remove services during deployments.<br />2. We prioritised a task to make our development environments configured in the same manner as our production one, as this problem could have been spotted earlier if that was the case.</p>tag:tipserstatus.com,2005:Incident/99829482022-05-02T10:00:00+02:002022-08-16T15:41:11+02:00Tipser Portal unavailable<p><small>May <var data-var='date'> 2</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>Resolved</strong> - Between 07:53 UTC and 08:10 UTC Tipser Portal was unavailable. One of our service providers was not operating correctly during that period. More details can be found here https://www.netlifystatus.com/incidents/m1d4vwctxsrs.</p>tag:tipserstatus.com,2005:Incident/99829442021-10-02T12:00:00+02:002022-08-16T15:41:11+02:00Issues with several Klarna payment methods in the EU<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>12:00</var> CEST</small><br><strong>Resolved</strong> - Due to the Klarna's internal technical error, customers had been experiencing difficulties in making purchases with Klarna (Klarna Checkout) mainly on Nordics markets. The incident was officially opened on Klarna's end at 9:39 CET and we (Tipser) started to observe an elevated error rates and impacted sales figures at around 12:00 CEST.<br /><br />This has been fixed by Klarna's side at 15:29 CEST. More details can be found here: https://status.klarna.com/incidents/4zfs12z3dftg.<br /><br />Shopping carts and created sessions were not be affected by this issue.</p>tag:tipserstatus.com,2005:Incident/99829372021-09-01T17:00:00+02:002022-08-16T15:41:11+02:00Service Error Rate Increase<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>17:00</var> CEST</small><br><strong>Resolved</strong> - We have noticed increased number of 503 returned by our service</p>tag:tipserstatus.com,2005:Incident/99829342021-08-18T23:30:00+02:002022-08-16T15:41:11+02:00Search service<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>23:30</var> CEST</small><br><strong>Resolved</strong> - One of node in Elasticsearch cluster was marked as down by monitoring system.</p>tag:tipserstatus.com,2005:Incident/99829262021-08-10T11:00:00+02:002022-08-16T15:41:11+02:00Tipser API intermittently unavailable<p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>11:00</var> CEST</small><br><strong>Resolved</strong> - From 7:35 to 7:40(UTC) and from 13:15 to 13:20(UTC) we experienced too long response rates and intermittent unavailability of Tipser API endpoints.<br />To mitigate this issue, resources on the infrastructure side were increased.<br /><br />After investigations, we're suspecting a performance issue in the Collection API. We're preparing a reproduction environment to identify the root cause.<br />All environments are now scaled out to mitigate any potential performance degradation and all systems are operational.</p>tag:tipserstatus.com,2005:Incident/99829032021-06-23T20:00:00+02:002022-08-16T15:41:11+02:00Tipser Portal is down<p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>20:00</var> CEST</small><br><strong>Resolved</strong> - Tipser Portal API is currently experiencing a major outage, possibly related to DNS changes we've performed last hour. No other components are recognized to be affected. We're identifying the rootcause.</p>tag:tipserstatus.com,2005:Incident/99828902021-06-23T13:00:00+02:002022-08-16T15:41:11+02:00Degraded performance of Transactions API<p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>13:00</var> CEST</small><br><strong>Resolved</strong> - Between 13:03 and 13:07 we experienced a degraded performance of Transactions API.<br />The issue is resolved and we're in a process of preparing the postmortem.</p>tag:tipserstatus.com,2005:Incident/99828782021-06-08T14:00:00+02:002022-08-16T15:41:11+02:00One of our service providers is experiencing issues<p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>14:00</var> CEST</small><br><strong>Resolved</strong> - Issue with upstream CDN provider: https://status.fastly.com/incidents/vpk0ssybt3bj.</p>tag:tipserstatus.com,2005:Incident/99828572021-06-01T11:30:00+02:002022-08-16T15:41:11+02:00Issue with shopping carts created before 2nd half 2019<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>11:30</var> CEST</small><br><strong>Resolved</strong> - Small number of shopping carts is experiencing Internal Server Errors. Error seems to be narrowed down to old carts (created before 2nd half 2019).<br />Affected endpoints v4/shoppingCart GET</p>