PROD Incident 1/20

[jeanepaul]

Greetings,

sorry to bother late night, just reporting of problem in prod.

currently, some partner app is failing reaching core app. but maybe later some would, unless take action…

what is currently known, and problem:

1. thursday maintenance (scheduled trigger), apicore, linker-api, notificator, importer were deployed yesterday 1/20 around 21:00
2. those apps are looking for gateway, but gateway is not deployed.
3. gw is supposed to be next week, after a week of gw test run in staging. (but can be deployed anytime, just need gw, and new settings)
4. result, linker-api can’t reach gw because of supposed to be settings… error in UseURLWhitelistPipeline Post "http://localhost:7001/apicore/validate_url_whitelist": dial tcp 127.0.0.1:7001: connect: connection refused
5. affected apps/services:
   • partner apps
   • hexabase, part only: csv import, and notifications.

sorry for the inconvenience. next time, I will take more caution for deployments, and maintenance!.

I will report back asap for recovery.

Thank you for your kind consideration.

[jeanepaul]

10:20am status report

recovery work for the trouble on missing the gateway deployments by @jeanepaul to production is being performed below:

@y.lin restored linker-api to old version. client/partner apps should be running.

@marwen.benhriz started deploying gateway service in production.

[jeanepaul]

15:00 the problem seems to be fix, no errors were reported, and everything is normal.

I take full responsibility for mis-deployment, and I regret.

please report for any problems!, thank you!