Root Cause Analysis

A faulty background job configuration change was inadvertently made in our production environment. This erroneous change disabled the part of the system that routes calls to the correct volunteer or agent. For a blind person calling, this meant that the call kept ringing without being answered.

The issue was traced back to a recent deployment. After noticing the problem, we promptly reverted the incorrect change. We will improve our monitoring system to detect call-related problems much faster in the near future. Additionally, testing of configuration changes beforehand will be enhanced.

We sincerely apologize for the inconvenience this has caused and appreciate your understanding as we work to prevent similar issues in the future.

Timeline

12:40 UTC - A new version of configuration was actived

12:41 UTC - Configuration became effective and shut down the part of the system that routes calls. Incident started.

14:00 UTC - Reports of now-working calls received

14:50 UTC - Rollback completed. Incident ended

Impact

Be My AI (chat): No impact.

Calls to volunteers and service directory profiles: Fully unavailable during the incident.

Other Services: No impact.

Resolution

All services were restored by 14:50 UTC, with system stability confirmed through checks. No user data was in danger due to the incident.