2026-04-23: Geofencing halted (April 22–23, 2026)

  • Tråden er låst

Geofencing halted (April 22–23, 2026)


Summary
Geofencing and activity generation stopped for all customers overnight. This caused delayed arrival and departure reporting across Port services.
All processing has been restored, and data has been reprocessed without loss.


Affected services
Port, Monitor and third-party webhook integrations


What happened
A peripheral dependency became unavailable, which disrupted the processing of incoming vessel position messages. Due to insufficient retry and recovery logic, this caused all processing nodes to stop handling real-time geofencing data.

As a result, location messages accumulated in the processing pipeline instead of being handled, leading to a growing backlog and halted activity generation across the system.


Impact
No activities were generated during the incident period, resulting in delays in arrival and departure reporting.


Timeline (UTC)
20:40 – Processing lag increasing
20:55 – Alert triggered (not handled as expected)
07:40 – Incident identified
07:55 – Mitigation started (nodes restarted and scaled for backlog processing)
16:00 – Processing backlog cleared and system normalized


Root cause
A service interruption in an external dependency, combined with insufficient error handling and recovery mechanisms, caused processing to halt across all nodes.


Follow-up actions
Improved retry and recovery mechanisms
Improved monitoring and alert response
Review of handling for external dependencies

0 Stemmer