Keboola - [GCP Europe-west3] Jobs stuck in terminating phase – Incident details

All systems operational

[GCP Europe-west3] Jobs stuck in terminating phase

Resolved
Operational
Started 7 months agoLasted about 2 hours

Affected

GCP EU (europe-west3)

Degraded performance from 6:18 PM to 8:15 PM, Operational from 8:15 PM to 8:20 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result. We identified a couple of jobs processing (or terminating) longer also on other stacks. Altogether less than 20 jobs were affected. We apologize for the inconvenience.

  • Identified
    UTC
    Identified

    The root cause has been identified. New jobs will now terminate correctly, but some jobs remain stuck in terminating state. We are continuing to work on a fix.

  • Investigating
    UTC
    Investigating

    We are aware of performance degradation affecting our GCP Europe-west3 infrastructure.

    Symptoms:

    • Jobs being stuck in termination phase

    Our team is working on resolving this issue.

    We apologize for the disruption and will provide an update within 1 hour or when new information is available.