Keboola - File uploads temporarily disabled – Incident details

Experiencing degraded performance

File uploads temporarily disabled

Identified
Degraded performance
Started about 11 hours ago

Affected

AWS EU (eu-central-1)

Degraded performance from 11:57 AM to 12:00 AM

GCP EU (europe-west3)

Degraded performance from 11:57 AM to 12:00 AM

AWS US (us-east-1)

Degraded performance from 11:57 AM to 12:00 AM

GCP US (us-east4)

Degraded performance from 11:57 AM to 12:00 AM

Azure NE (north-europe)

Degraded performance from 11:57 AM to 12:00 AM

Updates
  • Identified
    UTC
    Identified

    The security fix for the underlying issue has been released, and we are rolling it out across our services.

    File uploads through the Storage API Importer (import.*) are working again on all multi-tenant stacks, with the fix in place. This restores:

    • Uploading files in the Keboola UI

    • Attaching files in Kai chat

    • The /upload-file and /write-table endpoints of the Storage API Importer



    Still temporarily disabled:

    • Synchronous table create/import requests to the Storage API that use multipart file upload. These still return HTTP 403 until the fix reaches the Storage API.



    Asynchronous imports that use files/prepare work as before and can be used as a workaround.

    We will provide an update when new information is available.

  • Investigating
    UTC
    Investigating

    As a precautionary security measure, we have temporarily disabled multipart file uploads on all of our multi-tenant stacks

    Users may experience:

    • Failures when uploading files in the Keboola UI

    • Failures when attaching files in Kai chat

    • HTTP 403 errors from the /upload-file and /write-table endpoints of the Storage API Importer (import.*)

    • HTTP 403 errors on synchronous table create/import requests to the Storage API that use multipart file upload

    Asynchronous imports that use files/prepare still work and can be used as a workaround.

    We apologize for the disruption and will provide an update within 2 hours or when new information is available.