Keboola - File uploads temporarily disabled – Incident details

All systems operational

File uploads temporarily disabled

Resolved
Degraded performance
Started 2 days agoLasted 27 hours 21 minutes

Affected

AWS EU (eu-central-1)

Degraded performance from 11:57 AM to 7:07 AM, Operational from 7:07 AM to 3:19 PM

GCP EU (europe-west3)

Degraded performance from 11:57 AM to 7:07 AM, Operational from 7:07 AM to 3:19 PM

AWS US (us-east-1)

Degraded performance from 11:57 AM to 7:07 AM, Operational from 7:07 AM to 3:19 PM

GCP US (us-east4)

Degraded performance from 11:57 AM to 7:07 AM, Operational from 7:07 AM to 3:19 PM

Azure NE (north-europe)

Degraded performance from 11:57 AM to 7:07 AM, Operational from 7:07 AM to 3:19 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident is resolved.

    The security fix has been rolled out to all our services on all stacks, and file uploads work normally again, including uploads in the Keboola UI, file attachments in Kai chat, and table imports through the Storage API.

    No action is needed on your side.

    We apologize for the disruption. If you still experience any problems with file uploads, please contact our support.

  • Monitoring
    UTC
    Monitoring

    The security fix has been rolled out to all our services on all stacks, and file uploads are fully working again.

    This restores:

    * Synchronous table create/import requests to the Storage API that use multipart file upload, which no longer return HTTP 403


    Uploading files in the Keboola UI, attaching files in Kai chat, and the /upload-file and /write-table endpoints of the Storage API Importer were already restored in the previous update.

    No action is needed on your side. Asynchronous imports using files/prepare continued to work throughout.

    We apologize for the disruption. If you still see HTTP 403 errors on uploads, please contact our support.

  • Identified
    UTC
    Identified

    The security fix for the underlying issue has been released, and we are rolling it out across our services.

    File uploads through the Storage API Importer (import.*) are working again on all multi-tenant stacks, with the fix in place. This restores:

    • Uploading files in the Keboola UI

    • Attaching files in Kai chat

    • The /upload-file and /write-table endpoints of the Storage API Importer



    Still temporarily disabled:

    • Synchronous table create/import requests to the Storage API that use multipart file upload. These still return HTTP 403 until the fix reaches the Storage API.



    Asynchronous imports that use files/prepare work as before and can be used as a workaround.

    We will provide an update when new information is available.

  • Investigating
    UTC
    Investigating

    As a precautionary security measure, we have temporarily disabled multipart file uploads on all of our multi-tenant stacks

    Users may experience:

    • Failures when uploading files in the Keboola UI

    • Failures when attaching files in Kai chat

    • HTTP 403 errors from the /upload-file and /write-table endpoints of the Storage API Importer (import.*)

    • HTTP 403 errors on synchronous table create/import requests to the Storage API that use multipart file upload

    Asynchronous imports that use files/prepare still work and can be used as a workaround.

    We apologize for the disruption and will provide an update within 2 hours or when new information is available.