Keboola - Notice history

We have seen no more throttled requests since 16:40. The incident is resolved.

We see substantial reduction of throttled requests. Only one request was throttled in the past 15 minutes.

Incident resolved. Mitigation has been applied and systems have returned to normal operation. Follow-up work is in progress to implement a permanent fix and prevent recurrence.

We have identified the root cause of the failing storage jobs on the BigQuery backend. The issue is isolated to internal/system projects and does not impact customer data or workloads. We are applying mitigation steps and continue to monitor the system closely.

The situation has stabilized and we are no longer observing elevated 429 responses from GCP Vertex AI on the affected EU stacks. Kai AI features are operating normally again. The incident is now resolved, and we will continue to monitor the situation.

We apologize for the inconvenience.

Storage job processing speeds have been restored. The bottleneck was identified as resource contention from an internal service. Systems are now stable, and all queued jobs have completed.

It seems that reauthorizing the MCP server resolves the issue. If you are experiencing missing or limited tools, please disconnect and reconnect the MCP server in your client. We are continuing to investigate the root cause to prevent recurrence.

Kai assistant (and other forms of platform chats) are not affected by this issue. Only external use of the MCP server (e. g. in Claude) are affected.

The issue affecting SMTP Sender component has been fully resolved. All functionality has been restored.

Thank you for your patience while we worked to resolve this issue and sorry for any inconvenience this may have caused.

We have identified multiple Storage jobs stuck in a processing state, blocking downstream jobs from proceeding. All of them have been terminated, and job processing should now be unblocked.

We apologize for any inconvenience this may have caused.

The issue affecting stuck flow jobs has been resolved, and normal operation has been restored.

We apologize for any inconvenience this may have caused.

We have identified the root cause as database inconsistency. Only a low number of flow jobs are stuck in processing state despite all their child jobs are finished (mostly failed). Our team is now working on implementing a solution.

We apologize for the disruption and will provide an update within 1 hour or when new information is available.

We implemented a fix, jobs should be now starting normally. We apologize for the inconvenience.

We are currently investigating this incident. Jobs are stuck in "Created" state. Next update in 30 minutes or sooner.

This incident has been resolved.

Few chats started approximately between 6:30 UTC and 8:30 UTC may still show the errors below, if you encounter these, please start a new chat.

We have identified the root cause and deployed a fix. Please start a new fresh conversation in case you see any of the error messages:

messages.4: user messages must have non-empty content
messages.11.content.1.text: cache_control cannot be set for empty text blocks

The issue has been fixed. We reverted the changes that caused the problem and full access has been restored across all regions.

All users can now access their projects normally, including those with Guest and Read-only roles who were previously affected.

Thank you for your patience.

We have identified that this issue affects users with Guest and Read-only roles. Users with other roles should be able to use the platform normally.

The issue is related to a release deployed at 13:50 UTC. We are currently working on a rollback.

Next update within 30 minutes.