EnderDevelopment - Task and project stucked – Incident details

Task and project stucked

Resolved
Major outage
Started about 1 month agoLasted about 6 hours

Affected

EnderDevelopment - AI Platform

Degraded performance from 10:13 AM to 4:14 PM

Dashboard

Degraded performance from 10:13 AM to 4:14 PM

EndAI - Main Backend

Degraded performance from 10:13 AM to 4:14 PM

Updates
  • Resolved
    Resolved

    The platform has now fully recovered and is operating in a stable state.

    Following the mitigation and coordination with our main provider, we’ve achieved significant performance improvements across the system. Simple project generation times have been reduced from 1–2 minutes to ~15 seconds, with overall stability greatly enhanced.

    The infrastructure is now capable of handling up to 800+ tasks per day with improved reliability and throughput.

    The system is currently stronger and more efficient than before, with no active blocking issues detected.

    All services are operational. Let's go !

  • Monitoring
    Monitoring

    We have contacted our main infrastructure provider, who has assisted in identifying and resolving the underlying issue affecting our systems. Together with the provider, we have also improved the handling of the affected components to prevent further instability.

    The situation has significantly improved: services are no longer fully blocked. However, the system is still operating in a degraded state, with occasional latency and reduced reliability in task processing.

    Engineering teams are actively monitoring recovery and continuing stabilization efforts.

    Further updates will follow as performance returns to normal.

  • Investigating
    Investigating

    We are continuing to observe system instability affecting all project-related tasks.

    The issue initially identified in the circuit breaker layer now appears to be propagating across additional services, leading to increased latency, degraded performance, and continued task blocking.

    Engineering teams are actively containing the spread and working on mitigation steps to restore stability.

    At this time, tasks remain slow or fully blocked in several areas of the system. No ETA is available yet.

    Further updates will be provided as the situation evolves.

  • Identified
    Identified

    We are currently experiencing a system-wide block affecting all project-related tasks.

    During the investigation, we have identified issues within our circuit breaker mechanisms, which are currently triggering and preventing normal task execution across services.

    Engineering teams are actively working on stabilizing the affected components and restoring normal flow.

    At this stage, services remain degraded and task processing is still blocked. No ETA is available yet.

    Further updates will follow as soon as mitigation progresses.

  • Investigating
    Investigating

    All project-related tasks are currently blocked due to an ongoing technical issue currently under investigation.

    The team is actively working to identify the root cause and restore normal operations as quickly as possible.

    At this time, no estimated resolution time is available.
    Further updates will be provided as soon as more information becomes available.