This post provides an update regarding the ongoing infrastructure incident affecting the Docker host and several continuous runtime services used by community automation platforms.

While the initial expectation was that the issue could be resolved within a few hours, the investigation has revealed a deeper hardware and filesystem problem that has extended the recovery timeline.

Disclosure: I participate in the IONOS Premium Agency Partner program. Some infrastructure referenced in this article may involve services obtained through that relationship. This disclosure is provided for transparency only and does not imply endorsement or responsibility for the events described.

The incident began with a hard drive failure on the physical server responsible for hosting our Docker runtime environment.

After identifying the hardware issue, recovery work began immediately, including hardware diagnostics and standard restoration procedures.

Because the infrastructure is intentionally designed with service separation between serverless and continuous runtime systems, the outage has been limited in scope.

The following services remain fully operational:

• serverless API services
• database infrastructure
• event registration systems
• automation APIs
• other serverless platform components

These systems operate independently from the Docker runtime host and therefore continue functioning normally.

The following services depend on continuous runtime infrastructure and are currently unavailable:

• Docker container services
• Discord automation bots
• webmail services
• auxiliary DNS services
• domain forwarding systems
• WordPress hosting instances

These services rely on the affected storage subsystem and cannot start until the recovery process completes.

Once the hardware failure was confirmed, a standard recovery procedure was started.

This process included:

1. attempting to repair the failed drive
2. attempting to rebuild the storage configuration
3. diagnosing potential RAID inconsistencies

However, these recovery attempts were unsuccessful.

The affected system was originally configured with a RAID 1 mirrored storage configuration to provide redundancy.

The recovery process included attempts to:

• replace the failed drive
• rebuild the RAID 1 array

Unfortunately, the rebuild process also failed. At this stage, the remaining drive entered an error state, indicating potential filesystem or partition-level corruption.

To allow deeper investigation, the server has now been booted into rescue mode.

This allows direct access to the storage device for diagnostic and recovery operations without relying on the damaged runtime environment.

During analysis, we determined that the partition table on the affected drive had become corrupted, preventing the operating system from properly mounting the filesystem.

Because direct repair of the corrupted partition structure may risk further data loss, the current strategy focuses on disk cloning for recovery.

The process involves:

1. cloning the damaged drive
2. preserving the original disk image
3. attempting filesystem reconstruction from the cloned image
4. recovering container and service data where possible

This approach ensures that recovery attempts can proceed safely without risking further corruption of the original storage device.

The architecture of the platform separates services into two categories:

• serverless services
• continuous runtime services

Because the API and database infrastructure are separate from the Docker host, critical data and platform functionality remain safe.

Recovery work is still in progress.

The current focus is on:

• cloning the corrupted drive
• analyzing the filesystem damage
• recovering container data
• restoring continuous runtime services

Further updates will be provided once recovery progress advances or services are restored.

While hardware failures can occur unexpectedly, this incident reinforces the importance of designing infrastructure with clear separation between compute services and persistent data systems.

Thanks to this architecture, critical data and serverless systems remain fully operational while recovery efforts continue.

Additional updates will be shared as soon as new information becomes available.