As The Independent reports, Marco Marsala detailed his predicament on a forum called Server Fault. The forum aims to help server administrators resolve problems and software issues but it quickly became apparent Marsala is largely beyond help.
Marsala inadvertently ran the Linux command “rm -rf” while running maintenance operations on his servers. The command “rm” means delete and the “-rf” option tells Linux to force remove everything, even if it is open or being used, and hide the usual messages warning you this probably isn’t what you wanted to do.
Usually, the command would be run with a specified directory. However, because Marsala ran it by accident, no directory was given. It was triggered from a Bash script — a kind of Linux file that contains executable code — that would usually set the location programmatically.
A bug in the file meant this didn’t occur though. The directory to delete from was never added to the “rm” command. It ran from the top of the filesystem, deleting everything on all the company servers.
To top it all off, Marsala even deleted his backups. He did have offsite backups but he had just connected to the provider and mounted the drives to his computer for access. This gave the command access to the offsite backup server too, letting it wipe its contents alongside everything else.
Marsala headed to Server Fault, hoping to find advice on how to recover his data. However, he was quickly met by a consensus amongst users that the data is gone forever. One user, named Swen, said “I feel sorry to say that your company is now essentially dead.”
“You’re going out of business,” said Michael Hampton, another user. “You don’t need technical advice, you need to call your lawyer.”
The company previously hosted the websites of some 1,500 customers, providing servers for the website owners. The command also deleted the websites the company was responsible for. Customers will be reliant on their own backups to restore the damage and get their site registered with another provider.
Marsala quickly came under fire online for evidently failing to put proper security protections in place. It shouldn’t be possible to delete everything on several servers from one erroneous command. Marsala should have ensured that “rm -rf” could never run without an explicitly defined directory to delete from and, if it ever did, safeguards should have been in place to stop it in its tracks and ensure some form of backup remained available.
“This is not bad luck: it’s astonishingly bad design reinforced by complete carelessness,” wrote forum user Massimo. Rather than gain the help he had expected, Marsala has been directed to seek professional help from data recovery experts as soon as possible.
As long as the affected hard drives haven’t had time to overwrite the old contents, some of the data may still be recoverable using specialist techniques. It isn’t clear whether Marsala has contacted his customers yet or how he intends to recover from the disaster.
