Microsoft has released a fix for a spooky Exchange Server bug that shut down on-premises mail delivery around the world, just as the clocks rang in the new year.
The mass interruption stemmed from a date control error in Exchange Server 2016 and 2019 that made it impossible for servers to accommodate the year 2022, prompting some to call it the Y2K22 error. The mail programs saved dates and times as signed integers that are a maximum of 2147483647 or 231 – 1. Microsoft uses the first two numbers in an update release to indicate the year it was released. As long as the year was 2021 or earlier, everything worked fine.
“What in the world is Microsoft?”
However, when Microsoft released version 2201010001 on New Year’s Eve, the local servers went down because they were unable to interpret the date. Consequently, messages got stuck in transport queues. Administrators around the world were desperately left unsuccessful in trying to figure out what they were talking about instead. The only thing they had to continue were two cryptic log messages that looked like this:
Log Name: Application Source: FIPFS Logged: 1/1/2022 1:03:42 AM Event ID: 5300 Level: Error Computer: server1.contoso.com Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
Log Name: Application Source: FIPFS Logged: 1/1/2022 11:47:16 AM Event ID: 1106 Level: Error Computer: server1.contoso.com Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.
“What in the world is Microsoft !?” an admin wrote in this Reddit thread, which was one of the first forums to report the mass error. “New Years eve!? The first place I check is Reddit, and you’re saving my life before we even get an engineer on the phone. “
The next day, Microsoft released a fix. It comes in two forms: an automated PowerShell script or a manual solution in case the script did not work properly, as reported by some administrators. In either case, the fixes must be performed on all local Exchange 2016 and Exchange 2019 servers in an affected organization. The automated script can run on multiple servers in parallel. The software maker said the automated script “may take some time to run” and urged administrators to be patient.
The date and time control was performed when Exchange checked the version of FIP-FS, a scanning engine that is part of Exchange’s antimalware protection. Once FIP-FS versions started with the numbers 22, the check could not be carried out and mail delivery was abruptly stopped. The fix stops the Microsoft Filtering Management and Microsoft Exchange Transport services, deletes current AV engine files, and installs and launches a patched AV engine.
On Monday, things were about to return to normal for many affected organizations. It is not clear how long the buggy data warehouse had been in place, but judging by the two affected versions, it was possibly introduced when Exchange Server 2016 was under development.