codesigning

Codesigning is one of the worst issues we had been having since we started working on the new Opera for Mac. How Apple managed to screw this up never ceased to amaze us.

Since yesterday morning our build servers started to get CSSMERR_TP_NOT_TRUSTED error while code signing the Mac builds. Well, we didn’t notice until trying to release the new Opera Next build in the afternoon, which is obviously a bad timing. When it happened, immediate reaction was search for it in Google, unfortunately, when it happened words haven’t been spread yet so all results we got were from early 2009 ~ 2011, about some intermediate certificates missing, which completely mislead us. We spent a couple of hours inspecting certificates on all 3 of our Mac buildbot servers, none of them seemed wrong. One of my colleagues tried to resign a package locally with certificates/keys installed, got the same error as well.

Fortunately our build server didn’t get the same error every time so we managed to get a build for release.

When I later did a search for the same keyword but limit the results in last 24 hours, we finally found the real answer to the problem this time. According to this discussion:

Apple timestamp server(s) after all that is the problem here. If I add the --timestamp=none option, codesign always succeeds.

I have exactly the same problem. Probably Apple got two timeservers, with one broken, and a 50% chance for us to reach the working one.

And it worked for us perfectly as well. The only thing I didn’t know was whether it’s safe to release a build without requesting a timestamp (or where can we find other trusted timestamp servers).

This morning I woke up and saw this summary about yesterday’s incident.

According to Allan Odgaard (the author of TextMate):

As long as the key hasn’t expired, there should be no issue with shipping an app without a date stamp, and quite sure I have shipped a few builds without the signed date stamp.

That at least give us some confidence that if such incident happen again, it shouldn’t be a big issue to turn timestamp off.

Update: More explanations from Apple:

The point of cryptographic timestamps is to assist with situations where your key is compromised. You recover from key compromise by asking Apple to revoke your certificate, which will invalidate (as far as code signing and Gatekeeper are concerned) every signature ever made with it unless it has a cryptographic timestamp that proves it was made before you lost control of your key. Every signature that does not have such a timestamp will become invalid upon revocation.