VirtualGL | DeveloperInfo / Open Source Code Signing: A Tragedy in Four Acts

Background

Pursuant to my previous message, which was sent only to TurboVNC users, the Thawte individual code signing certificate that I had been using since 2015 expired earlier this year, and Thawte refused to renew it, because they don't offer that product anymore. The three open source projects that I develop and maintain for a living (libjpeg-turbo, VirtualGL, TurboVNC) used that code signing certificate for the following purposes:

Signing the TurboVNC Viewer JARs for use with Java Web Start
Java 7u51 stopped allowing self-signed JARs. You could no longer simply dismiss a warning dialog to get around it, and that’s what initially prompted me to purchase a code signing certificate in 2015. Flash forward to 2019, and Java Web Start is no longer available in any supported form except in Java 8 SE, for which public updates ended for commercial users in January 2019 and for which public updates will end for all users in 2020. Thus, while I have no doubt that there are still TurboVNC deployments that use JWS, it is now a legacy technology, and support for it is being maintained in TurboVNC on a break/fix basis only.
In short, while this was the primary reason for obtaining a code signing certificate in 2015, it is no longer the primary reason.
Signing the Windows installers for all three projects
The original impetus for this was to ensure the integrity of our Windows installers, which provided peace of mind for users in the wake of the SourceForge DevShare scandal. However:
- SourceForge isn't even owned by the same company anymore, and the first thing the new owners did was eliminate the DevShare program.
- The DevShare program never affected active projects such as ours.
- There are other ways to ensure the integrity of Windows installers (such as a GPG sidecar) that don't involve a code signing certificate.
In 2019, the primary reason for signing Windows installers using a code signing certificate is that Windows 10 complains if you don’t. At the moment, this is mainly a PR problem, not a technical problem, since you can click through that warning dialog. However, given that macOS Sierra and later suffers from a more severe version of this problem, whereby you cannot install an unsigned package without right-clicking on it and selecting "Open", it wouldn't surprise me if Windows will eventually follow suit.

A Note About macOS

Both Windows and macOS seem to be moving in the direction of disallowing unsigned applications and installers, but macOS specifically will not accept any code signing certificates except those obtained through the Apple Developer Program, which costs $99/year. While those certificates are easy enough to obtain, one major limitation seems to be that installer packages (.pkg) and DMG files cannot be signed with a timestamp authority, so those packages/files will be treated as unsigned whenever the Apple Developer ID Installer certificate used to sign them expires. Apple's documentation regarding this is not exactly clear, but reading between the lines, it seems that these certificates will expire if your Apple Developer subscription expires. Otherwise, they appear to have a 5-year lifespan.

Apple is also moving in the direction of requiring applications to be “notarized”, which involves submitting them to Apple for approval, even if they won’t ever be distributed in the App Store. And, BTW, the one application that I would really like to distribute in the App Store— the TurboVNC Viewer— would never be accepted because it is GPL-licensed.

How Not to Obtain a Code Signing Certificate

I spent significant unpaid labor (approximately a week’s worth) trying to sort through this mess and figure out what to do going forward. Things I investigated:

Apple Developer Program

I went as far as subscribing to this, because a fellow open source developer was successfully using certificates he obtained through this program to sign JARs and Windows installers. However, after two full days of unpaid labor trying to make that work, I had to declare failure. Both jarsigner and signtool kept giving me an "unhandled critical extension" error with the certificates, which is probably by design. I submitted a tech support request to Apple, but they didn't even attempt to answer my question. They just copied & pasted irrelevant canned responses into their reply and directed me to the online developers' community for help.

So far, Apple has allowed user-level Gatekeeper workarounds, such as right-clicking on an unsigned installer package (.pkg) to install it and using the Security & Privacy applet to immediately whitelist an unsigned app that fails to launch. However, given their decision to eliminate the “Allow from Anywhere” Gatekeeper option in Sierra unless you re-enable it using a Terminal command, I am not confident that we (or any open source project) will be able to release unsigned Mac apps and packages past Mojave.

Even if I had been able to successfully use an Apple Developer certificate outside of macOS, it would have still been necessary for end users to install the Apple root certificate on Windows or in Java prior to validating our Windows installers or using our JARs with JWS. Since this approach didn’t serve my immediate needs, I requested a refund from Apple, but they refused. Given the limitations of Developer ID Installer certificates described above, it barely seemed worth the trouble to integrate Mac signatures into the build systems of libjpeg-turbo, VirtualGL, and TurboVNC. I mainly did so in order to get something tangible out of the $99 I spent, but again, to the best of my understanding, it will be necessary to maintain my subscription in order for packages signed with my certificate to be accepted by Gatekeeper.

Certum

Certum used to offer free individual code signing certificates to open source developers, but not anymore. After paying VAT and currency exchange fees and purchasing a cryptographic card from them, even their discounted open source code signing certificate would cost me approximately $150 for the first year. Also, with all due respect, it makes me a bit nervous to send documents that could be used to steal my identity to a company based in another country (particularly one that is undergoing political upheaval.)

CAcert.org

This is a reputation-based certificate authority, but in order to get a code signing certificate from them, you have to undergo a training program, which I simply don't have time to do. Also, their root certificate is not trusted by any of the major O/S's we support.

Comodo Individual Code Signing Certificate

To the best of my knowledge, this is the only CA in the U.S. that still offers an individual code signing certificate, but it’s a pain in the butt to obtain one. As an organization, the CA can simply look you up in an online directory, but as an individual, you have to find a notary public and show them the following three items:

a government ID (passport, driver’s license, etc.)
financial identification (a credit or debit card, or an original mortgage or bank statement less than 6 months old)
non-financial identification (a utility bill for the same address you listed in your certificate, a certified birth certificate, a certified copy of a court order, a lease payment statement, or a local authority tax bill less than a year old)

You have to fill out the validation form in the presence of the notary, and they have to notarize it. You then have to file a support request with Comodo and attach a scanned copy of the notarized form and all identification documents to the request. It took a while to get confirmation of this procedure. When I purchased the certificate, the instructions the certificate reseller gave me were for organization validation, not individual validation. I had to get into a chat session with the reseller, and they had to confirm the correct procedure with Comodo and send me the correct link. The instructions for individual validation incorrectly asserted that Comodo couldn’t validate me without having my phone number listed in a regional telephone directory (which isn’t exactly a thing in 2019.) That required further back & forth with them and the certificate reseller. I then had to confirm that a PDF/print-out of a utility bill was acceptable, since their instructions specifically said that they needed an “original” utility bill. Further back & forth. The notary form was also ambiguous in that one section of it claimed that an original copy of a land-line telephone bill was optional and another section claimed that it was necessary. This is why it was so damaging for Thawte to discontinue their individual code signing product. I had to go through a similar (albeit more straightforward) procedure in 2015 to get a certificate from them, but you don’t have to go through that procedure again in order to renew.

To add insult to injury, Comodo’s support ticketing system is somewhat poorly designed. It sent me an e-mail confirmation before I had even pressed the “Submit” button on the ticket, and unless I used Firefox, it didn’t provide any visual feedback to let me know that my documents had been successfully attached to the ticket. Oh, and also, you have to use Firefox when generating the CSR and collecting the certificate from them. Chrome isn’t supported.

When the certificate was finally issued, two full business days after the validation documents were submitted, it had my home address embedded in it, and Comodo refused to re-issue it without this embedded personal information. :( As you can imagine, given that I am an independent open source developer with a user base (indirectly, via libjpeg-turbo) numbering in the millions, embedding my home address in binary distributions of my software would be a gross invasion of my privacy. I had no choice but to abandon this effort, throw away 3-4 days of work, and request a refund from the certificate reseller. Apparently the prospect of losing money greased the wheels, and the certificate reseller was able to goad Comodo into re-issuing the certificate without my home address embedded in it. Thus, I was finally able to get a usable code signing certificate for my projects, but the process was way too difficult.

Self-Signed Certificate Authority

Had I been unable to obtain a legitimate code signing certificate (which was almost the case), I was prepared to go down this road. It would have required users to install a self-signed root certificate from The libjpeg-turbo/VirtualGL Project in order to use our JARs with JWS or to validate our Windows installers. It would have also represented a feature regression for the upcoming TurboVNC 2.2.2 release, but this was very close to being the only option available to me.

Conclusions

Hopefully you can see how, from an open source point of view, code signing currently does more harm than good. I wouldn’t have even pursued it were it not for the fact that Apple and Microsoft (and Oracle before them) forced our hand. I firmly believe in letting users choose for themselves. If a user wants to install an unsigned package or application, or run an unsigned JAR file, then the O/S (or Java) should pop up a warning dialog but let them choose to proceed. Instead, independent software developers are forced to use an antiquated and draconian process (Windows, Java) or a vendor-locked and poorly-supported process with some major technical limitations (Apple) in order to identify themselves. From my point of view, this is more about reducing corporate liability than protecting users from malware. Code signing is antithetical to the idea of open source, since the philosophy behind open source is reproducibility— the ability to exactly reproduce the behavior of a binary distribution from source code given the same build procedure. Obviously that is impossible if the binary distribution has to be signed by a specific developer in order to install or run it without the O/S complaining. I'm doing about five jobs here on a shoestring budget, and dealing with this is yet another thing that I have to do but can’t get paid for. Therefore, this will be a lean month for me due to circumstances entirely out of my control. Therefore, VirtualGL 2.6.2 and TurboVNC 2.2.2 have been delayed by weeks.

Referring to my other recent tome about the problems inherent with monetizing open source development in general (which was buried in a Slashdot comment), this certificate issue is just another example of how software development is increasingly skewed in favor of corporations, and how it is increasingly difficult for independents to make enough money to survive at it (that is not just true of software development.) What do I expect? Well, if O/S’s are going to require code signing, then they should take steps to grant free certificates to open source developers who meet certain criteria. I mean, my software has been publicly available for 15 years, in the case of VirtualGL and TurboVNC, and for 10 years in the case of libjpeg-turbo. The former were Sun products for four years. The latter is an ISO reference implementation. I’m not sure how much more “validated” you can get in the online era in which we live. Microsoft, via GitHub, could do a lot to help this situation by starting a “trusted open source developer” program via that site. Apple should be giving free, perpetual Apple Developer subscriptions to major open source projects, particularly projects that they themselves leverage for their proprietary software. $200/year to maintain certificates for both Windows and macOS may not seem like much, but when you’re living month to month, every little bit counts.

Sorry for the tome, but this is an issue that affects the open source community at large, and I felt compelled to share my experiences. If macOS and Windows continue allowing some form of workaround for unsigned apps/packages, then it’s not a critical issue from a technical point of view, but given that I’m trying to compete with large companies (and, frankly, doing a fair job for a one-person operation, if I do say so myself), it’s not good PR when my users have to jump through hoops in order to install my software.

In short, there has to be a better way.

DRC