Plain Talk Series on Understanding Voting System Updates Part 5: What’s Required to Update?

This is the 5th of a 6-part series of slightly longer vignettes on the challenge of updating voting systems. It’s a slice-and-dice of a recent briefing on the topic.

It’s intended to acquaint relatively newcomers to understanding how voting system are purchased and maintained, and that includes anyone and everyone from concerned citizens, to journalists, to policy makers.

The Contract is in Place. What’s Next?

Assuming that a vendor actually agreed to develop, test, and release updated software for a voting system, implementing the update is far more complicated than what mainstream technology users are accustomed to, due to the unique operating and regulatory requirements for election infrastructure.

Unlike an Internet-connected personal computer at one’s home, for example, voting equipment cannot simply be modified by an clicking a button, based on new software “pushed” by the manufacturer, over a network.

In other words, for voting systems, there is nothing equivalent to familiar messages like this, which might appear on your computer screen:

Updates are available. Would you like to install the new version 12.5 of your web browser? Select Yes or No.

Voting Systems Aren’t Easy To Update

The two main reasons that voting system update requirements are very different from mainstream technology are:

1. All updated voting system software must go through federal and/or state approval processes before being released and installed; and
2. Because voting components are typically (but not always) “air-gapped” (meaning they are not connected to the Internet or other networks), changing their software usually requires physical labor in a warehouse, such as inserting a USB device, or replacing individual memory cards in each voting device. That takes time and money, and is subject to human error.

Let’s look at each of these challenges more closely.

Voting System Approval and Certification: Time-Consuming and Costly

Before voting system releases can be implemented and used, approximately 40 states in the U.S. require that the voting system configuration first be federally certified by the U.S. Election Assistance Commission (EAC).

This federal institution works with accredited third-party test labs, known as Voting System Test Laboratories (VSTLs), to ensure that each voting system release complies with the federal Voluntary Voting System Guidelines (VVSG).

The federal certification process poses significant challenges for rapid updates, however, because the EAC certifies only complete voting systems, irrespective of how incremental software changes might be.

Furthermore, although back-office EMS computers and individual voting devices include a combination of operating system software and proprietary voting system software, they cannot be changed independently of each other, without resulting in a new voting system configuration or “version,” which must be re-tested and re-certified.

Certification Process Limitations

These limitations (above) mean that none of the following update procedures are possible under current federal certification, without undergoing a comprehensive re-certification process:

• An operating system on a back-office EMS computer or on a voting device cannot be changed or updated separately from the proprietary voting software; the OS and the voting system software are bundled together and are considered an integrated “package.”
• Security “patches” or other minor updates cannot be applied to either operating systems or proprietary voting system software.
• Neither operating systems nor individual voting software applications can be changed without changing the “version number” of the overall voting system (e.g., if there is a small change in even one component of a voting system with many other parts, the entire combination of components is considered a different “system version”).

Given the need for manufacturers to achieve certification of the “total voting system,” even a change in the operating system alone (e.g., moving from Windows 7 to Windows 10) requires significant development and integration testing, followed by a long certification cycle.

Limits on Certification Delay Updates — and Increase Costs

Those factors also help to explain why vendors attempt to “hedge” their costs, with contract language about updates that reserves the vendors’ right to impose additional charges for updates (i.e., charges above and beyond annual license and support fees).

Neither development nor certification is free, and in the end, those costs flow down to election officials (and ultimately we, the taxpayers), as either hidden costs or explicit fees.

Because all of these restrictions make it impossible to develop and deploy software changes in a “modular” fashion, the net effect is that any updated voting system will almost certainly be “behind the times” – or even worse, out-of-date — by the moment it reaches an election jurisdiction’s warehouse.

How Long Will These Updates Actually Take to Complete?

Even at a relatively fast pace, a federal certification for an updated system might require 3 to 6 months (and it could be much longer), and state certifications typically require at least 1 to 3 months.

So, best case scenario, that’s at least half a year (and perhaps close to a year) before an update could be ready for implementation, not including the time it would take the manufacturer to develop and test the updated software – and that’s if the manufacturer chooses to do so, in the first place.

Bottom line, the dynamic between voting system manufacturers and the federal EAC is broken: on the one hand, the EAC does not compel vendors to make timely technology updates (e.g., by prohibiting the use of outdated operating systems), nor does the federal certification program facilitate such updates. Given those two choices, and until something in the federal certification program changes, vendors are likely to continue taking the easy way out, by selling older certified technology, even if it has not caught up with the latest advancements.

Software Installation Process: Labor-Intensive, with Potentially Marginal Gains

Assuming a software update were to be developed, certified, and made available to local election officials, there’s still the question of actually getting it installed on all of the jurisdiction’s PCs and voting devices.

This requires physical activities such as:

• replacing hard drives on EMS computers;
• labor-intensive staging of dozens, hundreds, or potentially even thousands of voting devices, so that USB sticks or other technology tools can be used to update firmware; and/or
• an intensive post-installation process of testing and formally “accepting” the updated devices (usually accomplished by running comprehensive diagnostics, or a mock election).

An Uncertain and Inflexible Voting System Update Process

Furthermore, as noted in Part 4 (“When Might Voting System Updates Happen? {insert link}”), due to the vague nature of contractual language around updates, which reserves wide discretion for vendors to do only what they want (including the possibility of charging for updates), local elections officials are left with much uncertainty:

• They may not be able to plan far in advance for when updates might be coming;
• Their flexibility is tightly constrained by immovable election cycles (which last several months at a time), during which changes to equipment cannot be made;
• They may not know how much the update process is going to cost;
• They may not have the budget to pay vendor fees associated with installation; and
• They may not have adequate personnel resources (in terms of either numbers or technical expertise) to install updates themselves.

Finally, depending on the particular state in which the jurisdiction is located, there may be additional limitations on the nature of the work that can be done by the vendor, or by local jurisdictions.

Some states (such as Colorado, for example) have an intensive “trusted build” process that is intended to protect the chain of custody of all voting system components:

1. The state receives the updated software directly from the appropriate testing lab;
2. the vendor must directly train authorized state technical staff to perform the software installation process; and
3. all installations must be performed (in each and every local jurisdiction) only by authorized state personnel – not by the vendor, and not by local officials.

The Current Voting System Update Process Means Voting Systems Are Vulnerable

As one can imagine, all of these uncertainties and restrictions have a direct impact on how local election officials might perceive the value or utility of installing any particular voting system software update.

Indeed, in some cases, they might not bother. And even for those officials that do regularly update, because of the complexities of certification and installation, they will still be “last in line,” with update software that is prematurely dated.

The net result of these challenges is that much of the nation’s voting infrastructure is likely to contain security vulnerabilities that were stamped out months or years ago by IT security teams in more mainstream organizations.

So what, if anything, might improve things in the future? We’ll provide some recommendations in our final post in this series. To read more, here are all of the articles in this Voting Systems Update Series published to date.