D.C. Reality Check – The Opportunities and Challenges of Transparency

Gentle Readers:
This is a long article/posting.  Under any other circumstance it would be just too long.

There has been much written regarding the public evaluation and testing of the District of Columbia’s Overseas “Digital Vote-by-Mail” Service (the D.C.’s label).  And there has been an equal amount of comment and speculation about technology supplied to the District from the OSDV Foundation’s TrustTheVote Project, and our role in the development of the D.C. service.  Although we’ve been mostly silent over the past couple of weeks, now enough has been determined so that we can speak to all readers (media sources included) about the project from our side of the effort.

The coverage has been extensive, with over 4-dozen stories reaching over 370 outlets not including syndication.  We believe it’s important to offer a single, contiguous commentary, to provide the OSDV Foundation’s point of view, as a complement to those of the many media outlets that have been covering the project.

0. The Working Relationship: D.C. BoEE & TrustTheVote Project
Only geeks start lists with item “0” but in this case its meant to suggest something “condition-precedent” to understanding anything about our work to put into production certain components of our open source elections technology framework in D.C. elections.  Given the misunderstanding of the mechanics of this relationship, we want readers to understand 6 points about this collaboration with the District of Columbia’s Board of Elections & Ethics (BoEE), and the D.C. I.T. organization:

1. Role: We acted in the capacity of a technology provider – somewhat similar to a software vendor, but with the critical difference of being a non-profit R&D organization.  Just as has been the case with other, more conventional, technology providers to D.C, there was generally a transom between the OSDV Foundation’s TTV Project and the I.T. arm of the District of Columbia.
2. Influence: We had very little (if any) influence over anything construed as policy, process, or procedure.
3. Access: We had no access or participation in D.C.’s IT organization and specifically its data center operations (including any physical entry or server log-in for any reason), and this was for policy and procedural reasons.
4. Advice: We were free to make recommendations and suggestions, and provide instructions and guidelines for server configurations, application deployment, and the like.
5. Collaboration: We collaborated with the BoEE on the service design, and provided our input on issues, opportunities, challenges, and concerns, including a design review meeting of security experts at Google in Mountain View, CA early on.
6. Advocacy: We advocated for the public review, cautioning that the digital ballot return aspect should be restricted to qualified overseas “UOCAVA” voters, but at all times, the BoEE, and the D.C. I.T. organization “called the shots” on their program.

And to go on record with an obvious but important point: we did not have any access to the ballot server, marked ballots, handling of voter data, or any control over any services for the same.  And no live data was used for testing.

Finally, we provided D.C. with several software components of our TTV Elections Technology Framework, made available under our OSDV Public License, an open source license for royalty-free use of software by government organizations.  Typical to nearly any deployment we have done or will do, the preexisting software did not fit seamlessly with D.C. election I.T. systems practices, and we received a “development grant” to make code extensions and enhancements to these software components, in order for them to comprise a D.C.-specific system for blank ballot download and an experimental digital ballot return mechanism (see #7 below).

The technology we delivered had two critically different elements and values.  The 1st, “main body of technology” included the election data management, ballot design, and voter user interface for online distribution of blank ballots to overseas voters.  With this in hand, the BoEE has acquired a finished MOVE Act compliant blank ballot delivery system, plus significant components of a new innovative elections management system that they own outright, including the source code and right to modify and extend the system.

For this system, BoEE obtained the pre-existing technology without cost; and for D.C-specific extensions, they paid a fraction of what any elections organization can pay for a standard commercial election management system with a multi-year right-to-use license including annual license fees.

D.C.’s acquired system is also a contrast to more than 20 other States that are piloting digital ballot delivery systems with DoD funding, but only for a one-time trial use.  Unlike D.C., if those States want to continue using their systems, they will have to find funding to pay for on-going software licenses, hosting, data center support, and the like.  There is no doubt, a comparison shows that the D.C. project has saved the District a significant amount of money over what they might have had to spend for ongoing support of overseas and military voters.

That noted, the other (2nd) element of the system – digital return of ballots – was an experimental extension to the base system that was tested prior to possible use in this year’s November election.  The experiment failed in testing to achieve the level of integrity necessary to take it into the November election.  This experimental component has been eliminated from the system used this year.  The balance of this long article discusses why that is the case, and what we saw from our point of view, and what we learned from this otherwise successful exercise.

1. Network Penetration and Vulnerabilities
There were two types of intrusions as a result of an assessment orchestrated by a team at the University of Michigan led by Dr. Alex Halderman, probing the D.C. network that had been made available to public inspection.  The first was at the network operations level.  During the time that the Michigan team was testing the network and probing for vulnerabilities, they witnessed what appeared to be intrusion attempts originating from machines abroad from headline generating countries such as China and Iran.  We anticipate soon learning from the D.C. IT Operations leaders what network security events actually transpired, because detailed review is underway.  And more to that point, these possible network vulnerabilities, while important for the District IT operations to understand, were unrelated to the actual application software that was deployed for the public test that involved a mock election, mock ballots, and fictitious voter identities provided to testers.

2. Server Penetration and Vulnerabilities
The second type of intrusion was directly on the District’s (let’s call it) “ballot server,” through a vulnerability in the software deployed on that server. That software included: the Red Hat Linux server operating system; the Apache Web server with standard add-ons; the add-on for the Rails application framework; the Ruby-on-Rails application software for the ballot delivery and return system; and some 3rd party library software, both to supplement the application software, and the Apache software.

The TrustTheVote Project provided 6 technology assets (see below, Section 7) to the BoEE project, plus a list of requirements for “deployment;” that is, the process of combining the application software with the other elements listed above, in order to create a working 3-tier application running on 3 servers: a web proxy server, an application server, and a database server.  One of those assets was a Web application for delivering users with a correct attestation document and the correct blank ballot, based on their registration records.  That was the “download” portion of the BoEE service, similar to the FVAP solutions that other states are using this year on a try-it-once basis.

3. Application Vulnerability
Another one of those technology assets was an “upload” component, which performed fairly typical Web application functions for file upload, local file management, and file storage – mostly relying on a 3rd-party library for these functions.  The key D.C.-specific function was to encrypt each uploaded ballot file to preserve ballot secrecy.  This was done using the GPG file encryption program, with a command shell to execute GPG with a very particular set of inputs.  One of those inputs was the name of the uploaded file. 

And here was the sticking point.  Except for this file-encryption command, the library software largely performed the local file management functions.  This included the very important function of renaming the uploaded file to avoid giving users the ability to define file names on the server.  Problem: during deployment, a new version of this library software package was installed, in which the file name checks were not performed as expected by the application software.  Result: carefully crafted file names, inserted into the shell command, gave attackers the ability to execute pretty much any shell command, with the userID and privileges of the application itself.

Just as the application requires the ability to rename, move, encrypt, and save files, the injected commands could also use the same abilities.  And this is the painfully ironic point: the main application-specific data security function (file encryption), by incorrectly relying on a library, exposed those ballot files (and the rest of the application) to external tampering.

4.  Consequences
The Michigan team was creative in their demonstration of the results of attacking a vulnerability in what Halderman calls a “brittle design,” a fair critique common to nearly every Web application deployed using application frameworks and application servers.  In such a design, the application and all of its code operates as a particular userID on the server.  No matter how much a deployment constrains the abilities of that user and the code running as that user, the code, by definition, has to be able to use the data that the application manages.

Therefore, if there is a “chink” in any of the pieces the collective armor (e.g., the server, its operating system, web server, application platform, application software, or libraries) or the way they fit together, then that “chink” can turn use into an abuse.  That abuse applies to any and all of the data managed by the application, as well as the file storage used by the application.  As the Michigan teamed demonstrated, this general rule also applies specifically, when the application data includes ballot files.

5.  Mea Culpa
Let’s be clear, the goof we made, and “our bad” in the application development was not anticipating a different version of the 3^rd-party library, and not locking in the specific version that did perform file name checking that we assumed was done to prevent exactly this type of vulnerability.  And in fact, we learned 4 valuable lessons from this stumble:

1. Factoring Time:  Overly compressed schedules will almost certainly ensure a failure point is triggered.  This project suffered from a series of cycle-time issues in getting stuff requisitioned, provisioned, and configured, and other intervening issues for the BoEE, including their Primary election which further negatively impacted the time frame.  This led to a very compressed amount of time to stage and conduct this entire exercise;
2. Transparency vs. Scrutiny:  The desired public transparency put everyone involved in a highly concentrated light of public scrutiny, and margins of otherwise tolerable error allowed during a typical test phase were nonexistent in this setting – even the slightest oversight typically caught in a normal testing phase was considered fault intolerant, as if the Pilot were already in production;
3. (Web) Application Design:  Web applications for high-value, high-risk data require substantial work to avoid brittleness.  Thankfully, none of the TrustTheVote Elections Technology Framework will require an Internet-connected Web application or service – so the 3^rd lesson is how much of a relief that is going forward for us; and
4. No Immunity from Mistake: Even the most experienced professionals are not immune from mistake or misstep, especially when they are working under very skeptical public scrutiny and a highly compressed time schedule – our development team, despite a combined total of 7 decades of experience, included.

So, we learned some valuable lessons from this exercise. We still believe in the public transparency mandate, and fully accept responsibility for the goof in the application development and release engineering process.

Now, there is more to say about some wholly disconnected issues regarding other discovered network vulnerabilities, completely beyond our control (see #0 above), but we’ll save comment on that until after the D.C. Office of the CTO completes their review of the Michigan intrusion exercise.   Next, we turn attention to some outcomes.

6. Outcomes
Let’s pull back up to the 30-thousand foot level, and consider what the discussion has been about (leaving aside foreign hackers).  This test revealed a security weakness of a Web application framework; how there can be flaws in application-specific extensions to routine Web functions like file upload, including flaws that can put those functions and files at risk.  Combine that with the use of Web applications for uploading files that are ballots.  Then, the discussion turns on whether it is possible (or prudent) to try to field any Web application software, or even any other form of software, that transfers marked ballots over the Internet.  We expect that discussion to vigorously continue, including efforts that we’d be happy to see, towards a legislative ruling on the notion, such to Ohio’s decision to ban digital ballot transfer for overseas voting or North Carolina’s recent enthusiastic embrace of it.

However, public examination, testing, and the related discussions and media coverage, were key objectives of this project.  Rancorous as that dialogue may have become, we think it’s better than the dueling monologues that we witnessed at the NIST conference on overseas digital voting (reported here earlier).

But this is an important discussion, because it bears on an important question about the use of the Internet, which could range from (a) universal Internet voting as practiced in other countries (which nearly everyone in this discussion, including the OSDV Foundation, agrees is a terrible idea for the U.S.), to (b) the type of limited-scope usage of the Internet that may be needed only for overseas and military voters who really have time-to-vote challenges, or (c) limited only to ballot distribution.  For some, the distinction is irrelevant.  For others, it could be highly relevant.  For many, it is a perilous slippery slope.  It’s just barely possible that worked examples and discussion could actually lead to sorting out this issue.

The community certainly does have some worked examples this year, not just the D.C. effort, and not just DoD’s FVAP pilots, but also other i-Voting efforts in West Virginia and elsewhere.  And thankfully, we hear rumors that NIST will be fostering more discussion with a follow-up conference in early 2011 to discuss what may have been learned from these efforts in 2010.  (We look forward to that, although our focus returns to open source elections technology that has nothing to do with the Internet!)

7. Our Technology Contributions
Finally, for the record, below we catalog the technology we contributed to the District of Columbia’s Overseas “Digital Vote-by-Mail” service (again, their label).  If warranted, we can expand on this, another day.  The assets included:

1. Three components of the open source TrustTheVote (TTV) Project Elections Technology Framework: [A] the Election Manager, [B] the Ballot Design Studio, and [C] the Ballot Generator.
2. We augmented the TTV Election Manager and TTV Ballot Design Studio to implement D.C.-specific features for election definition, ballot design, and ballot marking.
3. We extended some earlier work we’ve done in voter record management to accommodate the subset of D.C. voter records to be used in the D.C. service, including the import of D.C.-specific limited-scope voter records into an application-specific database.
4. We added a Web application user experience layer on top of that, so that voters can identify themselves as matching a voter database record, and obtain their correct ballot (the application and logic leading up to the blank ballot “download” function referred to above) and to provide users with content about how to complete the ballot and return via postal or express mail services.
5. We added a database extension to import ballot files (created by the TTV Ballot Generator), using a D.C.-specific method to connect them to the voter records in order to provide the right D.C.-specific ballot to each user.
6. We added the upload capability to the web application, so that users could choose the option of uploading a completed ballot PDF; this capability also included the server-side logic to encrypt the files on arrival.

All of these items, including the existing open-source TTV technology components listed above in 7.1 above, together with the several other off-the-shelf open-source operating system and application software packages listed in Section 2 above, were all integrated by D.C’s IT group to comprise the “test system” that we’ve discussed in this article.

In closing, needless to say, (but we do so anyway for the record) while items 7.1—7.5 can certainly be used to provide a complete solution for MOVE Act compliant digital blank ballot distribution, item 7.6 is not being used for any purpose, in any real election, any time soon.

One final point worth re-emphasizing: real election jurisdiction value from an open source solution…..

The components listed in 7.1—7.5 above provide a sound on-going production-ready operating component to the District’s elections administration and management for a fraction of the cost of limited alternative commercial solutions.  They ensure MOVE Act compliance, and do not require any digital ballot return.  And the District owns 100% of the source code, which is fully transparent and open source.  For the Foundation in general, and the TrustTheVote Project in particular, this portion of the project is an incontrovertible success of our non-profit charter and we believe a first of its kind.

And that is our view of D.C.‘s project to develop their “Digital Vote-by-Mail” service, and test it along with the digital ballot return function.  Thanks for plowing through it with us.