Jason Kitcat

This post is long overdue, I apologise, by-election campaigns and such like got in the way.

On Sunday June 7th the count for the South East region of the European Parliamentary elections was conducted at St Mary’s stadium, Southampton. In attendance were lots of media as well as candidates, agents and activists along with significant others.

I went along with my wife as one of the Green Party’s candidates and proceeded to experience a very long night with very little information and lots of frustration. What had to happen was for each local authority in the region to count its ballots and submit the results to the Southampton HQ. A few areas were delayed by recounts, mismatched ballot accounts (i.e. ballots lost or in the wrong pile) but there were clearly technical issues in Southampton also delaying matters.

I had learnt a few weeks earlier that the results from local authorities were to be transmitted to Southampton via a ‘secure website’. In essence, as I understand it because I never saw the system or any detailed specifications, returning officers would type the results (twice to verify) into an SSL form which was then emailed to Southampton and also stored in a database. I copy below the full response I received about my enquiries from the Regional Returning Officer Mark Heath.

I had concerns about this setup, what checks were being done and so on. So I ensured local Green agents texted us their results so we could check them against what the system claimed. I felt the returning officers should be collecting out-of-channel verification too via fax or telephone, but they weren’t interested in that idea — too quick and happy to trust the technology sadly.

On the night I saw the technology staffers and returning officer team looking tensely at a couple of computers. No surprise when all the informational displays were showing server errors, exceptions and so on. This left many unhappy candidates and agents who were quick to query the sense of these systems with the returning officer. How I wish they would remember these feelings in the weeks after… every election I observe with technology their are howlings about the problems on the night but a week later most are too busy celebrating their wins or analysising their losses to make the case about how the election was run.

Let’s run through the problems we had with the informational screens:

• They crashed regularly, especially earlier in the night;
• The colour coding was confusing as reds, greens, yellows were used in a non-political sense to inform what status various local counts had;
• They were often difficult to read with too small text or windows not at full size;
• The updates scrolled by so fast it was impossible to do much than see the top party on the first pass.

You can see the full range of problems screens on Flickr.

It’s worth noting that while they would have been detected in the end, someone could have caused chaos and mayhem by manipulating this results system either just the display (which was basically a webpage on a projector) or the tabulation/counting of results themselves. Given those possibilities I was concerned that the Electoral Commission had not had a role checking this software and that fairly serious failures were happening on the night.

I’m a technologist. I spend all day with computers, programming them, using them, talking about them. I remain deeply concerned by the use of technology in elections especially when it is done without the proper rigour of testing and certification. Things can and do go wrong, especially for high pressure events like elections.

I don’t think we would have been any worse off if in Southampton a fax had been received from every count with the results which was manually checked against the online results. These could have been tabulated in a public way the way ballots are checked. We have to be more cautious before jumping both feet first into a computer-only solution.

Responses from Regional Returning Officer to my queries prior to election day:

The system is secure, and has been fully tested already which has shown it to work fully  -and indeed without the potential errors that a system that requires data to be managed via Phone / FAX & re-inputted on several occasions – but I will let you know chapter & verse shortly. Thanks.

UPDATE: Adrian Windisch, Chair of Reading Green Party, writes to say Thanet Borough Council’s website reported 6,001 Green votes, but the South East region count recorded 3,001 votes. This was later corrected on the Thanet website following Adrian’s enquiries. Which goes to show these things do need checking!

….
On your question, the suppliers have advised us that:

“The European Regional Returning Officers Managements System (ERROMS) application along with the application databases reside on high powered servers within defined security level segments.

All hardware devices within ERS’ live hosting environment are duplicated to facilitate a highly redundant and resilient network. Market leading security appliances at the perimeter provide rich stateful inspection of traffic flows protecting the web servers from malicious activity. A further layer of security has been added to the servers using Anti-Reconnaissance software. The web servers are load balanced to enhance performance, should one of the servers fail the other will automatically service the entire load until the offending device is returned back to its functioning state. The database servers are hosted within an isolated network forcing database requests to be inspected by the firewall a second time. All databases using live replication software are replicated to a secondary offsite server which provides redundancy and disaster recovery.

Databases are further protected with database level passwords and access-granting security features. Intrusion Detection and Prevention Systems detect suspected efforts at server intrusion. A 24×7 automated monitoring system using specially designed intrusion detection parameters detects and blocks attempts at security breaches. The system logs all intrusion attempts, and these logs can be preserved to aid in prosecution of attackers, should such action be warranted.

All servers have been hardened to remove any non-essential code and are subject to strict operating system security such as permissions and password access. The hosting network and Web Applications are scanned weekly to ensure our web sites, servers, and internet-connected devices are free of known vulnerabilities. It also determines whether our site passes the SANS Top 20 Internet Security Vulnerabilities list as defined by SANS, the FBI and FedCIRC.”

The key elements to reduce error include;
• Initial entry of  results are submitted twice to reduce keying errors and are only accepted when both sets of results match.
• Additionally, submitted data is emailed to provide an electronic paper trail that can be used for confirmation of data submitted by both the RRO and LRO’s
• Declaration of Local Results is generated from the system with results authorised by the RRO and can be checked by the LRO’s against local records to ensure that the submitted values are correct.

Effectively this means the submitted results by the LRO are checked 3 times before local declaration and will help eliminate the transposing of figurers received via phone/fax which has been experienced before.
There are now 6 regions using this. We wouldn’t be doing it unless we were satisfied that it was secure. The risk of transposing figures data is one of the reasons for moving away from the phone/  fax route, although that remains available as a contingency / fall back option.