#NigeriaDecides 3.0: The Gubers

#NigeriaDecides 3.0: The Gubers

Morning all. I'm going to do a bit of liveblogging of today's Gubernatorial elections from Project 2011 Swift Count's secret lair again.

6:30 PM

Most of the labor-intensive work is done with the project now. The texts are in or the observers have been relentlessly hounded on the phone until they gave over the information, and the database is as up to date as it is going to get. You can never expect 100% in this kind of endeavor - with 8000 monitors, some people will have quit, gotten sick, lost their phone, or perhaps never existed in the first place. With all those caveats, the fact that we got well over 95% of our data is a real feat.

The data clerks have wrapped up and are heading home. They'll be back again to take care of some accounting and claim their prize: a certificate. Certificates of participation and commendation from well-regarded organizations like NDI are highly valued in a society where resumes ain't always trustworthy.

From the tech perspective, the Gods of IT were apparently appeased today; all of the computers and networks behaved beautifully.

Well folks, thanks for reading along with our adventures as the NDItech team supported Project 2011 Swift Count. I'm going to wrap this up here as the focus shifts to the inner sanctum. Late into the night our stats wizards, election experts, and the leadership of Swift Count's coalition will be sifting the data, staring into their crystal balls, and deciding What It All Means.

11:50 AM

Playing with maps. We had grand ambitions of getting a proper mapping platform up ala Burma Election Tracker, but our developers didn't get to it and I ran out of time. So now I'm working with Google's Fusion Tables, which is basically a really badass spreadsheet that can also contain mapping outlines of geographic areas. So one can, for example, have the vote results for a certain party by local government area in one column, and then shade the puzzle piece a pretty color in the map view. Maps are pretty eye candy, but also can tell some compelling stories.

We're also working on a lot of data cleanup today. When you're sending in text messages that look like this Psc79019ba4bb15bc2be3bf1bg1bh5bj5bk2bm1bp196ca4cb1cc1cd5ce5cf1cg2ch1cj1ck1cm2cn1cp1cq2 it isn't too surprising that people will make mistakes from time to time. We have some some automated processes to catch obvious screwups, but I've been working through some other database sweeps to catch other areas of concern.

7:00 AM

Good morning Nigeria! Thank you for not not crashing this week, Servers, and allowing me a solid 5 hrs of sleep. I got in before the clerks to test satellite fail-over in case we have another issue with "unscheduled maintenance" like we did yesterday. Sometimes feels a bit like I'm running the Department of Redundancy Department around here. It worked.

Social media should be a component of any organization's communications strategy. (NB: that does not mean the entire strategy.) Even for countries with very low net access, it's a useful tool for reaching important audiences outside - policymakers, news agencies, financial donors, and so on. While I've been able I've been assisting with Swift Count's Facebook page, their Twitter feed (@SwiftCount, our ICT team's Twitter feed (@NDItech), and when I can my personal Twitter feed (@CDoten). Updating these don't take a lot of time, but it takes a bit of discipline to keep remembering to put stuff out there. I definitely haven't had time to be riding the tweet stream to see what's going on, so I'm probably less well informed about the elections than most of you, dear readers.

Dawn breaks over Nigeria, the Sheraton and Project 2011 Swift Count. The second day.

1:30 AM

Tthe senior staff with Project Swift Count are writing initial drafts of The Statement. The Statement is the most important document of this whole process, since it's how Swift Count will characterize the information to the world.

We've got a lot of data, so now they have to decide what kind of story to tell with it. What are the real anomalies or outliers? Were there consistent trends across the country? What was particularly good or bad, and how did it compare to elections last week - and 4 years ago?

There's two parts of the information to consider in drafting a statement. First, the qualitative analysis of the election, which comes from a checklist of 27 questions. Were any security personnel present? Did anyone attempt to harass voters? Did polling officials mark a cuticle on a left finger of every accredited voter? (really) Second, the quantitative number-crunching from the final results. This includes the total votes cast, registered voters, accredited voters, spoiled ballots, etc. plus the votes per candidate.

That's it for me for the night. Time for a quick check-in on my mom, who just had knee surgery (hi mom!) and I'm off to bed.

9:50 PM

A lot of data in our systems now. The statisticians and elections experts are crunching numbers and starting calculations to see what all these texts tell us. The web system has some built-in analysis widgets, letting you see pie charts of whether (say) there was police protection, and graphs of projected vote results. As more info pours in, these become more accurate and trend lines start to stabilize.

Trying to figure out how to grab some dinner other than the sawdust cookies. Not exactly a lot of Chinese delivery options, sadly. My colleague Michelle has been making sure that our zone coordinators are getting some food - many of them have been working since 7 AM without a break or anything to eat. If they collapse they're not nearly as effective at keeping the data clerks motivated.

8:10 PM

Big milestone - over half of our observers have reported the closing summary information from their polling place. This isn't the counts per candidate, but rather the totals - number of registered voters, number of accredited voters, number of rejected ballots, number of total valid votes, etc.

Meanwhile, the clerks on the floor are chasing missing or confusing information. This involves a lot of screaming into the phone "No, BB! What is the number you wrote down for BB?" English is the primary language in Nigeria, but a lot of folks speak it with very thick accents. I almost spit out my coffee when I heard another American sitting near me apologizing for her poor speaking ability; she was being chastised by the gentleman on the other end of the phone for being unintelligible. Other amusing interactions are conversations that go something like "Are you at your polling place? You are? It sounds very quiet there. Can I speak to your observation partner? *click* Hello? Hello?"

Some of the more senior people are busily following up on critical incidents, trying to verify the details of what happened.

I'm currently attempting to eat my body weight in the weird woodmill-byproduct cookies they put out here.

7:40 PM

Whee. We lost our VSAT for a while. Apparently our provider decided to do some maintenance on Election Day. Thanks, buddy. When it came back up one of the servers got very confused, so it had to be restarted. Fortunately we had our twin system for the other half of the team, so while we were diagnosing and fixing the issue, our data clerks were able to double up on the other.

Everyone's getting a little tired around here. We've switched shifts, so there is a new group of data clerks in. For those of us staff who've been here all day Coffee Round 2 arrived at an excellent time.

5:05 PM

Looks like we're doing OK on the message flow. We appear to be close to peak throughput, but test messages are going through. Still, when you have 8000 observers sending as many as 5 messages each, it's a lot to deal with. Trying to diagnose things like this is a real challenge due to the opacity of the various systems. How do you know what observer sent a message when? How do you know when the network passed it along? How do you know what the gateway provider is doing with the information?

Once the messages hit our server we can watch them pretty closely, checking flow rates on our remote RapidSMS server (proving its name today) and bandwidth utilization from our VSAT. In the data center I'm sitting like a mother hen over our "servers" and making sure that they're not running out of processing power, RAM, or bandwidth, and that our local network is staying up. Actually, everything has been pretty darn stable today.

* Three of them are laptops. I heart how much processing power has improved in the last few years.

3:30 PM

Messages flowing through from observers reporting in on the end of the accreditation process, but slower than we're expecting. Could be they're taking longer to report than expected, or it can be there's a bottleneck in the system somewhere. Developing...

1:20 PM

Incident reporting is a core element of any election monitoring mission. Critical Incidents, as they're called, are reported by observers whenever they see something flagrantly wrong at their voting place: the polls don't open at all, there is no security present, there are egregious violations of the rules for accreditation, multiple voting, etc.

These incident reports are important, but easily distorted. Every election has flaws, but not every flaw has a significant impact. We're filtering them through their local coordinators who can confirm the details, double-check with other people, and act as needed. Verified problems then get passed up to our national center where our leaders can get on the horn to the election commission, local police forces, or any other powers that be to yell at them to fix problems.

Critical incidents on their own can paint a very misleading picture of an election - if anyone can report any minor problem, and there's no sense of context, you can get the 10:00 News effect - there's nothing but problems to be seen. This can have distort the view of the electorate and increase their mistrust in the results, despite the fact that the election may overall have been very clean.

Since we want to do some recurring processing of incident data to look for patterns, I've set up a script that pulls the current information from the database, saves it to a spreadsheet, and emails it to designated recipients every 15 minutes.

Also, it's lunchtime. Pizza from Luigi's it is - surprisingly good stuff. Far superior to the dissipointing DC delivery I usually get.

10:55 AM

Working on IT in development means that you rarely get to do things the "right" way- you have to make do with less-than-ideal conditions and without fat private sector budgets. So we get creative.

Tim, our software developer, is a bit like Scotty from Star Trek in this way - able to make things work way outside of their supposed limits. According to the MySQL docs, to have two databases copy their information back and forth in real time (master-master synchronization) requires a 100-meg hard-wired Ethernet connection. We're doing it over a VSAT, which is, oh, less than 1/100th of what we are supposed to have for bandwidth with high latency and built-in flakeyness to boot. Thank God it's not raining.

It's working way better than it should be, but there was a bit of an alarm when we noticed we'd lost sync earlier. Since we're processing text messages on one server and phoned-in responses on another, it became clear that the numbers were skewing between them. It's back up and running now.

10:15 AM

So what's a PVT? Can you eat it? Partial Vote Tabulation is a system by which you get election returns from a statistically based sample of a targeted region. Because voting tends to be highly homogeneous by area, the results from one polling place tend to tell you a lot about other polling places in that region. This is as true in the US as it is in Nigeria.

If you grab a sufficient amount of information from a carefully selected set of locations, you can project with a high degree of accuracy the aggregate totals across the entire region within a certain margin of error. The more data, the lower the margin of error.

The thing that makes today unprecedented in the annals of election monitoring is the fact that Project Swift Count is not only tracking results nationally, but also the Gubernatorial results in several high priority states. That means that each targeted state also needs a sufficient number of observers at an internally-selected sample of locations. Because each is independent, one could get a strong response from one state and be able to make strong statements about the results, but not get enough data in a another to be able to make an authoritative statement.

9:45 AM

Things are humming along well here at Swift Count. We're at about 80% of observers checked in, so at this point our data clerks are hunting them down on the phone while we send out blast SMS reminders for those who may have forgotten or accidentally screwed up the messages. Since our observers are randomly distributed across the country, some of them are also in places with no phone service, so we will only get results from them when they are able to get back into range.

Today we're doing several concurrent PVTs (partial vote tabulations) in high-priority states selected because of their significance, the likely closeness of the vote, or a history of violence. At the same time, all our other observers have fanned out to their normal polling places across the country so we will be able to watch the nation as a whole as well.

It's probably time to explain a PVT in more detail. One turn around the floor and a quick server check, then I'll do so.

Coffee watch: it arrived at 9:15. Whew. Of course, it was unlabled, so I started with a cup full of tea. #fail.

8:30 AM

It's been a busy but relatively stress-free morning here. The pre-dawn web browser reconfiguration worked fine, and we got our morning allotment of rousing speeches from our Program Manager and some of the other leadership. I've been wandering the floor with some of the other managers to look for problems. Pro tip: if a dataclerk appears to be asleep, it's likely because their computer isn't working.

Coffee watch: 2.5 hours. No signs of liquid sleep replacement yet. Crew beginning to grumble.

6:30 AM

Turn on the lights, check the networks and computers. Looks like only about half of them were set up as promised. Time to scurry about and reset Chrome bookmarks.

Coffee watch: much too early for reasonable hopes of delivery. Made some Nescafe. (George Clooney, you've done some good things in your life, but shilling for this swill?)