Our Tanzanian Pilot, in Numbers

Reflecting on our completed pilot study last Thursday as we slowly bumped our way towards the airport in Dar es Salaam, Kat, Hugo, and I were very pleased with our progress. Although we faced a few hurdles (detailed below), we managed to adapt and carry out our study adequately. Most importantly, the games are better than ever, and the kids thoroughly enjoyed them–from the first minute to the last. Here are 1) some challenges we faced, and 2) some (but not final!) measures of what we and the kids were up to. First, the challenges:

• 10/10 tablets that we carried with us on the plane (from Brazil, NYC, and the Netherlands) arrived safely
• 7/15 tablets that we shipped via FedEx from NYC were “lost/pilfered”
• 8/15 tablets that arrived in Dar es Salaam have (still!) not been released from customs, even after
• 7 days we waited for the shipment, as advised by the local shipping agents, before we spent
• 2 days sitting in the airport cargo bay, trying to speed up the arcane customs-clearing procedures
• A depressing number of emails to FedEx International Distressed Shipments urging them to contact their affiliate CDS in Dar and asking them what the delay and/or cause of loss was, all of which they seemed ignorant of. (Note: We are very grateful to the teams at NYU and Global TIES for their support during these difficulties!)
• 14 children of 40 enrolled showed up on the first day of the study, forcing us to re-design it to some extent*
• 2 days the school delayed the start of the study, because exams did not finish as early as expected (meanwhile, we hoped the tablets would arrive..)

Now the positives, which more than compensated for the hurdles:

• >40 commits by Hugo and I to our git repository (not only fixing bugs, but adding great features!)
• 39 children given End of Grade Reading and Math Assessments (EGMA/EGRA) at the beginning and end of the study (many thanks to Claudia and Miu!!)
• >21,000 trials completed by the 23 children in the treatment condition, who paired up on 10 tablets, playing our 5 games we were most eager to test
• 5 hours (maximum) time spent on tablets over 5 days by children in the treatment condition, with some stratification due to sporadic absences
• 5,000 trials completed by 12 of the children in the control condition on the final day (after their post-tests were completed)
• >3 children, informally surveyed, named each of our 5 tested games as their favorite (one child couldn’t decide, and said all were!), and yet we observed all children playing every game each day, switching (sometimes often!) as the mood struck them
• >10 new friends in Tanzania, including the wonderful staff of Soma-E&D Readership and Development Agency, our local NGO partner, the friendly hotel staff, locals of Zanzibar, and a member of parliament and political consultant who gave us a grand tour of Bagamoyo. It’s not possible to put numbers of any kind on our gratitude to and experiences with these fine folks!

My next post will (finally!) describe the philosophical, psychological, and design principles behind our games–although the explanation I prefer is to hand you a tablet with no other explanation, and let you explore: this might be the best explanation of our philosophy, at any rate. ;))

*Worried that waiting for all control and treatment participants to show up and finish pretesting before starting treatment would put us in a position of having virtually no time left for treatment, we chose to re-assign the participants who showed up in the first two days to the treatment group (23, in the end), and the participants who showed up later to the control group (16). Of course, this introduced a confound that could compromise the study: if it turned out the participants who showed up on-time were different (e.g., easy to imagine they have better memory–they showed up! :)) than the delayed students, we would have an unbalanced baseline for the two groups. As it turned out, the baselines are not significantly different, but the two groups are still not necessarily equivalent. But for a pilot, we’re content to have treated a large group of students, and to have a similar control group to compare to. Detailed results soon!