Inconclusive “surveys”, hyperbolic claims
The PTI, in a recent publicity campaign, brought up it’s claim, again, of scores of students from private schools in Khyber Pakhtunkhwa shifting over to government schools. Don’t take our word for it, says the party, for these were the finding of a survey conducted by the Adam Smith International.
خیبر پختونخواہ میں طلبا کا سرکاری اسکولوں پر اعتماد کا اظہار
2سال میں ایک لاکھ 4ہزار بچے پرائیویٹ سے سرکاری اسکولوں میں منتقل ہوے #KPKUpdates pic.twitter.com/0zoYqJ9lat— PTI Rwp Official (@PTIOfficialRWP) May 26, 2018
Before discussing reporting of the survey in electronic & print media, we will first analyze the survey methodology, its conclusions and the organization which carried out the survey.
Over the years, enrollment in government schools as a percentage of total enrollment is on the decline across Pakistan. In Khyber Pakhtunkhwa, primary level enrollment in Government schools was 78% of total enrollment in 2004-05 which declines to 69% in 2014-15 according to PSLM surveys. Both demand and supply side factors are being cited in the literature for this shift from government schools to private schools. In this backdrop, any report or survey claiming that parents are shifting their kids from private schools to government schools is indeed eye-catching.
Being a skeptic of PTI’s ‘change’ in education sector in KP, I obtained ASI’s survey report (18 pages in total) which was published in May 2016 with the title ‘Survey of Children Leaving Private Schools And Joining Government Schools’. Following are some key points in the survey.
- The survey was conducted by ASI which works in close collaboration with government of Khyber Pakhtunkhwa in improving education sector delivery in the province. Thus, the survey is NOT third party evaluation of education reforms in the province, as best practice evaluation would require.
- The figure of 34,000 students leaving private school for government schools, is a claim of KP Elementary & Secondary Education Department and not of the survey itself. Reports in print and electronic media wrongly attributed the figure to an ‘independent’ survey.
- The survey report says that it first conducted a pilot survey in one district, District Haripur, to find out the reasons behind this migration. Interestingly, the report says that the results from the pilot survey were ‘inconclusive’ as the majority of students left private schools because either their private schools were closed down or no further grades were available in these private schools. How they can term these results inconclusive when they took a sample from the population of those total 34,000 students in the province? Surprisingly, the pilot survey from one district has a much greater sample size (182 students) as compared to a ‘detailed’ survey undertaken in four (4) districts with a total sample size of only 278 students. Also, the survey managers prefer to exclude the Haripur district from the more ‘detailed’ survey.
- The detailed survey conducted to ‘delve deeper’ for understanding the phenomenon took the government claim for granted and did not look at the reverse migration. There may be a lot of parents who may by switching their kids from government schools to private schools over the same period. A better understanding of the phenomenon can only be achieved if the survey have looked at net migration.
- For the second detailed survey, the report claims that it only looked at those students who migrated from private schools to government schools for ‘other reasons pertinent to the quality of education’. This is a classic case of selection bias where the total population is no more 34,000 migrated students but a portion of it which left private schools for government schools due to quality of education. Kids leaving private schools for government schools as parents may not be able to afford it any more were left out from the survey.
6.The detailed survey selected four districts for the analysis with very tricky reasoning. The report claims that at the first level, ‘districts that only had data from either male or female schools’ were dropped. This, according to the report, drop five (5) districts from the analysis which means that either education reforms were not uniform or the data is not available. The sampling strategy further says that another four (4) districts were removed from the survey for security reason. The survey report did not name these four (4) districts. How it is possible that while kids can attend schools in these districts but parents of the same kids cannot be surveyed due to security reasons? At the third level, another four (4) districts were dropped out from the analysis where migration of students from private schools to government schools is less than 500 students. Does this mean that the province-wise education ‘reforms’ couldn’t reach these four (4) districts? Remember, replies to Imran Khan question of education reforms in his public gathering at Bannu? For the final analysis, a total of 4 districts were selected; Abbottabad, D I Khan, Peshawar & Swat.
- After tricky selection of districts for the survey, it is now turn to select students’ families for the survey. The survey team obtained phone numbers of 1,124 randomly selected students whose guardians would be reached via phone calls. On phone calls, the survey team could establish contact with guardians of 945 students where they select 278 households for the detailed survey. The remaining (945-278=667) 71% households of the total were considered ‘invalid’ households because they shifted their children from private schools to government schools either because their private schools is being closed down or no further grades were available in these private schools. The survey now analyzes these 278 households to find out reasons behind their migration. Imagine!
- The ASI survey report claims that its survey team was accompanied by a focal person from either the DEO Male or Female’s office to conduct individual interviews with the head of households. This was probably done to ensure transparency and independence of the survey? Even here, 52 interviews were done via phone.
- Questions were asked about past decisions of the guardians about the reasons for enrolling in private schools and not in government schools. Majority responded that at that time, private schools has better quality than government schools. The interesting part in the report comes when the survey team asked about their current choices. Table 6 and Table 7 in the survey reported answers to two (which is in fact one?) question. According to table 6 titled ‘Grade-wise Reasons for Leaving Private School’ during last 6 months, 139 guardians of the students responded that they left private schools because these schools are too expensive. This comes around 50% of the total households which mean that half of the kids were shifted to government schools due to poverty. Only 78 guardians responded that they left private schools because of low quality. Remember selection of the total sample size of 278 households? Despite removing the ‘invalid’ households, it still has households where kids are being shifted from private schools to government schools due to their financial constraints. Interestingly and surprisingly, table 7 titled “Grade-Wise Reasons for Enrolling into Government Schools” is also being produced. Are table 6 & 7 not reporting same exact question? Here the survey team found that 45% of total children are shifting to government schools because its quality is better than private schools.
Reporting of the Survey:
Now let’s come to the reporting of this survey in electronic and print media. One report appeared on SAMAA TV and tweeted from the official account of PTI. The video report says “But with [PTI} government efforts, quality of education improved in government schools and thus 34,000 students got admission in government schools. This fact was revealed in a survey by a non-governmental organization”. As shown in the earlier analysis, neither the survey revealed this nor the findings suggest any such thing. Also the whole selection process of the sample students is faulty.
