MONEM FAROOQI
The Asian Development Bank (ADB) has shown its reservation on reliability of Federal Bureau of Statistics (FBS) by saying that frequently changing data collection methodology of the FBS has affected the reliability and comparability of its survey data.
In a latest appraisal documents available with Pakistan Today, the ADB has blamed its field supervisors for the negligence. ‘This is a poor defense – this is a serious sampling task and should be managed regularly well before the survey is started, it added, proposing adoption of reliable and independent internal mechanisms for supervision and validation of fieldwork to address the problem.
For instance, the documents pointed out that the different data sources were not necessarily comparable in terms of sample design, seasonality, or methodology, often used to examine poverty trends, making the data unreliable.
Criticising the Pakistan Social and Living Standards Measurement (PSLM) survey, the bank said that it did not contain adequate monitoring indicators. ‘Fertility and child mortality rates, for instance, cannot be estimated through this survey’ the documents pointed out.
The bank said that the credibility of data depended largely on how independent the executing agency was, and, more importantly, the quality of the data itself.
‘The FBS ensures data quality by carrying out consistency checks and ‘cleaning’, but validating survey data in the field is the key to enhancing data quality’. ‘This is usually done through a Post-Enumeration Survey (PES), which is not common in Pakistan’. An alternative would be to validate FBS fieldwork externally, although this is not a common practice among data generating agencies in the developing world.
The Household Integrated Economic Survey (HIES) is the main source of data for poverty estimates in Pakistan, and should be strengthened, at least to the extent of producing regular, credible data, even at intervals of two or three years.
“The reliability and credibility of Pakistan’s poverty database, which is generated primarily through household surveys, has been debated for a long time. Issues of concern include updating of the sampling frame; survey comparability; availability when needed; frequent changes in field methodology; and the quality of questionnaires used.” The bank’s documents mentioned.
ADB has questioned that method of third-party validation, which it marked that may in fact weaken the FBS’s fieldwork supervision ability. Rather than introducing third-party validation to verify FBS household-level data, reliable and independent internal mechanisms for fieldwork supervision and validation could be developed to address the problem, it added.
‘It is important that such a mechanism establishes its credibility through long-term improvement in its data collection techniques. This will boost the confidence of the local staff and strengthen the FBS’s ability to carry out surveys’, it mentioned.
The bank’s background paper-2 assesses the reliability and credibility of the household surveys that generate data for poverty analysis in Pakistan. It focuses on the data generated by four household surveys: the Household Integrated Economic Survey (HIES), Pakistan Integrated Household Survey (PIHS), Pakistan Demographic Survey (PDS), and Labour Force Survey (LFS), all the four are carried out by the Federal Bureau of Statistics (FBS).
‘The FBS has initiated a new survey, the Pakistan Social and Living Standards Measurement (PSLM) survey, but it does not contain a module on birth history to estimate fertility and child mortality rates. The PDS remains the main data source for these indicators.
The bank said that in general, labour data remained inadequate; it would be worth investing more in the Labour Force Survey (LFS) to make it an annual survey. The quality of data produced by the household surveys so far implies that the quality of training and supervision of surveyors needs to be improved. One reason for the recent controversy over household size, which is an important variable in estimating poverty, was that ill-trained enumerators did not enlist household members properly.
The bank said that the supervision system should be changed to allow supervisors to manage their field teams for the entire survey period, as was done by the FBS in the PIHS rounds. Household survey monitoring teams consisting of regular field staff posted at national, regional and field offices across the country could also monitor surveys more effectively.
