WELL - Resource Centre Network for Water, Sanitation and Environmental Health

Monitoring of water supply coverage will help to ensure water supply for the millions of people who still lack convenient access to a ‘safe’, reliable and affordable water source. According to the WHO/UNICEF Joint Monitoring Programme, more than one in four people in the developing world lack access to water (WHO/UNICEF 2004). Targets have been set in the Millennium Development Goals (MDG) to accelerate the improvement of this situation. Quantifying access through monitoring will be essential to attain such goals, although the process to achieve these goals will be as important in order to achieve sustainable coverage.

This fact sheet aims to clarify different steps involved in measuring access to water by water coverage surveys and highlight some of the problems that may affect current monitoring.

There is a common misconception that monitoring should only be done by specialised professionals. However, everybody uses similar techniques in their daily lives. When we buy fruit or vegetables we look at them, feel them or even smell them to assess the quality of the product. When all of them cannot be checked individually, we examine a small sample and consider it representative enough to give the buyer confidence in what is being purchased. Monitoring uses a similar process of sampling when a number of households is asked in a survey about their access to water. While, monitoring statistics may often appear daunting, they are in reality only a small part of the monitoring process, as shown below. Despite their small role, the statistics are often given a central place.

Monitoring is not a stand-alone activity but a tool used to feed information into other activities. It can be divided into different steps as shown in Figure 1 . The first four steps ( Why, ‚ What, ƒ How and „ Validate) relate to the definition and indicators to be measured, while the three following steps (… Sample, † Analyse and ‡ Infer) are about the representative data collection and analysis. Often statistics is seen as the central part of monitoring, but in fact only one step in the whole process, step † (analyse), uses statistical formulae. Steps ˆ, ‰,, and Š follow the data analysis and feed back to the need ( Why Measure?) for a future round of data collection.

Collecting water coverage information without any clear purpose or failing to use collected data is a waste of resources, money and staff time as well as the other people’s time, including the target population.

Despite the importance of collecting information it is surprising how little measuring actually happens and how often monitoring is used only for reporting.

First of all we need to define clearly what to measure. In terms of water coverage, possible definitions could be:

Water in adequate quantity for hygiene purposes and of adequate quality for human consumption.

Access to a reliable source of water which supplies adequate quantity and adequate quality of water in a convenient way.

Although these definitions seem clear, they still cannot be measured as they lack clear measurable outcomes. Terms like convenient, reliable, a source, adequate quantity and adequate quality need to be defined so they become objectively measurable.

One of the factors of a ‘reliable’ source might be that it is not intermittent. A measurable definition of ‘reliable’ could be a source that is available ‘without a day of interrupted supply in the past 7 days’. This might not be a complete description of a ‘reliable’ source, but it is a clear and satisfactory measurable definition for monitoring.

It is often impossible to measure a variable of interest in large household surveys. For instance, to assess water quality might require expensive laboratory tests. The ‘type of water source’ such as ‘piped water’ (often good quality) or ‘river water’ (often bad quality) could then be used as an indicator being ‘close’ to the information required. The ‘type of water source’ is referred to as a measurable proxy indicator for water quality. Although this is not always a perfect proxy it helps to get a better idea of water quality through a measurable indicator.

Households do not always use a single source of water. In rural areas in particular, seasonal changes often occur. Where seasonality and multiple sources are relevant to coverage, definitions and measurable indicators might have to take this into account. An indicator can be built up from multiple answers, as often a single question cannot elicit all the critical aspects of access.

Definitions are context-specific. The poor would have a different definition for access from the more affluent, a government or a funding agency. Women, involved in the day-to-day water collection, will define access to water differently from men. Involving the target population in defining access (and how to collect the necessary information) will promote a consensus among the target population and give a better understanding into the local water access problems.

Benefits from water supply can only be achieved if water sources are used. In measuring coverage there is often discussion about whether use (which is practice) should be measured, or access (Which is more theoretical, but for which providers can more easily be accountable)

Figure 2: Aspects of data collection for water coverage monitoring in a survey

There are different ways of collecting water-related data and the methodology will depend largely on the purpose of the monitoring work and the resources available for it. No single data collection method will be able to provide all the information needed for projects as different types of information will be collected in different ways.

There are two main and complementary methodologies in data collection; qualitative and quantitative methods. Quantitative methods aim to measure a small number of quantitative indicators and characterise these in figures. Quantitative data such as water coverage are often expressed in percentages of people or households having or not having access. Qualitative methods are more exploratory and analytic, seeking a diagnosis or description of a problem. They are also better able to discover information that was unexpected and therefore not explicitly asked for in the survey.

Some qualitative approaches have been proposed lately, which aim to quantify qualitative data to facilitate interpretation and comparison (van Wijk 2001) . This blurs the line between the two approaches.

Participatory methods aim for a high involvement of the target population in the process of data collection, while non-participatory methods use mainly professional ‘external’ people for data collection.

Continuous or routine data collection is a continuous process of collecting and updating information, which is in contrast with point in time activities such as surveys which aim to collect information over a short period.

Survey methodologies often use a mix of approaches to collect data and the way each factor will be used can vary on a scale between the extremes shown in Figure 2. Often coverage data will be collected over a short period to obtain a point estimate. They generally are quantitative, non-participatory cross-sectional survey (as illustrated in the right column of the scale in Figure 2.). This is also the focus of this fact sheet, but it is not the only or preferred way of obtaining accurate data.

Usually water coverage surveys are done in a non-participatory way by an interviewer asking questions. However people can or do not always answer the questions accurately, so other methods of data collection like spot observations and demonstrations, should be considered. Information collected through different methods can be compared to check its validity. This is called “triangulation”.

4. Validate

Validation is the part where what is measured and how it is measured is scrutinised to make sure that it can give the information for which the data were collected. Validation is often not a part of a survey as it needs far more information and rigour in data collection than the basic data collection. However the collected data and the method of collection have to be based on some type of past validation before being included in a survey. Typically validation would compare the collected data with information which is judged to be as reliable as possible. This can be done during pre-testing, which is essential before any survey and consists of doing a small survey in a similar population under similar conditions. For example, if the type of water source is used as a proxy for water quality, a sub-sample during the survey can be submitted to more rigorous tests such as water quality tests to determine if water quality is related to the type of water source as has been assumed.

Validation should not be limited to what is being measured, but how it is collected. When data are collected by asking a question, validation involves checking whether the people are able or inclined to give accurate information.

Water collection time is often used as a proxy for water quantity according to the relationship shown in. However, it might be useful to measure both, in order to confirm this relationship for a local context. It is also essential to verify if interviewees estimate correctly the time required for a water collection journey. The question can compare water collection time to the duration of certain day-to-day activities, such as the cooking of certain foods, to obtain more reliable results. Validation feeds back into what and how to measure until indicators are suitable to be used.

The basic sampling unit is the unit on which the data are collected. A practical basic sampling unit for water coverage can be ‘the household’ because all people in the household are likely to use the same water source at home. However for some aspects such as ‘sustainability of water source’ or ‘number of beneficiaries per source’ the water source can be a more suitable basic sampling unit.

Collecting data from every household is often impossible or impractical, so a sample of households is taken from the target population. The conclusion based on the sample can apply to the total population if the sample was representative.

Taking representative samples is often complex, particularly in low income countries. It is seldom straightforward to make a list of households which is needed in traditional sampling methods, because of a lack of addresses or other household locators. Also people who have access to water are usually clustered around a water source. This geographical clustering of people with access to water causes bias in the sample, and even if this bias is controlled, it requires increased sample sizes. To make data collection more accessible for management and evaluation of projects, more suitable sampling methods need to be developed.

Practical implementation

Although practical implementation of a survey will be crucial to the validity of the collected data, it does not always get the attention it deserves. Practical implementation starts with the decision to do a survey until the data are analysed and made available. At each level the convenience of implementation and the level of training received for the task ahead will determine how valid the outcome will be. Three particular points can be identified:

Before any data are collected or even any pilot survey is done, it is important to think how the collected data will be used, what assumptions have to be made (possibly checked) and what analysis will be done. Necessary information is often found to be lacking only during the analysis stage. A complete analysis of the pilot data might help to identify such problems in good time.

Various agencies invest in water and sanitation not because they are a basic right, but to improve public health. Often there is a temptation to measure the health impacts of water and sanitation in such programmes as the outcome. However this has many pitfalls, and often results in meaningless results.

7. Infer

Once the sample is analysed, it becomes possible to apply the results to the whole of the target population. This is possible only if the sample was representative.

Questions worth asking include: Do all groups of people, even marginal groups, have an equal chance of being included in the sample? How could excluded groups affect the result? Also those values missing due to non-response are worth analysing when possible. It enables us to document limitations of the methods used and problems encountered, which will add credibility to the analysis.

In statistical jargon, the process of assuming that results from the sample are also true for the population is called inferring or inference.

Correct analysis of a non-representative sample will give a result that is not representative for the population as a whole. Therefore, no inference or conclusions can be made from it that will be valid for the whole population.

There is a difference between analysing sample data (statistical work) and the analysing of results (a managerial responsibility). The conclusions drawn from the statistical results will feed into step ‰ ‘action’. One action will be to define the information for the next round of data collection which will feed back into step ‘why?’. An additional need in a consequent round of data collection could be to make the collected data comparable with data previously collected, in order to measure the extent of progress.

Surveys are often done by people external to the surveyed population. The collected data are often not used outside the organisation for which they have been collected which makes them de facto the sole ‘owner’ of the data, despite the rights of the target population.

Information is crucial to any decision and advocacy process. Collected information should be documented for possible further use and where possible made accessible to a wider public. Water coverage data should be disseminated vertically, i.e. aggregated (upwards) or disaggregated (downwards) as well as disseminated horizontally among different sectors.

Documenting data and data analysis is essential but requires additional resources such as time and funding which have to be allocated. This is particularly true for data sets and data analysis that are made freely available to the public.

Cost-benefit of monitoring activities

With resources in short supply, authorities are often reluctant to get involved in data collection, as these activities appear to have no direct impact on beneficiaries. This is particularly true if the authorities are uncomfortable (or unfamiliar) with collecting such data. However, accurate data can allow better targeting of resources and can help to justify further investment. Monitoring can often help to make substantial savings and to avoid redundant activities. It can help to justify actions that are counter-intuitive.

Useful tools for surveys

Various simple tools are an invaluable help during surveys. For instance, an automatic numbering stamp ensures unique numbering of each questionnaire form.

Free software programs are available for entering data into a computer and for analysis. The most popular programs are:

The subject – Monitoring of coverage – discussed in this fact sheet is just a small part in the monitoring efforts of the water ‘sector’, which also includes other factors such as user demand, and satisfaction, institutional capacity and responsibilities, operation and maintenance as well as environmentally sustained services.

The challenge … “is getting the best possible information to the people who need it – and then getting those people to actually use the information in appropriate ways for the intended purpose.” (Patton 1990)

Further information on assessing hygiene improvements can be found on the Website of the former USAID Environmental Health Project:

WELL FACTSHEET Go Back

4. Validate

Practical implementation

7. Infer

Cost-benefit of monitoring activities

Useful tools for surveys