This page has been developed by M.Peyre, CIRAD-AGIRs and L.Hoinville, RVC based on the RISKSUR consortium expert work on evaluation attributes.
Evaluation attributes : definition
The different types of evaluation attributes
Evaluation attributes have been divided into four categories
- Organisational attributes – assess the overall structure and processes of surveillance which will have an impact on the function, effectiveness and value of surveillance
- Functional attributes – assess how well surveillance functions, the function of surveillance will influence its effectiveness and value
- Effectiveness attributes – assess how effectively the surveillance achieves its objectives, the effectiveness of surveillance influences its value
- Value attributes – assess the value of surveillance for stakeholders
Organisational attributes
(reference to design wiki)
| Surveillance system organization | An assessment of the organisational structures and management of the surveillance system including the existence of clear, relevant objectives, the existence of steering and technical committees whose members have relevant expertise and clearly defined roles and responsibilities, stakeholder involvement and the existence of effective processes for data management and dissemination of information. |
| Risk based criterion definition | Validity and relevance of the criteria used to define risk in risk-based surveillance and the approach/method used for their identification |
Functional attributes
| Acceptability and engagement | Willingness of persons and organizations to participate in the surveillance system, the degree to which each of these users is involved in the surveillance. It is closely related to their beliefs about the benefits or adverse consequences of their participation in the system. |
| Availability | Ability of the surveillance system to be operational when needed. |
| Sustainability | Robustness and ability of system to be ongoing in the long term. |
| Compatibility | Ability of the surveillance system to integrate data from other sources and surveillance components. |
| Flexibility | Ability to adapt to changing information needs or operating conditions with little additional time, personnel or allocated funds. Flexible systems can accommodate new health-hazards, changes in case definitions or technology, and variations in funding or reporting sources |
| Multiple hazard | Refers to whether the system captures information about more than one hazard. this can be : a) Surveillance which has been designed to investigate the occurrence of one main hazard but is also used to investigate the occurrence of other hazards (mother-child). b) Surveillance which is designed to investigate the occurrence of multiple hazards in parallel (multiple-hazard) c) Surveillance which is inherently multi-hazard e.g. passive surveillance and surveillance carried out at abattoirs (general) |
| Risk-based criterion definition | Validity and relevance of the criteria used to define risk in risk-based surveillance and the approach/method used for their identification |
| Simplicity | Refers to the surveillance system structure, ease of operation and flow of data through the system. |
Effectiveness attributes
| Bias | The extent to which a prevalence estimate produced by the surveillance system deviates from the true prevalence value. Bias is reduced as representativeness is increased. |
| Coverage | Refers to the proportion of the target population that is included in the surveillance activity when using risk-based approaches coverage refers to the proportion of the selected populations included. |
| False alarm rate (inverse of specificity) | The proportion of negative events (e.g. non-outbreak periods) incorrectly classified as events (outbreaks). This is the inverse of the specificity but is more easily understood than specificity. |
| Negative predictive value (NPV) | Negative predictive value refers to the proportion of epidemiological units (e.g. animal, holding, herd) classified as free from disease or infection by the surveillance system which are actually free from disease or infection. It depends partly on the sensitivity and specificity of the surveillance system, but is also influenced by the disease prevalence in the target population |
| Positive predictive value (PPV) | Positive Predictive Value refers to the proportion of epidemiological units (e.g. animal, holding, herd) classified as diseased or infected by the surveillance system which are actually diseased or infected. It partly depends on the sensitivity and specificity of the surveillance system, but is also influenced by the disease prevalence in the target population. |
| Precision | The confidence interval of a numerical estimate. A precise estimate has a narrow confidence interval. Precision is influenced by prevalence, sample size and surveillance approach used. |
| Representativeness | The extent to which the features of the population of interest are reflected by the population included in the surveillance activity, these features may include herd size, production type, age, sex or geographical location or time of sampling (important for some systems e.g. for vector borne disease). |
| Robustness | The ability of the surveillance system to produce acceptable outcomes over a range of assumptions about uncertainty by maximising the reliability of an adequate outcome. |
| Sensitivity (/detection probability/detection fraction) | Sensitivity of a surveillance system can be considered on three levels. • Surveillance sensitivity (relevant for case detection and prevalence estimation) refers to the proportion of individual animals or herds in the target population of interest that have the health-related condition of interest that the surveillance system is able to detect. This is determined by the sensitivity of the testing protocol used and the coverage of the population and has also been referred to as detection fraction or detection probability. • Surveillance sensitivity (for outbreak detection) refers to the probability that the surveillance system will detect a significant increase (outbreak) of disease. This may be an increase in the level of a disease that is currently present in the population at a low level or the occurrence of any cases of a disease that is not currently present. • Surveillance sensitivity (for demonstrating freedom and early detection) refers to the probability that at least one case will be detected if disease is present at a certain level (prevalence) in the population. |
| Timeliness | Timeliness is usually defined as the time between any two defined steps in a surveillance system, the time points chosen are likely to vary depending on the purpose of the surveillance activity. For planning purposes timeliness can also be defined as whether surveillance detects changes in time for risk mitigation measures to reduce the likelihood of further spread. |
Value attributes
| Cost | The concept of economic cost includes 1) the losses due to disease (e.g. reduced milk yield, mortality), and 2) the resources required to react to disease in a system (e.g. time, services, consumables for surveillance). In economic evaluation, the resources used to manage disease are compared with the disease losses with the aim to identify an optimal balance where a higher economic efficiency is achieved. Estimation of the total economic cost stemming from losses and expenditures is called a disease impact assessment. Estimation of the resource expenditures only is called a cost analysis. |
| Benefit | The benefit of surveillance quantifies the monetary and non-monetary positive direct and indirect consequences produced by the surveillance system and assesses whether users are satisfied that their requirements have been met. This includesfinancial savings, better use of resources and any losses avoided due to the existence of the system and the information it provides. These avoided losses may include the avoidance of • Animal production losses • Human mortality and morbidity • Decrease in consumer confidence • Threatened livelihoods • Harmed ecosystems • Utility loss Often, the benefit of surveillance estimated as losses avoided can only be realised by implementing an intervention. Hence, it is necessary to also assess the effect of the intervention and look at surveillance, intervention and loss avoidance as a three-variable relationship. Further benefits of surveillance include maintained or increased trade, improved ability to react in case of an outbreak of disease, maintaining a structured network of professionals able to react appropriately against a (future) threat, maintaining a critical level of infrastructure for disease control, increased understanding about a disease, and improved ability to react in case of an outbreak of disease. |
You could leave a comment if you were logged in.