Communicating Uncertainty in Climate Information for China: Recommendations and Lessons Learned for Climate Services

Uncertainty in climate information arises from a range of sources including deficits in understanding, limitations, and the inherent unpredictability of the climate system (Risbey and O’Kane, 2011; Slingo and Palmer, 2011). At longer timescales, uncertainties about future greenhouse gas emissions scenarios become increasingly important (Dessai and Hulme, 2004). Effective mobilization of scientific information requires recipients to have some awareness of the uncertainty surrounding it and the quality of the underlying scientific process that produced the information (Fischhoff and Davis, 2014). If these are not adequately communicated to those who use this information to inform decision-making, it may result in a false sense of certainty, maladaptive decision-making, and a loss of trust in providers (Macintosh, 2013; LeClerc and Joslyn, 2015). It is therefore important to identify appropriate ways to characterize and communicate uncertainty in climate information. While the question of how to communicate uncertainty in climate information has received an increasing attention in international research (Taylor et al., 2015), there is a dearth of research on how to convey this in a specifically Chinese cultural context. This paper brings together qualitative evidence from a series of interviews with climate scientists, along with current and potential Chinese climate information users to explore: current provision of uncertainty information in climate forecasts and projections, users’ preferences for receiving this information, and the communication challenges faced by providers and users.

Many fields have explored how to best characterize and communicate scientific uncertainty (Spiegelhalter et al., 2011). In the context of climate services, Otto et al. (2016) highlighted the challenge of adequately characterizing uncertainty in climate information, while tailoring it to meet the needs of users. The World Meteorological Organization (WMO) has promoted more responsible forecast provision by moving away from potentially misleading deterministic products (typically using mean model outputs) towards expressing climate anomalies in terms of probabilities (i.e., tercile categories based on the distribution of model outputs; Goddard et al., 2010). Studies suggest that when tailored to a specific task and the recipients’ cognitive processes, non-experts can effectively use probabilistic forecasts in decision-making (Joslyn and LeClerc, 2012, 2016; Savelli and Joslyn, 2013). Indeed, providing information about uncertainty may help to sustain trust in cases where forecasts do not match subsequent events (i.e., false alarms; Joslyn and LeClerc, 2013). Nonetheless, organizational users of climate and weather information services may vary in their tolerance for uncertainty in climate information (Taylor et al., 2015).

While the last decade has seen a proliferation of research on the communication of climate information, relatively few peer reviewed studies have taken place in China, with a majority in North America, Europe, and Australia. This represents a gap in understanding, as the comparatively limited amount of work focused on user needs and climate and weather communication in China suggests that cross-cultural differences exist. For instance, while the ordinal nature of the yellow, amber, red “traffic light” weather warning system is well understood in the UK and US (Lesch et al., 2009; Taylor et al., 2019), this is not the case in China, where red is not intuitively interpreted as representing a higher threat level than amber (Wong and Yan, 2002; Lesch et al., 2009). Moreover, scoping work has suggested that institutional preferences for deterministic information may be particularly strong in China (Nobert et al., 2015, personal communication; Golding et al., 2017a). Given China’s exposure to climate hazards such as heavy rainfall, flooding, tropical cyclones, heatwaves, and drought, addressing this is important for the development of effective climate services (Hewitt and Golding, 2018; Wang et al., 2020).

As part of the Newton Fund’s Climate Science for Services Partnership (CSSP) China program, a project on improving the treatment of uncertainty was undertaken, with a core focus on addressing the following research questions: (1) What information about uncertainty in climate forecasts and projections is currently provided to users? (2) What do climate information providers believe that users need to know about uncertainty? (3) What information about uncertainty would users (and potential users) like to receive? (4) What challenges in the communication of uncertainty exist for the providers and users of climate information?

In this paper, we report findings from a series of interviews with climate experts and (potential) climate information users to address these questions, and outline the key recommendations emerging from this work.

Between November 2017 and April 2018, we conducted 28 expert interviews with climate scientists from China and the UK. Eighteen had expertise in seasonal precipitation forecasting for China (China = 13, UK = 5). Ten (all Chinese scientists) had expertise in multidecadal temperature and precipitation projections relevant to China. As Part 1 of the interviews required a constrained geographic focus, we concentrated on regions where other CSSP China projects were focusing on climate services for seasonal precipitation forecasting (Middle Yangtze) and climate adaptation (Lower Yangtze) (Bett et al., 2017; Golding et al., 2017a, b; Sun et al., 2019).

Experts were identified through the CSSP China programme and a review of the literature. Participants were approached through the UK Met Office, China Meteorological Administration (CMA), Institute of Atmospheric Physics (IAP), the 2018 Forum on Regional Climate Monitoring, and the Forum on Regional Climate Monitoring, Assessment, and Prediction for Asia (FOCRAII). Interviews comprised two stages, and were conducted in English or Mandarin. Part 1 elicited expert judgments about the importance of different sources of predictability and uncertainty in seasonal forecasts or multidecadal projections. Part 2 focused on experts’ perceptions of users’ needs for receiving information about uncertainty including: what they believed that users needed to know about uncertainty, their experience of providing this information, and any challenges that they had encountered or anticipated with respect to communicating uncertainty.

In this paper, we focus on responses to Part 2, with in-depth analyses of Part 1 reported in Grainger et al. (2018, personal communication).

Between March 2018 and July 2018, we interviewed 18 current and potential users of climate information. Participants were initially recruited through contacts in CMA and other CSSP China projects, and asked to recommend other potential contacts who may be willing to take part. As two other projects within the CSSP China program were conducting interviews with some of the same target participants, a joint interview protocol combining questions from each of the projects was developed to limit the risk of stakeholder fatigue (Verdon-Kidd et al., 2012). Participant characteristics are summarized in Table 1, in which 5 of the 18 participants were identified as decision makers, while 11 had intermediary roles as either in-house meteorologists (n = 3) or researchers/analysts (n = 8) who provide information to advise decision makers. The remaining two were academic researchers. Six participants currently received seasonal climate forecasts, while two received multidecadal projections. The remainder (n = 10) did not currently receive climate information, but were interested in doing so (seasonal = 5, multidecadal = 2, general = 4). All organizations operated at either a city, province, or river basin level.

Interviews were conducted in Mandarin or English. Participants were first asked about their organization’s approach to uncertainty. This was followed by questions about information about uncertainty in climate products currently received (current users only), preferences for receiving information about uncertainty in climate products, and any challenges in using or interpreting this information. Those interested in seasonal forecasts were asked to provide feedback on the format and layout of a Chinese translation of a seasonal forecast produced by the Met Office for the Three Gorges Dam (Bett et al., 2017).

Thematic analysis, a procedure for identifying and coding key themes in qualitative data (Guest et al., 2011), was used to analyze the interviews. We applied a mixture of deductive coding, where we examined whether themes suggested by prior research were present in the interviews (i.e., preference for deterministic inform-ation), and inductive coding, where themes emerge from interviews.

Around half of the Chinese experts in seasonal forecasting (6 out of 13) reported direct experience of providing forecasts to sectoral decision makers. They indicated that numeric probabilities were rarely delivered to end-users in formal forecast communications. However, where established relationships between providers and users existed, informal discussions relating to uncertainty (i.e., forecast reliability) do take place. This was consistent with findings from user interviews, where only 1 of 6 seasonal forecast recipients reported receiving probabilistic information. This took the form of qualitative probabilistic statements (i.e., likely, unlikely, etc.). The remaining five reported receiving deterministic forecasts, although one noted that being provided with ranges (i.e., min, max) did capture forecast uncertainty to some extent.

“It’s deterministic. It gives a max/min range and mean in mm but not probabilities.” [Interviewee 2; Energy and Water Sector]

“The range itself is already a symbol of uncertainty.” [Interviewee 15; Water Sector]

While most seasonal forecast users received deterministic forecasts, all were aware that forecasts are inherently uncertain. Despite the potential for “false-alarms” to undermine trust (White and Eiser, 2006; Ripberger et al., 2015; Trainor et al., 2015), seasonal forecast users indicated a generally high level of trust in CMA, despite recognition that forecasts are not always accurate.

“Uncertainty comes with the forecast. The certain forecast itself brings uncertainty.” [Interviewee 1; Energy and Water Sector]

At multidecadal timescales, we found less evidence of demand for longer-term projections among decision makers. Interviews with experts also indicated lower levels of interaction between providers and non-academic users of projections. Scientists reporting experience of providing climate projections to the central government indicated that they had little direct interaction or feedback from policy makers. Both academic users of climate projections reported attending training workshops run by climate scientists. However, they received only raw or processed data, without additional summaries of uncertainty.

“…the group who is working on these GCM models, they will give us like a one-day or two-day training and also we have regular meetings. And they will use examples to tell us what are the uncertainties…or why they cannot change this.” [Interviewee 17; Academic researcher]

There is currently limited formal provision of information about uncertainty in climate products. Seasonal forecasts are usually presented deterministically, although issues related to forecast quality and expert confidence may be conveyed informally. This is consistent with earlier research indicating a predominance of deterministic information in climate provision in China (Nobert et al., 2015, personal communication). However, as has been observed in other contexts (Morss et al., 2008), users understand that uncertainty surrounds determinis-tic forecasts.

While we observed strong connections between seasonal forecast users and CMA providers, there appears to be less interaction between providers and recipients of climate projections. This resonates with findings from work with Chinese water managers, showing that while there is frequent interaction between CMA and users, when it comes to forecasts and observations, this is not yet the case for projections (Khosravi et al., 2021). Indeed, when it comes to long-term planning, historical observations may be preferred to projections (Khosravi et al., 2021). Lack of engagement with projections may also reflect a stronger focus on mitigation than adaptation in China’s climate policy (He, 2013; Engels, 2018).

Experts’ beliefs about users’ needs do not always correspond with actual needs (Bruine de Bruin and Bostrom, 2013). However, it is important to identify what experts believe that users need to know to avoid misleading interpretations, and where expert and user perceptions differ.

All of the experts interviewed agreed that uncertainty should be communicated to users. However, most perceived a tension between providing a detailed account of probability and reliability and information that is readily understandable. Trade-offs between the detail and understandability are recognized in the wider climate communication literature (Stephens et al., 2012), but may be of particular importance in China, where there is currently limited exposure to probabilistic forecasts. Most experts felt that information needed to be tailored to specific users, with Chinese providers noting variability in the level of complexity that different users wanted and had the capacity to understand.

While current provision of seasonal forecasts is largely deterministic, several Chinese experts felt that probabilistic information should be provided. However, some concerns about this may not be accepted or understood. Echoing previous findings (Nobert et al., 2015, personal communication), they perceived a preference for deterministic information to probabilistic information among users. When asked how they thought that probabilities should be presented, one participant suggested that verbal descriptions may be more acceptable to users than numeric probabilities. Others highlighted a need for education about the probabilistic nature of forecasts.

Few scientists believed that users required a full account of sources of predictability in seasonal forecasts. However, some felt that credibility may be bolstered by users believing that experts know these things. Echoing this, an intermediary commenting on the seasonal briefing indicated that while decision makers within their organization were unlikely to consult such a document—relying instead on advice provided “in house”—its “scientific” appearance may instill confidence in its quality. While experts agreed that detailed descriptions of sources of predictability and uncertainty were not needed, many felt that some explanation of why these things affected forecast quality should be provided. The El Niño–Southern Oscillation (ENSO) was identified as the most important source of predictability for seasonal precipitation in the Middle Yangtze, with subjective judgments of forecast quality tending to be conditional on whether it was an El Niño year or not. It was therefore suggested that brief explanations for forecast quality varying from year-to-year could be beneficial. Indeed, one provider expressed concern that the year-to-year variability could harm credibility.

For multidecadal projections, providers had far less interaction with recipients and thus fewer expectations on what information about uncertainty that users required. Some felt that provision information about different areas of uncertainty (i.e., natural variability, model uncertainty, and scenario uncertainty) could be useful. Indeed, other recent studies suggest that some potential users are unaware of the scenario-based nature of projections, conflating them with forecasts (Khosravi et al., 2021). One participant suggested providing confidence categories and “worst case scenario” statements. However, these were acknowledged as speculative suggestions.

Credibility and legitimacy are core components of usable climate knowledge (Lemos et al., 2012). The seasonal forecast providers interviewed emphasized that this is especially true in a Chinese cultural context, where adherence to procedures and hierarchy within the delivery processes may have an importance beyond the forecast information itself. Indeed, one expert expressed that this could be more important than the accuracy of forecasts.

Experts perceived trade-offs between providing detailed explanations of uncertainty and overloading users with information. They agreed that probabilistic forecast information should be communicated, though opinions on how to do this varied. Likewise, explaining why the performance of forecast models varies from year to year was felt to be potentially useful. At multidecadal timescales, the experts interviewed had less interaction with recipients of this information, and thus fewer expectations regarding user needs.

Consistent with prior research (Nobert et al., 2015, personal communication), experts perceived a user preference for deterministic forecasts. However, user interviews revealed a more nuanced picture. While most potential users preferred deterministic forecasts (4 in 5), all the six experienced users wished to receive probabilistic information and some detail about the forecast process.

“Yes, the probability is actually very essential.... I would love to receive a relatively clear statement like there’s an 80% of probability to have such weather.” [Interviewee 13; Energy Sector]

Nonetheless, some noted that while they welcomed probabilistic information, higher level decision makers within their organization required deterministic input.

[On probabilities] “They just ask directly for an accurate number. This is how it works.” [Interviewee 14; Water Sector]

This highlights that in some organizations, decision makers do not consult forecast information directly, relying on interpretations from technical staff who have scientific expertise in areas other than climatology (i.e., hydrology, engineering). Climate service development should therefore take into account that the end-users of climate products may not always be decision makers, but those who advise them. Nonetheless, engaged decision makers do exist, as illustrated by one energy sector decision maker who actively sought explanations for why particular climatic conditions are expected.

“…at the end of January in 2015 and 2016, during that very cold period, I learned from the (online) forum that there was an abnormal weather in the Arctic pushed a cold air toward here. With such an explanation, I began to understand how this cold air happens.” [Interviewee 13; Energy Sector]

While current forecast users wanted to receive probabilistic information, a caveat was that only very high probabilities (> 80%) were felt to be useful for decision making. Users were less inclined to engage with lower probabilities (< 60%) as the forecast may be perceived as lacking credibility.

“If the forecast is highly probable, for example, more than ninety, it’s useful. But if it’s less than ninety or less than eighty or whatever, it’s not.” [Interviewee 15; Water Sector]

“40%–50% probabilities are confusing and seem not so credible.” [Interviewee 1; Energy Sector]

This resonates with international research on seasonal forecast uptake, where lower probabilities may be perceived as too uncertain to support decision-making (Bruno Soares and Dessai, 2016). However, the quotes above were made in response to statements about seaso-nal conditions being above or below historical averages, where probabilities close to 50% may be interpreted as reflecting a lack of knowledge. Hence, the usefulness of forecasts showing lower probabilities of seasonal extremes may be perceived differently. Indeed, a desire for forecasts for seasonal extremes or user-defined thre-sholds was expressed throughout the user interviews. This is consistent with research results in other countries, where tercile-based forecasts have been found to have limited value for decision-making (Haines, 2019). Additionally, some participants indicated a preference for communications linking forecasts to specific decisions and actions; with three expressing a preference for reports where explicit recommendations are provided. Again, this resonates with findings in the broader literature, emphasizing demand for forecasts linked to specific actions (de Perez et al., 2015; Weyrich et al., 2018; Nkiaka et al., 2019).

While detailed scientific information about forecast processes was not desirable for most users, having some explanation was felt to be helpful. Seven participants expressed a preference for concise sentences qualifying the forecast in terms of: types of climate variability considered; justification of high probability for particular conditions; model reliability; and overall forecaster confidence. For probabilistic information, five participants indicated a preference for numerical/graphical information linked to weather variables and expected ranges. A further three also expressed a preference for tailored statements (i.e., spatially appropriate and related to specific decisions or actions). One potential user from the urban transport sector, elaborated on the type of statement that they would like to receive.

“…this March or April, there’s [a high] probability to have high temperature, heavy rain…along our [transport] Line 1 or Line 2 and the intensity of it. If we can just know it in advance, that would be really helpful for us.” [Interviewee 4; Urban Transport Sector]

For multidecadal projections, our sample size made it impossible to identify common themes and preferences with respect to how users would like to receive this information. However, the academic users interviewed indicated a wish for support in integrating projection uncertainties into their research.

We find that while those with lower experience of seasonal forecasts may prefer deterministic forecasts, experienced users wish to receive probabilistic information. However, where forecast probabilities are near to climatology, they may not be perceived as credible. We also observe demand for forecasts linked to user-defined thresholds rather than historical averages, and for decision-relevant advisory statements. While lengthy technical descriptions are unlikely to be directly consulted, short explanations justifying the forecast are welcomed.

While climate experts perceived preferences for deterministic information to be a barrier to the provision of probabilistic seasonal forecasts, we found that this was not unanimously the case, with more experienced users wishing to receive probabilistic information. Recognizing that users vary in their tolerance for uncertainty is important.

Experts identified the management of user expectations as a key challenge. Tensions between what users want and what can reasonably be provided by the available science were underscored in user interviews. For instance, seasonal forecast users often expressed preferences for spatial resolutions that are currently impossible to realize. Similarly, an 80% chance of conditions being above/below average was cited as a threshold for forecasts being used in decision-making, a value that may rarely be reached. While we recommend that providers work toward providing forecasts for user-relevant thre-sholds and extremes, this is contingent on its scientific feasibility.

Both expert and user interviews highlighted challenges related to understanding and interpreting complex scientific information. When asked to provide feedback on the sample seasonal precipitation forecast for the Middle Yangtze, most participants focused their attention on the summary box at the top of the page instead of the more detailed information in the main body. This demonstrates the importance of ensuring that the most salient characteristics of communication contain the most important information, and that this is easily understood (Kloprogge et al., 2007; Spiegelhalter et al., 2011).

At multidecadal timescales, a key challenge for providers was a lack of feedback from non-academic users. The two academic users interviewed did however indicate that while they received training on global climate models, they were currently receiving raw and processed data without means to integrate uncertainties into their own research.

“Here are the raw data…many tiers of data. We copy from them [information providers], and we use the exact same way they presented data and we use it in our research. So, for me, this is a big issue.” [Interviewee 17; Academic]

While preferences for deterministic information can be a challenge for the communication of uncertainty, experts may overestimate this. However, managing user expectations of what science can feasibly provide is of critical importance, especially if tailoring climate infor-mation products to decision relevant thresholds or extremes, where high probabilities of exceedance may be rare. While it is important for climate information products to address decision-making needs, it should be recognized that in some cases end-users may be intermediaries advising decision makers rather than decision makers themselves. While we have less evidence regarding user needs for multidecadal projections, our findings do highlight a current dearth of products for academic users that allow them to integrate uncertainties into their own work.

Based on the findings of this work, we produced a set of recommendations for providers on the treatment of uncertainty in climate information for climate services in China (Grainger et al., 2019, personal communication). In this section, we outline recommendations for the communication of uncertainty in seasonal forecasts, develop the provision of multidecadal projections, and reflect on the challenges and lessons learned in undertaking this work.

Forecasts for the exceedance of user-defined thres-holds may be more useful than forecasts for conditions being above/below average. Indeed, likelihoods within 40%–60% for above/below average conditions were felt to offer little useful information. We recommend that, where scientifically appropriate, providers work with users to identify decision-relevant thresholds and explore whether providing information about the likelihood of exceedance is possible.

Forecast quality is conditional on sources of predictability, such as ENSO, meaning that forecast models may perform better in some years than others. To avoid confusion and loss of trust that may result from this variability in forecast quality, we recommend that forecasters explain that forecast performance (i.e., skill) is conditional on these sources of predictability (i.e., precipitation forecasts for the Middle Yangtze being more reliable in El Niño years).

Non-specialist users may not want to receive detailed technical information. However, having an indication of forecasters’ confidence regarding forecast quality is valued by users. Some users already receive this through discussions with CMA providers.

Where seasonal forecasts do not perform better than historical data (climatology), they cannot provide added value, and may be misleading. However, Chinese forecast providers indicate that they cannot fail to provide a forecast when requested by users. Where forecasting models lack skill, we recommend that forecasts be provided based on climatology, with it being explained that the decision to base the forecast on models or observations is made on the best available science.

Our user interviews did not reveal a unanimous consensus as to the precise format in which information regarding probability and forecast quality is presented (i.e., graphs, numeric probabilities). However, feedback on the precipitation forecast for the Middle Yangtze highlighted the importance of ensuring that the most critical information is the most salient. When presented with the forecast, most participants focused predominantly on the summary box at the top of the page, with limited attention to the more detailed text and diagrams below. Where a briefing style is used, the summary at the start should contain the most decision relevant elements of the forecast. This could include the likelihood of decision relevant threshold exceedance, spatial resolution of forecast, and any user-specific advisory statements provided as part of the forecast.

Throughout this project, we observed low engagement with climate projections, and limited interaction between providers and users. To develop the provision of multidecadal climate services, it may be necessary to actively explore the potential for climate projections to inform long-term planning decisions with users. For instance, through examining the benefits of using climate change projections alongside historical observations.

While our sample sizes fall within the recommended ranges for qualitative interviews (Sim et al., 2018), sampling was constrained by the need for introductions to be made through the established contacts. While this work enabled us to identify a coherent set of themes, challenges, and recommendations, we must acknowledge that the user sample was comparatively small and concentrated in state-run water and energy organizations. Other sectors and private organizations were underrepresented. Despite an initial intention to focus on decision makers as end-users of climate information, we found that within some organizations, decision makers do not directly consult climate information, relying on others to interpret this information and provide recommendations. Nonetheless, in tailoring climate products, it is important to understand the choices that they will inform, even if they will not be directly consulted by decision makers.

A key challenge in this research was the limited uptake and interest in multidecadal projections among (potential) users. However, a current lack of engagement with multidecadal projections does not indicate that these timescales are irrelevant to decision-making. Historical observations may be used in preference to projections for long-term planning (Khosravi et al., 2021). Bringing together providers and potential users to raise awareness of the added value that projections may provide may be crucial to promote uptake—an approach taken by Sun et al. (2019), who brought together providers, decision makers, and researchers to explore climate services for urban sector adaptation.

Another notable challenge is related to the fact that there were not always direct Chinese analogues for English-language terminology related to uncertainty in climate information (i.e., probability, accuracy, reliability, and skill having distinct meanings; American Meteorological Society, 2020). The lack of a common vocabulary for discussing different aspects of uncertainty with experts did pose a challenge, suggesting the need for appropriate terminologies to be identified for cross-cultural collaborations.

This work was undertaken to examine the current provision of information about uncertainty in climate forecasts and projections for China, assess users’ preferences and experts’ perceptions of user needs, and explore the challenges associated with communicating uncertainty. Key recommendations are summarized in Table 2.

At seasonal timescales, we find that current provision is mainly deterministic. However, while experts perceive a preference for deterministic information among users, this is not universally the case, with experienced users wishing to receive probabilistic forecasts. Nonetheless, when it comes to forecasts presented as the likelihood of above/below average, only high probabilities (> 60%–80%) are perceived as useful, with probabilities around 50% perceived as not conveying useful information. As anomaly-based forecasts for above/below average conditions can be challenging to integrate into decision making processes, we recommend that developers of seaso-nal climate services for China explore the feasibility of providing probabilistic forecasts based on user-defined thresholds. Our exploration of preferences for receiving information about uncertainty did not identify a “most preferred” format. However, it did highlight the importance of having “summary boxes” that contain all decision critical information. While detailed technical information may be of limited interest, many users did welcome to have some explanation and justification for the forecast. Indeed, as forecast performance depends on sources of predictability, we suggest that short statements regarding forecast quality be provided. For instance, when model skill is low, providing forecasts based on historical averages (climatology) and explaining that this represents the best available science, may offer a credible way to address the conditional nature of forecast quality. At multidecadal timescales, limited user engagement made it impossible to provide evidence-based recommendations for communication. However, our findings suggest that the development of climate services at multidecadal timescales will require exploration of the added value that projections may provide for long-term planning.

Acknowledgments. We thank the CSSP China Work Package 5 leaders Drs. Chris Hewitt, Nicola Golding, Tyrone Dunbar, and Peiqun Zhang, for their input, feedback, and support in this work; as well as all of those who facilitated this research in China, including Prof. Liangmin Du, Drs. Min Lin, Qingchen Chao, Ying Sun, and Mr. Yuan Sun. We thank all of our participants in China and the UK for contributing their time and insights.