Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (2024)

Keywords: Social science; Operational forecasting; Communication/decision-making; Decision support; Vulnerability

1. BVOT: An inductive approach to actionable vulnerability decision support

The mission of the U.S. NWS is to “provide weather, water, and climate data, forecasts, warnings, and impact-based decision support services for the protection of life and property, and enhancement of the national economy.” In many cases, this provision is in the form of impact-based decision support services (IDSS), which was codified into law with the Weather Research and Forecasting Innovation Act of 2017 as part of the Weather-Ready Nation initiative (Public Law 115-25). This paper describes the Brief Vulnerability Overview Tool (BVOT), a social science–derived tool being developed that can help the NWS improve IDSS by integrating local vulnerability data into the decision-making and messaging of NWS Weather Forecasting Office (WFO) meteorologists.

For IDSS to be most effective throughout the forecast to warning continuum, NWS meteorologists need insight and understanding of local communities and their vulnerabilities (NWS 2019) (see the “Vulnerability” sidebar for more information). While many vulnerability-based tools already exist, it is clear when considering the specific needs of the NWS that these vulnerability data do not fit well into operational practices. Rather than considering the universe of ways of subdividing, categorizing, and analyzing vulnerability data and then searching for a “use for” some dashboard or database, a different approach would be to consider—inductively—what NWS meteorologists do with vulnerability data, and then consider the structure of decision support needs based on actual practices and mission requirements. Our in situ, ethnographic observations began in 2015 (Friedman 2016, 2017; Friedman and Wagner 2017) and have revealed that real-world NWS WFO vulnerability data needs generally fall into two temporal categories: 1) undefined, long-term, strategic needs and 2) discrete, hazard-specific, short-term tactical needs.

Long-term (strategic) needs for vulnerability data can address mission requirements including establishing outreach and engagement with core partners; determining and implementing established protocols regarding warnings; and/or planning for and implementing focused interventions regarding “structural” factors associated with vulnerability to hazardous weather (e.g., working with coastal communities to understand the extent of storm surge, sea level rise, and/or tsunami exposure so that communities and planners/decision-makers might better prepare). Many of these longer-term, strategic NWS goals fall under the umbrella of the NWS’s Weather-Ready Nation mission and combine both outreach and engagement to assess and address the broad needs of core partners and publics at the national, tribal, regional, state, and local levels. These general outreach and engagement needs can be improved by drawing on many of the existing vulnerability-data-informed decision support tools that have been built by Federal Emergency Management Agency (FEMA) (discussed below)—including the National Risk Index (NRI; Zuzak et al. 2022, 2023) and the Resilience Analysis and Planning Tool (RAPT; FEMA 2023c). Those FEMA tools can meet these outreach and engagement needs because they allow the NWS WFO to identify broad patterns of vulnerability across their county warning area (CWA) that can act as the building blocks for identifying partners for longer-term strategic engagement.

However, while there is an explicit relationship between vulnerability data and many of the long-term, strategic needs of the NWS, the relationship between vulnerability data and short-term, everyday needs (e.g., the strategic vulnerability data one needs to prepare a town for the risk of tornadoes vs the tactical vulnerability data one needs to operationalize that plan with 15-min notice) is less clearly distinguished in the NWS. Short-term (tactical) vulnerability data needs can address NWS mission requirements that focus on the (relatively) immediate threat to life and property posed by discrete, existing weather incidents: a forecasted hurricane, ice storm, blizzard, dust storm, riverine flood, flash flood, tornado, etc. Short-term warnings are, ideally, purely focused on the scientific understanding of atmospheric or hydrological conditions over and above discrete impacts. But this ideal does not do justice to the real-world decision-making of most NWS meteorologists who tacitly account for vulnerabilities throughout their decision-making.

While there have been a limited number of studies that have been conducted to quantitatively assess the potential impact of awareness of vulnerabilities on NWS product issuance (Davis and LaDue 2004; Dobur 2005; Barrett 2008, 2012; White and Stallins 2017; Naylor and Sexton 2018), this has generally remained an understudied topic. Most of these studies have confirmed what most operational meteorologists know—if there is a known concentration of people (i.e., a city or other municipality), it is more likely that the awareness of the assumed vulnerability of humans to severe weather will result in the higher likelihood of warning being issued on “marginal” events. The influence of known vulnerability awareness on warning issuance should hardly be a surprise, despite the fact that there have been few (if any) efforts to systematically study the pathways and nature of this influence on warning issuance. While this remains “anecdotally accepted” as just the way that meteorological warnings happen, this almost everyday occurrence is understudied and underdocumented. This lack of research and/or documentation on the influence of vulnerability awareness on NWS decision-making and product issuance can be seen as a gap in our empirical understanding of how to improve operations and meet core partner needs.

The BVOT was a response to this gap. Specifically, we observed that, on the one hand, NWS meteorologists are aware of the influence of vulnerability knowledge on their product issuance; while, on the other hand, they struggle with justifying and communicating these decisions. This means that there tends to be a very uneven distribution of vulnerability awareness in any given NWS WFO: some meteorologists have “learned” where vulnerabilities exist, others have not. In addition, this knowledge can ebb and flow at any NWS WFO as meteorologists retire or leave for other offices. To add more complication to this institutional knowledge of vulnerabilities, though, emergency managers (EMs) themselves can regularly rotate through jobs, meaning that it becomes increasingly difficult to catalog and share a definitive understanding of weather-hazard-relevant vulnerabilities at any WFO. The BVOT is created by working with local and county EMs who identify vulnerabilities of greatest concern, then mapping, and annotating those spatially discrete vulnerabilities in a shapefile (see below). The BVOT is meant to provide an experimentally grounded, well-tested, shared, and durable understanding of vulnerabilities across an NWS WFO so that the decision to factor in vulnerability information to any IDSS is both defensible and is based on core partners’ forecast needs.

This paper provides an introduction to the BVOT, an experimental (not operational) decision support tool—and an accompanying set of methods—designed to provide hazard-specific, operationally useful vulnerability information to NWS meteorologists in WFOs around the United States. As a tool, the BVOT is, at its most basic level, a shapefile that displays locally known, spatially specific, and weather-hazard-related vulnerabilities in a format that can be easily integrated into the existing operational forecasting, warning responsibilities, and tasks of NWS meteorologists. The method for mapping those vulnerabilities in a WFO’s BVOT builds, strengthens, and maintains the relationships that NWS WFOs already have with their local EMs and core partners through the process of working together to collect, map, and sort those local vulnerability data that will be included in a BVOT. Each CWA has unique sociodemographic and political histories, cultural contexts, economic realities, built and natural environments, and jurisdictional infrastructures that engender what vulnerabilities exist. Therefore, the BVOT is populated with discrete, locally known vulnerability data in order to ensure that NWS meteorologists are spatially situationally aware of those people, places, and things that are of greatest concern to their core partners.

The BVOT improves spatial situational awareness of NWS meteorologists and improves messaging between them and their core partners. The BVOT has been designed to be integrated into any GIS-based software platform, including the NWS’s Advanced Weather Interactive Processing System (AWIPS), used by the NWS meteorologists to view a variety of weather models, aid in the production of official forecasts, and issue weather alerts. Each BVOT shapefile identifies areas of people, places, and things that are most vulnerable to a particular meteorological hazard. Meteorologists can, then, overlay each BVOT layer with meteorological data [e.g., radar, satellite, convection-allowing models (CAMs), and surface observations]. Each BVOT shapefile can also be toggled on or off like any data in AWIPS. Having the BVOT shapefiles separated by weather hazard allows forecasters to focus on only those vulnerabilities susceptible to a particular relevant hazard, helping to eliminate unnecessary information (Fig. 1). BVOT polygons have a built-in text layer that provides specific information about what each vulnerability is and why it is vulnerable (Fig. 2). This text information can be used by the meteorologists to know when and how to relay information to core partners. Central to the BVOT design is the update-able nature of an office’s BVOT. Vulnerabilities may change over time; likewise, knowledge about vulnerabilities can and should develop over time. As such, BVOT can be thought of as a living document for vulnerability knowledge within a WFO that we recommend is updated/reviewed at least once a year.

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (1)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (2)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (3)

Image of the four Birmingham, AL, and Huntsville, AL, BVOT layers. Their hazards of concern are tornadoes, flooding, winter, and wildfires.

Citation: Bulletin of the American Meteorological Society 105, 6; 10.1175/BAMS-D-23-0042.1

  • Download Figure
  • Download figure as PowerPoint slide

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (4)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (5)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (6)

Image from the El Paso, TX (EPZ), WFO BVOT flood hazard layer that provides an example of the text that appears as a user moves their cursor over a BVOT polygon in AWIPS. This text provides the name of the vulnerability, the hazard it is vulnerable to, any vulnerability concerns, and any action or other information that the NWS meteorologist should be aware of. For the above polygon, the green text says “Talavera Neighborhood; Flood; Roads blocked due to poor drainage with heavy rains; consider early FFA.”

Citation: Bulletin of the American Meteorological Society 105, 6; 10.1175/BAMS-D-23-0042.1

  • Download Figure
  • Download figure as PowerPoint slide

The BVOT was created with both NWS meteorologists and a particular core partner, EMs, in mind. The term “emergency manager” is multifaceted in nature, depending on the scale of responsible jurisdiction. EMs work on all levels of government operations ranging from federal, state, local, tribal, or territorial. Their positions include part-time volunteers, working with or under local government, or specialized to a single entity such as a hospital or university. EMs have a number of responsibilities primarily focusing on prevention, protection, mitigation, response, and recovery (FEMA 2020). At the local level, EMs are uniquely positioned to know how weather hazards can—and do—impact rural and urban areas across the United States. While they are responsible, among other things, for identifying risks and mapping vulnerabilities (discussed below), they have no particular existing mechanism for identifying the instances of vulnerabilities that are particularly acute (e.g., the school in a flood plain). However, they are often bearers of significant knowledge and subtle, tacit understandings of who or what is most vulnerable to different hazards. The creation of a BVOT provides EMs with the opportunity to share these concerns to the NWS.

The BVOT is designed to prompt NWS meteorologists to more effectively communicate that specific, identified locations may be at risk from weather hazards. The BVOT may also help to close the “hedge gap” (see the “Closing the hedge gap” sidebar for more information) in situations of uncertainty. While the goal of the BVOT is to display critical, known vulnerabilities, the BVOT is always a subsample of all potential vulnerabilities (i.e., it is not designed to capture all vulnerabilities) and is used to make meteorologists aware of those weather-hazard-specific vulnerabilities that “keep their partners awake at night.”

2. Rationale behind the BVOT

The BVOT concept emerged from the real-time, in situ observation of and interviews with NWS meteorologists and their partner EMs before, during, and after severe weather. As we collaborated across these two groups, the authors increasingly recognized that understanding these actors individually meant understanding how these actors interacted with each other. For instance, NWS meteorologists worried about briefing their core partners at regular times before forecasted severe weather. However, these briefings had, as reported to the authors, increasingly been shaped by meteorologists’ growing understanding of the needs of those core partners, such as the need to make decisions at specific times of the day regarding school closures, opening Emergency Operations Centers (EOCs), mobilizing and preparing resources, and staffing. At the same time, EMs and other core partners had gradually learned, over years of interaction with NWS meteorologists, various limitations faced by meteorologists in their day-to-day work, and those EMs had learned to adapt to what they perceived as the cautious “hedging” (e.g., limiting or qualifying forecast communication, often due to uncertainties) of the NWS (cf. Roeder et al. 2021).

To study the feedbacks between these multiple elements and how they influence the work of NWS meteorologists, we developed an approach that sought to both study and assist these meteorologists to improve their awareness of vulnerabilities across their warning area. What follows, then, is a discussion of 1) some of the existing efforts to turn vulnerability data into decision support tools; 2) what makes BVOT different; 3) what observed impact BVOT has had on NWS meteorologists’ practices and how this has been received and understood by EMs; 4) the co-development of methods that have been developed to implement the BVOT across the NWS; and 5) some of the outcomes, implications, and continuing challenges that emerged from this work.

3. Existing efforts to integrate vulnerability into decision support tools

Many efforts to make sense of vulnerabilities have drawn on pre-existing data—usually at the census-tract level—to identify spatial units that can be tied to various social, economic, demographic, and/or built environment data as proxies for social vulnerability. For instance, vulnerabilities in the Centers for Disease Control’s social vulnerability index (CDC SVI; CDC 2020) can provide a spatially aggregated measure of social vulnerability (based on census-tract geographical units) and can also be displayed based on four major vulnerability themes that aggregate categories of census data in order to shed light on types of vulnerabilities (Table 1).

Table 1.

Census-derived vulnerability data clustered in CDC SVI themes and social factors.

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (7)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (8)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (9)

Something like the CDC’s SVI can be used to get a bird’s eye view of specific census tracts that might be of concern to the NWS and others who are trying to draw on vulnerability data to decide on longer-term strategies. But, in reality, we have seen few meteorologists in the NWS actually use the SVI data in operations. One of the reasons for this is that the SVI, by itself, requires significant social science background and skill in order to interpret these data. A more practical and important reason is that the SVI is hazard-agnostic and focused on the census tract, which is rarely the actionable and operationalizable geographic unit of concern for most NWS meteorologists.

Other recent efforts have focused on bridging these gaps between census-derived vulnerability indicators and the need for actionable units of analysis. Notably, recently, the U.S. FEMA created the NRI in an effort to produce a broadly accessible dashboard that brings together SVI data on both vulnerability and resilience along with hazard exposure and historical loss data for 18 natural hazards. Similar to other efforts (cf., Edgemon et al. 2023), the NRI balances vulnerabilities with a region’s capacity to absorb and/or recover from loss through the inclusion of measures of resilience and provides more granular data on specific natural hazards so that a decision-maker can understand the historical impacts of some hazards (e.g., tornadoes in the southeast). The NRI provides useful decision-support-focused risk assessments for both county and census-tract-level regions. A second, but related, effort by FEMA—the RAPT—focuses more narrowly, but in greater detail, on mapping the intersection of critical vulnerabilities (e.g., schools, hospitals, and mobile home parks) with a range of natural hazards.

While FEMA’s NRI and RAPT are improvements over naive uses of SVI data, we would argue, they still fail to meet the day-to-day operational needs of most NWS meteorologists. This is, of course, not a fault in these FEMA tools or their developers. They, in fact, were never designed for the kind of operational needs of an NWS meteorologist since they were mostly designed to meet requirements for longer-term, strategic decision-making regarding mitigation or recovery planning. Thus, while the NRI or RAPT allows an operational meteorologist to visualize the level of risk posed by certain weather hazards to a census tract or county, this level of awareness, in general, does not translate into short-term, storm-specific, actionable information that can inform communicating risks or hazards to core partners or refining product issuance. What operational meteorologists need to identify to help their core partners fulfill some of their most critical roles is whether objects of concern—people, places, or things—are currently or imminently at risk from a specific weather hazard. Ultimately, by helping NWS operational meteorologists better communicate risks or hazards to core partners through more specific messaging about potential impacts on vulnerabilities, the BVOT aids in the larger NWS mission to provide IDSS.

4. Methods for creating a BVOT

An IDSS approach requires fostering and maintaining relationships between NWS WFOs and their core partners. Creating a BVOT involves both an internal training and mapping process that happens within the WFO, as well as the external mapping that happens between WFOs and their core partners. The methods for building a BVOT should encourage and leverage collaboration between the local WFO and core partners, which may reduce uncertainty (regarding communication and a shared understanding of the vulnerabilities of greatest concern to core partners) and increase trust between partners (see the “Reducing interpersonal uncertainty through collaborative mapping” sidebar for more information). The building of a BVOT, then, can act as a doorway for outreach, helping to improve and maintain relationships between the WFO and core partners. Unlike other existing vulnerability maps—such as SVI, NRI, and RAPT—the BVOT is created both for and by individual NWS WFO. Crucially, the methods for collecting vulnerability data for use in the BVOT rely on eliciting specific and localized knowledge from locals: the NWS meteorologists and their core partners.

The initial BVOT interview protocol for mapping vulnerabilities, Interactive Mapping of Vulnerabilities Exercise (IMoVE; Friedman and Wagner 2018), involved one-on-one sessions with participants (n = 84, ∼60–90 min each). Researchers created the maps later, based on recorded interviews and screen recordings of the informant identifying and describing discrete, weather-related vulnerabilities (Fig. 3). This was a time-intensive process that could not, realistically, be scaled up. The next iteration retained a high level of researcher involvement but was interactive, with participants telling researchers where the vulnerabilities were, and the participant mapping those vulnerabilities. In both initial iterations, the research team combined and created maps using open-source software and created the GIS shapefiles for the participating WFOs to use in operations.

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (10)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (11)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (12)

Versions of BVOT mapping methods. Four versions of BVOT mapping have now been tested, which each requires various levels of researcher involvement and collaboration with the WFO. The term “We” refers to collaboration between both individuals in the WFO and researchers, “Researchers” refers to the research team, and “You” refers to members of the WFO.

Citation: Bulletin of the American Meteorological Society 105, 6; 10.1175/BAMS-D-23-0042.1

  • Download Figure
  • Download figure as PowerPoint slide

Successful implementation of the BVOT across all WFOs will require less technical skill to be feasible. Ongoing research is piloting ways to provide adequate training resources for the BVOT, along with streamlined mapping and configuration processes, so that WFOs can complete all necessary steps entirely independently, without the help of researchers. At present, WFOs spend approximately 30–60 min completing the mapping process individually or in small groups within the WFO, as well as 30–60-min sessions with contributing partners. Following this process, additional time is required to assemble the maps, which has varied across partnering WFOs. Not only does this updated set of methods allow the BVOT to be scaled across the NWS without the need for a team of social scientists to build it with them, but it also provides opportunities to better engage with their EM partners so that the process engenders a sense of ownership and trust in the vulnerability data.

Again, the BVOT is made for and by the WFO. Thus, in the next iteration of the development of methods for collecting vulnerability knowledge and information to populate a BVOT, training modules were created for the WFO to enable researchers to take a supportive, rather than primary role. In this iteration, the research team met with the NWS meteorologists and a subset of partnering EMs within the CWA and coached the partnering WFOs in their efforts to collect, map, and prepare vulnerability data for use in operations. The researchers guided the WFO along the way and worked on ways to make the mapping and preparation of files for operations more user-friendly. As the methods for building a BVOT are further tested, less focus is placed on researcher expertise; instead, the individuals who are doing the mapping are empowered as the experts. In other words, NWS meteorologists and core partners involved in collaboratively mapping BVOT vulnerabilities become stewards of local knowledge.

The continued streamlining of the methods for collecting BVOT data is being tested in WFOs across the country. We have conducted in-person fieldwork, online interviews, and surveys with the participating WFOs to learn about how to improve these methods as well as learn about benefits from building and using the BVOT. Before beginning outreach with offices for participation, all 122 CWAs were compared across socioeconomic variables derived from census data, geography, and climate. We have deliberately selected each study WFO, and the CWA that they cover, due to its unique concerns. They vary in terms of climate, infrastructure, culture, history, geography, and more. Successful BVOT implementation across the country will ultimately require a clear set of methods for mapping, creating, and updating a BVOT that will meet the needs and provide guidance for all CWAs.

During in-person setup and piloting of BVOT methods, the research team engaged in conversations with WFO management regarding ways in which BVOT methods capture institutional knowledge, both externally, on the part of EM knowledge, and internally, regarding WFO knowledge. Many regions experience high EM turnover. Likewise, many WFOs experience turnover among meteorologists. The methods for building a BVOT allow for the capturing of individual local knowledge about the people, places, and things most vulnerable to hazardous weather; that knowledge is not lost when an EM or meteorologist moves. Discussions with participating EMs and meteorologists have helped researchers think about the communication implications resulting from the BVOT methods. EMs have reflected on linear communication transactions: the NWS WFO provides IDSS to the EMs before hazardous weather, and after hazardous weather, EMs send observation information back to the WFO. The methods we have tested to create a local BVOT allow for more transactional flows of communication around potential impacts, and ongoing relationships can be formed between the WFO and their core partners. During this research, warning coordination meteorologists (WCMs) have also noted how the process of working with partners to build a BVOT provides a script for outreach with their partners. It opens the door to increased communication and information exchange (Hurst et al. 2022).

5. Is a BVOT effective?

a. Hazardous weather testbed.

A critical step in the development of the BVOT was to stress test the tool under simulated NWS operational conditions. To do this, the BVOT was tested over 8 weeks in NOAA’s Hazardous Weather Testbed (HWT) from 2021 to 2022. Social and behavioral science methodologies were used to assess how NWS meteorologists and EMs would integrate the BVOT into their operational workflow, including interviews, surveys, observation of decision-making, and talk aloud protocols (during the observation of decision-making). To facilitate realistic experimental testing, this study bridged the first six elements of Forecasting A Continuum of Environmental Threats (FACETs) including 1) hazard information, 2) guidance, 3) meteorologist decisions, 4) forecasting tools, 5) the outputs of these meteorologists, and 6) effective responses (Rothfusz et al. 2018). The ultimate goal of this HWT project was to understand the impact of having increased vulnerability knowledge and awareness at the local CWA level.

In each week-long experiment (Table 2), NWS meteorologists and EMs worked through eight, recorded, real-world weather data cases, to test how the BVOT could be used in a variety of severe weather scenarios. Each case consisted of three temporal periods, Period 1: 24–48 h before a severe weather event; Period 2: 6–12 h before a severe weather event; and Period 3: during active severe weather, to test at what periods during the forecasting/warning process a BVOT might be found most operationally useful. Our meteorologist and EM participants were divided into three teams that corresponded to specific experimental conditions, allowing two out of three participant teams to have access to the BVOT for each case. During each case, meteorologist teams divided operational tasks such as monitoring radar and warning on severe weather hazards as well as providing decision support services to EMs through briefings and messaging.

Table 2.

Key information and parameters for the BVOT Hazardous Weather Testbed experiment.

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (13)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (14)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (15)

The HWT experiment found that the BVOT was most heavily used during Period 3, when active severe weather was occurring. NWS meteorologists delivered most vulnerability messaging using Slack (at the time, a proxy for NWSChat) where they could choose to directly communicate vulnerabilities using specific place names and information from the BVOT that were most at risk to a particular weather hazard. Therefore, meteorologist teams using BVOT were able to enhance their IDSS efforts by communicating hazard-specific vulnerability information, while teams without BVOT had to rely on the standard “cities” layer in AWIPS. Some forecasters who used BVOT during warning operations expressed that they appreciated having the additional situational awareness of what vulnerabilities existed in relation to where the weather hazards were located. However, a few felt that having additional data (BVOT polygons) added too much visually on their AWIPS screen and would prefer to either toggle the tool on and off or to allow their forecast teammate to monitor BVOT areas. Discussions at the end of the week with all forecaster participants suggested that in some cases, the mesoscale analyst or social media (DSS) focal point might be the person responsible for monitoring BVOT areas as the task integrated well into their workflow.

Of particular concern was whether a new tool would be usable during warning operations. The System Usability Scale (SUS) survey (Brooke 1996) data, which measure the effectiveness, efficiency, and satisfaction of a tool or system, show that 70% of HWT meteorologist participants surveyed found BVOT more usable as they became comfortable with the tool (Fig. 4). SUS scores range from 0 to 100. The mean SUS score at the start of the experiment week was 77.2 and 83.2 at the end of the experiment week. To measure the subjective workload of participants as they used the BVOT in each case, we used the NASA Task Load Index (NASA-TLX) which measures the mental demand, physical demand, temporal demand, performance, effort, and frustration of users as they perform tasks (Hart 2006). Mental demand and effort were found to have the greatest magnitude. Cognitive load was initially greater but decreased as meteorologists became more comfortable with using the BVOT and only increased during the most severe test case (Table 3). The confidence continuum survey, adapted from Heinselman et al. (2012), measured meteorologists’ confidence after participating in each experimental case. Overall confidence in making operational decisions increased and was greatest during the most severe test case (Table 3).

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (16)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (17)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (18)

The SUS differences of HWT NWS meteorologist participants comparing the end of the week to the beginning of the week. The usability of BVOT increased for 29 NWS meteorologists after using it during the experiment week.

Citation: Bulletin of the American Meteorological Society 105, 6; 10.1175/BAMS-D-23-0042.1

  • Download Figure
  • Download figure as PowerPoint slide

Table 3.

Key findings from our HWT experiment about the BVOT regarding usability, confidence, and how the tool was used for communication.

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (19)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (20)

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (21)

We generally found BVOT, and the impact of having increased vulnerability awareness during operations, an asset in our simulated NWS operations (Table 3). Teams using BVOT provided EMs with specific, targeted decision support about key vulnerabilities under threat. Further, the BVOT impacted meteorologists’ decisions to upgrade a warning. In a case involving an EF3 tornado, this meant upgrading a “Particularly Dangerous Situation” warning to a tornado emergency: six of the seven teams that issued a tornado emergency had access to the BVOT. During another case, involving an EF3 tornado, 12 out of 13 teams that upgraded a base tornado warning to a considerable tag (Particularly Dangerous Situation warning) had access to the BVOT. Table 3 summarizes some key, preliminary findings; additional analyses continue and will be submitted for peer review in the upcoming months.

During the HWT experiment, we recruited most EM participants from the city and county level to consider how the BVOT can be used to improve communication between NWS WFOs and core partners (NWS 2019) and included a few university, military, state, and health EMs to gain additional perspectives. City and county EMs are responsible for such things as determining the level of emergency operations center staffing, providing guidance to other local officials regarding the local weather threat, and much more. EMs’ practices vary based on the characteristics—and character—of their local jurisdiction, which has unique sociodemographic and political histories, cultural contexts, economic realities, built and natural environments, and jurisdictional infrastructures that drive local policies and capabilities. EMs—or, in some cases, a contracted company—analyze their risks and identify actions to address those risks through a Threat and Hazard Identification and Risk Assessment (THIRA). The results are then detailed in a hazard mitigation plan (HMP), designed to help an EM minimize loss of life and property from disasters (FEMA 2023a,b). Creation of a BVOT differs from this process, in that it contains only the most acute or problematic instances of vulnerabilities, and, crucially, their intersection with one or more particular weather hazards.

We found that EM participants in our HWT project did not only limit themselves to BVOT-provided insights into their area but also searched online to find the types of information a THIRA might reveal. EMs then took a holistic approach to the scenarios, considering both the background state of their assigned area and the BVOT in assessing threats and considering how they would work each case. Our HWT study revealed that EM participants most appreciated an NWS meteorologist’s use of BVOT during the active weather period, Period 3, due to its ability to specify and localize the threat. The BVOT also helped EM participants consider possible higher-end impacts in advance of a potential severe weather event. EMs told us that knowing that a specific vulnerability may be impacted would aid in faster response. In addition, they noted that having specific knowledge of what may have been impacted by a weather hazard in nearby counties would help EMs anticipate potential mutual aid requests.

In one participant’s words, the BVOT helped meteorologists “see more of what’s there” (NWS-wk7-3), when looking at the maps in AWIPS, and it reminded EMs of their greatest concerns. During the storm-on-the-ground period (Period 3), one meteorologist noted that “It was nice to be able to put in a little more value than just that the tornado is moving toward these cities” (NWS-wk7-5) which was echoed by others. Some meteorologists became accustomed to and appreciated the additional information so much that, when they switched to a non-BVOT experimental condition, they missed having it:

I felt negligent. I felt like I was missing this element… it felt like there was an intentional touch to the process that was missing, and it felt weird not seeing those vulnerabilities out there for me to be thinking about and to be messaging to Bob [pseudonym]. To say, you know, “Hey, Bob, the couplet’s going right toward blank.” (NWS-wk3-1)

EM participants recognized this from the other side, with one of them stating, “I liked that I was specifically able to ask the meteorologists where the tornado was in relation to [one of my biggest concerns]…. They were able to…provide me an update that was very spatially and operationally relevant for me” (EM-wk8-1). And, underscoring the value of having relevant reminders/job aids, during a case in an urban area with its many considerations, an EM was caught off-guard at one point: “I may be focused on something else and have that come across the screen and [realize] I need to reach out to the fire department up there…” (EM-wk7-5). Another EM, also caught off-guard, explained, “it’s good for [NWS] to have [this vulnerability] information, because sometimes it’s a memory jogger for us” (EM-wk7-1). BVOT is among a few tools that can help EMs before or after the storms as well. As one EM pointed out, “the BVOT was helpful during Period 2 [when I] focused some messaging about you don’t want to be in those houses by 4-5 o’clock…” (EM-wk7-1). Finally, when storms were ending, an EM shared what can be overwhelming about response and recovery: “After the storm is through, then you’re looking at all that critical infrastructure and also those areas that need attention first if the bad things really did happen in your county” (EM-wk3-2). To that EM, having BVOT data led them to think of a more rapid and targeted response to affected vulnerable regions.

b. Real-world example.

In addition to findings from the HWT experiment, the BVOT has also been used effectively during real-world events. The first test office created their BVOT in 2018 and has successfully used it in operations. In one example, the warning coordinator/mesoscale meteorologist saw a tornadic debris signature passing near a mobile home community identified in their BVOT, prompting the meteorologist to call the EM of that county. As a result, the EM requested the Sheriff to check on the community and to see if assistance was needed. This WFO also used their BVOT during a flooding event when they decided to extend their flash flood warning to include a BVOT point that would have been just outside the initial warning. They used their BVOT outside of operations as well: new meteorologists who transferred in used their BVOT to familiarize themselves with their new CWA and their EMs’ greatest concerns.

6. Conclusions

This paper has described the origin of the BVOT, how we have created and tested methods for putting it into practice, and how we have validated and tested its usefulness and usability. Throughout, our goal has been to demonstrate how social science–informed research can be operationalized within the institutional and organizational context of the NWS and across to its relationships with core partners like EMs. From its inception, collaboration has been at the heart of the BVOT. The methods used for collecting vulnerability data are used to build a local BVOT that can provide actionable decision support for NWS meteorologists. The process for collecting this information requires collaboration across meteorologists within a WFO, capturing institutional knowledge, as well as interorganizational collaboration between the WFO and core partners, such as EMs. In addition to developing and assessing these collaborative methods, this research has also experimentally evaluated the impact of the BVOT on NWS product issuance and communication with EMs, as well as examined how EMs value the “enhanced” messaging that access to a BVOT can afford NWS meteorologists. This work has demonstrated the potential value of moving the BVOT from research to operations.

But, still, the BVOT, as with many social science–derived tools, faces challenges in its transition to operations. Studies have shown that NWS meteorologists are enthusiastic in their support for social science (Sherman-Morris et al. 2018). NOAA, itself, prioritizes matching “understanding and prediction of high-impact weather” with “the urgent need imposed by climate trends, population and infrastructure increase, and disproportionate impacts on vulnerable communities” (NOAA Science Advisory Board 2021). Regardless, social science–derived tools like the BVOT face an uphill battle to find their way into everyday WFO operations. Operational innovations have traditionally involved new technologies, software, or models that, while at times facing resistance from operational meteorologists (e.g., Fine 2007; Daipha 2015), fit the expected paradigm of the meteorological work: highly technical, focused on atmospheric observations, and focused on complex models. The BVOT, as with other innovations, could find itself lost in the “valley of death” (or the “valley of lost opportunities”; National Research Council 2003, 12–21) in the transition from research to operations (Friedman et al. 2023). We conclude, then, with a call for action on the part of the NWS and researchers across the academic, industry, and government spectrum to come together to support and encourage efforts to transition social science–derived, operationally relevant research into meteorological operations.


We define vulnerability as the aspects of a community, system, or asset that make it susceptible to adverse impacts from hazards or threats. Usually, these aspects can be understood as a function of three elements: exposure to the hazard, sensitivity to the hazard, and adaptive capacity to respond to the hazard (Ludwig et al. 2018). Having said this, though, it is important to clarify how “vulnerability” was operationalized within this study. In our research, we have taken an inductive approach to defining vulnerability, allowing our research partners to define it for themselves. This approach has the benefit of avoiding the risk of influencing our research partners into only focusing on specific types of vulnerabilities—social vulnerabilities, infrastructure vulnerabilities, environmental vulnerabilities, etc. We found that if we explicitly asked research subjects to tell us about their “social vulnerabilities,” then they would focus on certain types of categories of vulnerabilities—race, ethnicity, language, etc.—and would ignore other types of categories—infrastructure, ecosystems, etc. On the other hand, if we simply asked meteorologists and emergency managers to tell us about the “people, places, and things that they worry about when faced with hazardous weather,” then respondents tended to mention a wide range of concerns that included all of the types listed above. So, while we provide our own “working definition” of vulnerability here, we stress that this definition was not provided to respondents and that our data reflect our research subjects’ understandings of vulnerability.

Closing the hedge gap

The term hedge gap includes both the recognition that there is a “gap” between the levels of uncertainty that meteorologists and their core partners are willing to accept but, also, acknowledges the fact that these uncertainties lead both sets of actors to “hedge” in their decision-making. Both sets of actors—both NWS meteorologists and their EM core partners—have a “gut feeling” based on their experience about what will probably happen, but both sets of actors hedge and either go forward or hold back until they have reached a point where they are comfortable to take action under conditions of uncertainty.

The BVOT helps NWS meteorologists balance their desire to provide the most accurate weather information possible with an EM’s need for information even if it is couched within significant uncertainties (especially between the watch and warning; cf. Murphy and Epstein 1967). EMs may desire to make longer-term staging decisions, for example, that require more lead time than a typical warning. A BVOT reminds meteorologists of EM decision-making and helps nudge them to provide time-critical information within and outside formal warning processes. EMs need defensible sources of weather information that authorities will recognize as reliable and trustworthy. The NWS is the source of this information for most EMs. EMs are accustomed to imperfect weather information, and this impacts what they do. They make preparatory decisions using forecast information received from their local WFO, their own experiences with severe weather, and environmental cues from the evolving weather they are experiencing (Olson et al. 2023). The imperfection of forecasts engenders an increased reliance on and motivation to strengthen the relationship with the NWS. EMs need the very latest information, and they understand that the forecast can change quickly.

Reducing interpersonal uncertainty through collaborative mapping

The time for building and maintaining trustful relationships is not during or after severe weather events, but rather, relationship maintenance occurs through ongoing communication with core partners on “blue sky days,” as well as providing support on impactful weather days (Hurst et al. 2022). The key to maintaining relationships is the idea that each communication partner knows what to expect from the other. Put differently, on an interpersonal level, knowing who the other person is reduces uncertainty. The lens of uncertainty reduction theory (URT) may help us understand how the process of building a local BVOT helps to reduce this interpersonal uncertainty. According to URT, strangers seek to reduce uncertainty by increasing the predictability of the other person (or organization; Berger and Calabrese 1975). The BVOT acts as a catalyst for communication interactions and information exchange; WCMs and other meteorologists within the WFO have the opportunity to reach out to and learn about their core partners’ needs. Verbal communication, information seeking, perceived similarity, liking, and shared social networks all reduce levels of interpersonal uncertainty (Berger and Calabrese 1975; Berger 1979). By collaboratively building a BVOT with their core partners, WFOs get to know their partners and the partners get to know them; through conversation about the BVOT, they are able to talk about shared concerns related to local vulnerabilities, they exhibit a shared cause in the protection of lives and properties, and they show belonging to a group dedicated to preparedness and response.


Thanks to the NWS offices and emergency managers from the NWS Huntsville and Birmingham CWAs for hosting the two lead authors and sharing your insights with us. Thanks to the 46 NWS meteorologists and 52 emergency managers who participated in our Hazardous Weather Testbed (HWT) project, testing the potential of the BVOT and providing thoughtful insights. Thanks to several undergraduate students who assisted in running the HWT, especially Claire Doyle and Jill Olson. This material is based upon work supported by the VORTEX-SE, HWT, and SBES Programs within the NOAA/OAR Weather Program Office under Awards NA11OAR4320072, NA16OAR4590223, NA17OAR4590203, and NA19OAR4590139.

Data availability statement.

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to confidentiality restrictions (e.g., they contain information that could compromise the privacy of research participants) established through agreement with the University of Oklahoma’s Institutional Review Board to protect human subjects involved in this study.


Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT) (2024)


What are some of the tools meteorologists use to make observations? ›

Besides weather satellites, atmospheric measurements, and computer models, meteorologists use weather radar to monitor and forecast the weather. Weather radar provides important information on where rain or snow is falling.

What is the definition of meteorologist in science? ›

Meteorologists are observers and researchers. They note the physical conditions of the atmosphere above them, and they study maps, satellite data, and radar information. They also compare various kinds of weather data from local, regional, and global sources.

What are the tools for predicting weather? ›

Observational data collected by doppler radar, radiosondes, weather satellites, buoys and other instruments are fed into computerized NWS numerical forecast models. The models use equations, along with new and past weather data, to provide forecast guidance to our meteorologists.

Which tools do meteorologists use to help create weather forecasts in Quizlet? ›

Meteorologists use maps, charts, and computers to analyze weather data and prepare forecasts.

What do you call a female meteorologist? ›

A weather presenter (also known as a weather girl, weatherman or weather broadcaster) is a person who presents the weather forecast daily on radio, television or internet news broadcasts.

How do meteorologists use science? ›

To understand weather, meteorologists study atmospheric patterns. The most familiar atmospheric pattern that drives weather and climate is convection, the process of warmer air rising, and colder, denser air sinking. Convection results in a transfer of energy, heat, and moisture—the basic building blocks of weather.

What is a meteorologist answer? ›

What is a Meteorologist? Meteorology is the science concerned with the Earth's atmosphere and its physical processes. A meteorologist is a physical scientist who observes, studies, or forecasts the weather.

What are the 12 weather instruments and their uses? ›

There are many instruments used to collect weather data, but some of the main ones used are thermometers, barometers, rainfall gauges, wind vanes, anemometers, and hygrometers. They measure temperature, air pressure, rainfall, wind direction, wind speed, and humidity, respectively.

Which of the following are tools used by meteorologists? ›

What's the weather like today?
Weather InstrumentData Collected
BarometerAir Pressure
AnemometerWind Speed and Direction
1 more row

What is a meteorologist tool? ›

You may already be familiar with common weather instruments, like thermometers and wind vanes. But to get the full picture of the weather around us, we need hygrometers, anemometers, barometers, rain gauges and sometimes even lightning detectors.

What tool do meteorologists use to observe clouds? ›

Satellites take pictures of Earth at regular intervals from space, telling us where clouds are located. Meteorologists watch these pictures over time to watch for rapidly growing clouds, a clue to a possible thunderstorm.

Top Articles
Latest Posts
Article information

Author: Jerrold Considine

Last Updated:

Views: 5835

Rating: 4.8 / 5 (58 voted)

Reviews: 81% of readers found this page helpful

Author information

Name: Jerrold Considine

Birthday: 1993-11-03

Address: Suite 447 3463 Marybelle Circles, New Marlin, AL 20765

Phone: +5816749283868

Job: Sales Executive

Hobby: Air sports, Sand art, Electronics, LARPing, Baseball, Book restoration, Puzzles

Introduction: My name is Jerrold Considine, I am a combative, cheerful, encouraging, happy, enthusiastic, funny, kind person who loves writing and wants to share my knowledge and understanding with you.