Session 3: Building Damage and Recovery Analyses with IN-CORE Web Services#

Objective:

  • Learn how to execute the analysis

  • Learn how to do building damamage analysis, Monte Carlo Simulation, and building recovery analysis

  • Learn how to explore outputs of analysis

Agenda

1. Overview#

  • Tornadoes occur at a high frequency in the United States compared with other natural hazards such as earthquakes and tsunamis but have a substantially smaller footprint. Even a single high-intensity tornado can result in high casualty rates and catastrophic economic losses and social consequences, particularly for small to medium communities.

  • The city of Joplin, Missouri, USA, was hit by an EF-5 tornado on May 22, 2011. The National Institute of Standards and Technology (NIST) conducted a technical investigation of this devastating event which can be found at: https://nvlpubs.nist.gov/nistpubs/NCSTAR/NIST.NCSTAR.3.pdf . The Center for Risk-Based Community Resilience Planning simulated this event for buildings and the electrical power network of Joplin in IN-CORE. This Juypter Notebook provides an example of how to use IN-CORE.

  • The initial damage estimation utilized the tornado path, and tornado fragility curves representative of a 19- archetype building dataset. Generic tornado paths are also available in IN-CORE, or a user defined tornado path is possible.

  • This session demonstrates how users interact with the IN-CORE computational environment

2. Explore Hazard, Data, and Fragility#

This section introduces the input for the infrastructure damage analysis including

  • tornado path,

  • building dataset, and

  • building fragility curves for tornado.

2.1. Tornado Path#

A tornado path we will use is the 2011 Joplin tornado path. The tornado path represents the wind speed within the vortex (multi-vortex in the case of Joplin) that was estimated to have EF5 wind speeds of more than 200 mph, reducing to EF4 wind speeds as the areas move outward from the vortex, and eventually reaching EF1 zone. (Attary et al. 2018)

# Import modules
from pyincore import IncoreClient, HazardService
from pyincore import FragilityService, MappingSet, Dataset, DataService
from pyincore_viz.plotutil import PlotUtil as frag_plot
from pyincore_viz.geoutil import GeoUtil as viz
from pyincore.models.fragilitycurveset import FragilityCurveSet
import wsviz

# Connect to IN-CORE serivce by creating IncoreClient
client = IncoreClient()

# The 2011 Joplin Tornado is stored at IN-CORE Hazard service

tornado_id = "5d07cbeab9219c065b080930"
viz.plot_tornado(tornado_id, client, basemap=True)

# show the tornado path on interactive map

wsviz.show_tornado_by_id(client, tornado_id)

2.2. Building Archetypes#

The 19 archetype buildings are used herein to represent the Joplin community. The selected building archetypes consider building characteristics such as footprint area, roof structure, number of stories, and so on. (Attary et al. 2018, Memari et al. 2018)

Joplin building archetypes

Archetype

Building Type

1, 5

Residential wood building

6

Business and retail building

7

Light industrial building

8

Heavy industrial building

9

Elementary/middle school

10

High school

11

Fire/police station

12

Hospital

13

Community center/church

14

Government building

15

Large big-box

16

Small big-box

17

Mobile home

18

Shopping center

19

Office building

  • DataService: once the IncoreClient has been initiated, the DataService module uses the information from that client to administer existing information about known or new dataset. DataService can:

    • Retrieve existing metadata for various dataset

    • Search datasets per specific information

    • Request creation of a new dataset

    • Request deletion of a new dataset

  • Data Viewer: https://incore.ncsa.illinois.edu/DataViewer

    • Let’s browse the Joplin building inventory 5dbc8478b9219c06dd242c0d on the Data Veiwer

# Load the Joplin building dataset.
bldg_dataset_id = "5dbc8478b9219c06dd242c0d"

bldg_dataset = Dataset.from_data_service(bldg_dataset_id, DataService(client))

viz.plot_map(bldg_dataset, column="archetype", category="True")

# Display the buldiing inventory on the interactive map by using Geopandas

bldg_gdf = bldg_dataset.get_dataframe_from_shapefile()

bldg_gdf.explore(
        column="archetype", # column for archetype info
        tooltip="archetype", 
        popup=True,  
        cmap="Set1"  # use "Set1" matplotlib colormap)
)

2.3. Fragility Curves#

Damage analyses use fragility curve sets to calculate the limit state (LS) and damage stsate (DS). Mapping then determines which fragility curve set applying to which inventory. For example, a building that has certain number of stories could be mapped to a specific set of fragility curves.

In IN-CORE, A user can both upload a set of fragility curves and mapping to the DFR3 services, or use local curve and mapping set object DFR3 stands for Damage, Functionality, Repair, Reconstruction, and Recovery.

  • DFR3 Service is the IN-CORE web service to manage curves (equations) of fragility, repair, damage, etc.s.

This section briefly shows how to access fragility curves. A detailed instrcution of how to create your own fragility entities and mapping entities can be found in our Mannual: Toutorial - Create DFR3 Ob

Note that you will learn detail about this matter at Session 4.ject

User can easily browsing and searching Fragility Curves and Mappings via DFR3

  • DFR3 Viewer: https://incore.ncsa.illinois.edu/DFR3Viewer

Let’s browse the building fragility curve (tornado) 5d8942dfb9219c068ea795ea on the DFR3 Veiwer.mvices)

There 19 fragility curves set for these 19 building types in the four damage states, which covers the entire range of wind speeds associated with tornadoes (Attary et al. 2018, Memari et al. 2018). Below we selected 3 types to plot as examples.

# Using FragilityService to show the fragility curve

fragility_service = FragilityService(client)
mapping_id = "5d8942dfb9219c068ea795ea"
mapping_set = MappingSet(fragility_service.get_mapping(mapping_id))

# plot fragility for the first 3 archetypes using pyincore viz method
for mapping in mapping_set.mappings[:3]:
    fragility_id = mapping.entry["Non-Retrofit Fragility ID Code"]
    fragility_set = FragilityCurveSet(fragility_service.get_dfr3_set(fragility_id))
    plt = frag_plot.get_fragility_plot(fragility_set, start=20, end=80)
    plt.show()

3. Building Damage Analysis#

The models in this section implement the estimation of community-level damage to the buildings for the 2011 Joplin tornado.

3.1. Executing Building Damage Analysis#

This analysis computes building damage based on a particular hazard. Currently supported hazards are: earthquake, tsunami, tornado, hurricane and flood.

The process for computing the structural damage is similar to other parts of the built environment.

  1. First, a fragility is obtained based on the hazard type and attributes of the building.

  2. Based on the fragility, the hazard intensity at the location of the building is computed.

  3. Using this information, the probability of exceeding each limit state is computed, along with the probability of damage state.

The outputs of this analysis are CSV file with probabilities of damage and JSON file with information about hazard and fragilities.

You can Import the BuildingDamage Analysis from pyincore as following:

from pyincore.analyses.buildingdamage import BuildingDamage

Input variables to the BuildingDamage Class are:

key name

type

name

result_name *

str

Result name

hazard_type *

str

Hazard type

hazard_id *

str

Hazard id

Input Datasets to BuildingDamage Class are:

key name

type

name

buildings *

ergo:buildingInventoryVer4
ergo:buildingInventoryVer5
ergo:buildingInventoryVer6
ergo:buildingInventoryVer7

Building dataset

dfr3_mapping_set *

incore:dfr3MappingSet

DFR3 Mapping Set

Output Datasets of BuildingDamage class are:

key name

type

parent key

ds_result *

ergo:buildingDamageVer5

buildings

damage_result *

incore:buildingDamageSupplement

buildings

Note: * means it is a required input or an output

Details can be found: https://incore.ncsa.illinois.edu/doc/incore/analyses/building_dmg.html

Lets explore both the Hazard and buildings on an interactive map

# Lets get the tornado Dataset from service

# tornado_id is defined in previous section
tornado_dataset_id = HazardService(client).get_tornado_hazard_metadata(tornado_id)["hazardDatasets"][0]["datasetId"]
tornado_dataset = Dataset.from_data_service(tornado_dataset_id, DataService(client))

# Reusing the merged building damage geodataframe bldg_dmg_gdf to visualize the tornado path and the damage values
viz.get_gdf_wms_map([tornado_dataset], [bldg_dataset])

3.1.1. Setting up and Executing analysis#

The following cell sets up the buiding damage analysis using the variables we defined previous sections:

  • buiding inventory: bldg_dataset

  • fragility mapping: mapping_set

  • hazard_id: tornado_id

# TODO: Please type in the variables and execute this cell

# Import building damage module integrated into pyIncore.
from pyincore.analyses.buildingdamage import BuildingDamage

bldg_dmg = BuildingDamage(client)

# setting input data and parameters
bldg_dmg.set_input_dataset("buildings", bldg_dataset  ) # needs a variable
bldg_dmg.set_input_dataset("dfr3_mapping_set", mapping_set ) # needs a variable
bldg_dmg.set_parameter("hazard_type", "tornado")
bldg_dmg.set_parameter("hazard_id", tornado_id ) # needs a variable
bldg_dmg.set_parameter("num_cpu", 4)  # Define the result name, etc., and choose the number of CPU locally

# setting output info
result_name = "Joplin_bldg_dmg_result"
bldg_dmg.set_parameter("result_name", result_name)

# to run the model parallelly.
bldg_dmg.run_analysis()  # Run the building damage module to get building damage results for Joplin in a .csv file format.

3.1.2. Exploring Results#

# The building damage results herein are referred to fragilities at three damage states (moderate,
# heavy, complete) for 28152 buildings located in Joplin that fall within the tornado path or not.
building_dmg_result = bldg_dmg.get_output_dataset("ds_result")

# Convert dataset to Pandas DataFrame
df_bldg_dmg = building_dmg_result.get_dataframe_from_csv()
df_bldg_dmg.head()

3.1.3. Joining dataset#

Data preparation and data post-processing are common procedures. Prior to using pyIncore, users often encounter situation that they need to reshape their own dataset to make it compliant with the input dataset format of pyIncore.

After acquiring outputs from pyIncore analyses, often time user would need to perform data aggregation to gain statitical insights. The below tutorial gives a few examples on how to join datasets and generate some basic visualizations.

# read building inventory as gdf
bldg_gdf = bldg_dataset.get_dataframe_from_shapefile()

# you can choose columns to be merged
df_bldg_dmg_merged = bldg_gdf[['guid', 'archetype', 'geometry']].merge(df_bldg_dmg, on='guid')
df_bldg_dmg_merged.head()

3.1.4. Show statistical summary on a column#

Try out different columns to see statistical info of other columns

df_bldg_dmg_merged["LS_0"].describe()

3.1.5. Show table group by archetype#

grouped_bldg_dmg = df_bldg_dmg_merged.groupby(by=['archetype'], as_index=True).agg({'DS_0': 'mean', 'DS_1':'mean', 'DS_2': 'mean', 'DS_3': 'mean', 'guid': 'count'})
grouped_bldg_dmg.rename(columns={'guid': 'total_count'}, inplace=True)
grouped_bldg_dmg.head(19)

3.2. Visualizing Building Damage Results#

# Lets look at how many buildings were exposed to the Hazard
count = df_bldg_dmg_merged["haz_expose"].value_counts()
plt = count.plot(kind="bar")

# Plot Damage state by archetype
ax = grouped_bldg_dmg[["DS_0", "DS_1", "DS_2", "DS_3"]].plot.barh(stacked=True)
ax.set_title("Stacked Bar Chart of Damage State Grouped by Archetype Type", fontsize=12)
ax.set_xlabel("complete damage value", fontsize=12)
ax.legend(loc='center left', bbox_to_anchor=(1.0, 0.5)) #here is the magic

Now using Pyincore-viz, Vizualize Damage states

from pyincore_viz.geoutil import GeoUtil as geoviz

In the below cell, modify the string between DS_0 - DS_3 to see buildings in various damage states.

# Plot a map with GeoDataframe
geoviz.plot_gdf_map(df_bldg_dmg_merged, "DS_3", basemap=True)

# Interactive Map

df_bldg_dmg_merged.explore(
    column="DS_3", 
    tooltip="DS_3", 
    popup=True,  
    cmap="bwr"  # use "Set1" matplotlib colormap)
)

Only the residential buildings

residential_archetypes = [1, 5]
mask = df_bldg_dmg_merged['archetype'].isin(residential_archetypes)
df_residential_bldg_dmg = df_bldg_dmg_merged.loc[mask]
geoviz.plot_gdf_map(df_residential_bldg_dmg, "DS_3", basemap=True)

df_residential_bldg_dmg.explore(
    column="DS_3", 
    tooltip="DS_3", 
    popup=True,  
    cmap="bwr"  # use "Set1" matplotlib colormap)
)

Only the commercial buildings

commercial_archetypes = [6, 7, 8, 15, 16, 18, 19]
mask = df_bldg_dmg_merged['archetype'].isin(commercial_archetypes)
df_commercial_bldg_dmg = df_bldg_dmg_merged.loc[mask]
geoviz.plot_gdf_map(df_commercial_bldg_dmg, "DS_3", basemap=True)

df_commercial_bldg_dmg.explore(
    column="DS_3", 
    tooltip="DS_3", 
    popup=True,  
    cmap="bwr"  # use "Set1" matplotlib colormap)
)

## TODO: Please create an interactive map of commercial buildings with column "DS_1" below

4. Monte Carlo Simulation (MCS)#

Researchers can use Monte Carlo Simulation to estimate the probability of each building being in a particular damage state. This example uses 500 iterations to determine the failure probability of buildings reaching damage state 2, damage state 3, and damage state 4. Users can run 10000 samples or even more for a more accurate Monte Carlo Simulation to determine the building failure probabilities. Note that this takes several minutes and we are working on developing a more efficient algorithm.

4.1. Monte Carlo Simulation with Building damages#

This analysis calculates a probability of failure using a stochastic process. Failure probability and Failure state are derived using the dictionary of failed damage states in the input infrastructure dataset. Failure probability is calculated from all stochastic runs, failure state shows all infrastructure standings as a string of failed (0) and not failed (1) states of each individual run.

The output of this analysis are two CSV files;

  1. a failure proability base_name_failure_probability.csv with allocated house units and,

  2. base_name_failure_state.csv.

You can import the MCS class from pyincore as follows:

from pyincore.analyses.montecarlofailureprobability import MonteCarloFailureProbability

The input variables that MCS takes in are:

key name

type

name

result_name *

str

Result name

num_samples *

int

Samples

damage_interval_keys *

list[str]

Damage keys

failure_state_keys *

list[str]

Failure keys

Input Dataset to MCS Class is:

key name

type

name

damage *

ergo:buildingDamageVer4
ergo:buildingDamageVer5
ergo:bridgeDamage
ergo:bridgeDamageVer2
incore:epfDamage
incore:epfDamageVer2
ergo:nsBuildingInventoryDamage
ergo:nsBuildingInventoryDamageVer2
incore:pipelineDamage
incore:pipelineDamageVer2
ergo:roadDamageergo:roadDamageVer2
ergo:waterFacilityDamageVer4
ergo:waterFacilityDamageVer5

Infrastructure damage

Schema of the dataset type can be found leveraging Dataset Type Viewer

Output Datasets of MCS class are:

key name

type

name

failure_probability *

incore:failureProbability

Results

sample_failure_state *

incore:sampleFailureState

Results

Note: * means it is a required input or an output

More details can be found Monte Carlo failure probability analysis

3.2. MCS chaining with Joplin building damage#

The MCS analysis will use the results of building damage analysis in previous section

  • building damage result: building_dmg_result

# TODO: Please type in the variable and exeucte this cell

# Import Monte Carlo failure probability module integrated into pyIncore.
from pyincore.analyses.montecarlofailureprobability import MonteCarloFailureProbability

mc_bldg = MonteCarloFailureProbability(client)

mc_bldg.set_input_dataset("damage",building_dmg_result )  # needs a varialbe
mc_bldg.set_parameter("num_samples", 500)
mc_bldg.set_parameter("damage_interval_keys", ["DS_0", "DS_1", "DS_2", "DS_3"])
mc_bldg.set_parameter("failure_state_keys", ["DS_1", "DS_2", "DS_3"])
mc_bldg.set_parameter("num_cpu", 8)

# name of csv file with results
mc_bldg.set_parameter("result_name", "tornado_mc_failure_probability_buildings")

# Run the Monte Carlo Simulation module to obtain the building failure probabilities. The building failure
# probabilities herein only consider the physical damage without the interdependency.
mc_bldg.run_analysis()

# get buildings probability of non-functionality
building_failure_probability = mc_bldg.get_output_dataset("failure_probability")

df_bldg_fail = building_failure_probability.get_dataframe_from_csv()
df_bldg_fail.head()

# get buildings failure states
building_failure_mcs_samples = mc_bldg.get_output_dataset("sample_failure_state")

bdmcs = building_failure_mcs_samples.get_dataframe_from_csv()
bdmcs.head()

building_damage_states_mcs_output = mc_bldg.get_output_dataset("sample_damage_states")
building_damage_states_mcs_output.get_dataframe_from_csv().head()

3.3. Visualize probability of non-functionality#

# getting geodataframe of building dataset and merge with output
bldg_gdf = bldg_dataset.get_dataframe_from_shapefile()
bldg_fail_gdf = bldg_gdf.merge(df_bldg_fail, on="guid")

viz.plot_gdf_map(bldg_fail_gdf, column="failure_probability")

bldg_fail_gdf.explore(
    column="failure_probability", 
    tooltip="failure_probability", 
    popup=True,  
    cmap="bwr"  # use "Set1" matplotlib colormap)
)

5. Commercial Building Recovery Analysis#

This analysis computes the recovery time needed for each commercial building from any damage states to receive the full restoration. Currently, supported hazards are tornadoes.

The methodology incorporates the multi-layer Monte Carlo simulation approach and determines the two-step recovery time that includes delay and repair. The delay model was modified based on the REDi framework and calculated the end-result outcomes resulted from delay impeding factors such as post-disaster inspection, insurance claim, financing and government permit. The repair model followed the FEMA P-58 approach and was controlled by fragility functions.

The outputs of this analysis is a CSV file with time-stepping recovery probabilities at the building level.

You can import the Commercial Recovery Analysis from pyincore as follows:

from pyincore.analyses.commercialbuildingrecovery.commercialbuildingrecovery import CommercialBuildingRecovery

The input variables for Commercial Recovery Analysis are:

key name

type

name

result_name *

str

Result name

num_samples *

int

Samples number

The input Datasets for Commercial Recovery Analysis are:

key name

type

name

buildings *

ergo:buildingInventoryVer4
ergo:buildingInventoryVer5
ergo:buildingInventoryVer6
ergo:buildingInventoryVer7

Building dataset

dfr3_mapping_set *

incore:dfr3MappingSet

DFR3 Mapping Set

sample_damage_states *

incore:sampleDamageState

Damage states

mcs_failure *

incore:failureProbability

MCS failure

delay_factors *

incore:buildingRecoveryFactors

Delay factors

The output dataset for Commercial Recovery Analysis are:

key name

type

name

time_stepping_recovery *

incore:buildingRecovery

Buildings

recovery *

incore:buildingRecoveryTime

buildings

total_delay *

incore:buildingRecoveryDelay

buildings

Note: * means it is a required input or an output

Details can be found at Commercial Building Recovery

5.1. Repair Curves and Mapping#

In order to perform recovery analysis, we need repair curves and mapping.

  • DFR3 Viewer: https://incore.ncsa.illinois.edu/DFR3Viewer

Let’s browse the building repair curve (tornado) 60edfa3efc0f3a7af53a21b5 on the DFR3 Veiwer.

from pyincore import RepairService

# Repair mapping
repair_mapping_id = "60edfa3efc0f3a7af53a21b5"
# Create repair service
repair_service = RepairService(client)
repair_mapping_set = MappingSet(repair_service.get_mapping(repair_mapping_id))

5.2. Setting up and Executing Analysis#

The Commercial Building Recovery analysis will use the results of MCS analysis in previous section

  • building inventory: bldg_dataset

  • DFR3 mapping: repair_mapping_set

  • MCS result - sample damage state: building_damage_states_mcs_output

  • MCS result - failure probability: building_failure_probability

  • building damage result: building_dmg_result

from pyincore.analyses.commercialbuildingrecovery.commercialbuildingrecovery import CommercialBuildingRecovery

# Create commercial recovery instance
com_recovery = CommercialBuildingRecovery(client)
com_recovery.set_input_dataset("buildings", bldg_dataset)
com_recovery.set_input_dataset("dfr3_mapping_set", repair_mapping_set)

# using the building_damage_mcs_samples output dataset from the MCS analysis
com_recovery.set_input_dataset("sample_damage_states", building_damage_states_mcs_output)

# using the building_failure_probability output dataset from the MCS analysis
com_recovery.set_input_dataset("mcs_failure", building_failure_probability)
delay_factors = "64ee0bcd553ecf0768e21e55"
com_recovery.load_remote_input_dataset("delay_factors", delay_factors)
com_recovery.set_parameter("num_samples", 50)
# using the building damage results from the building damage analysis
com_recovery.set_input_dataset("building_dmg", building_dmg_result)

# Specify the result name
result_name = "joplin_commercial_test"
com_recovery.set_parameter("result_name", result_name)

# Run the analysis
com_recovery.run_analysis()

6.2 Visualizing Results#

# Retrieve result dataset
com_result = com_recovery.get_output_dataset("time_stepping_recovery")

# Convert dataset to Pandas DataFrame
com_df = com_result.get_dataframe_from_csv()

# Display top 5 rows of output data
com_df.head()

# we need to first get archetypes from the building inventory dataset
# and merge with the recovery results to group by archetypes
bldg_gdf = bldg_dataset.get_dataframe_from_shapefile()

# merge/join two dataframe
# you can choose columns to be merged
# we need guid to merge the two dataframes
# we can then drop it for the final dataframe
bldg_com_rec_gdf = bldg_gdf[["geometry","guid", "archetype"]].merge(com_df, on="guid").drop(["guid"], axis=1)

# visualizing the repair status at Quarter 6
bldg_com_rec_gdf.explore(
    column="quarter_6", 
    tooltip="quarter_6", 
    popup=True,  
    cmap="RdBu"  # use "Set1" matplotlib colormap)
)

# visualizing the repair status at Quarter 10

bldg_com_rec_gdf.explore(
    column="quarter_10", 
    tooltip="quarter_10", 
    popup=True,  
    cmap="RdBu"  # use "Set1" matplotlib colormap)
)

# Unique Archetypes
bldg_com_rec_gdf["archetype"].unique()

bldg_mean_com_rec_df = bldg_com_rec_gdf.drop(["geometry"], axis=1).groupby("archetype").agg("mean")
bldg_mean_com_rec_df.head(10)

# plot the recovery curves
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(10, 6))
# we need to transpose the dataframe to change axes
bldg_mean_com_rec_df.T.plot(kind="line", ax=ax)
ax.set_xlabel("Recovery Time (Quarters)")
ax.set_ylabel("Recovery State")  # 0 means damaged completely and 1 means fully recovered
ax.set_title("Joplin Commercial Building Recovery Curves by Archetype")