Session 3: Building Damage and Recovery Analyses with IN-CORE Web Services#
Objective:
Learn how to execute the analysis
Learn how to do building damamage analysis, Monte Carlo Simulation, and building recovery analysis
Learn how to explore outputs of analysis
Agenda
1. Overview
2. Explore Hazard, Data, and Fragility
3. Building Damage Analysis
4. Monte Carlo Simulation (MCS)
5. Commercial Building Recovery Analysis
1. Overview#
Tornadoes occur at a high frequency in the United States compared with other natural hazards such as earthquakes and tsunamis but have a substantially smaller footprint. Even a single high-intensity tornado can result in high casualty rates and catastrophic economic losses and social consequences, particularly for small to medium communities.
The city of Joplin, Missouri, USA, was hit by an EF-5 tornado on May 22, 2011. The National Institute of Standards and Technology (NIST) conducted a technical investigation of this devastating event which can be found at: https://nvlpubs.nist.gov/nistpubs/NCSTAR/NIST.NCSTAR.3.pdf . The Center for Risk-Based Community Resilience Planning simulated this event for buildings and the electrical power network of Joplin in IN-CORE. This Juypter Notebook provides an example of how to use IN-CORE.
The initial damage estimation utilized the tornado path, and tornado fragility curves representative of a 19- archetype building dataset. Generic tornado paths are also available in IN-CORE, or a user defined tornado path is possible.
This session demonstrates how users interact with the IN-CORE computational environment
2. Explore Hazard, Data, and Fragility#
This section introduces the input for the infrastructure damage analysis including
tornado path,
building dataset, and
building fragility curves for tornado.
2.1. Tornado Path#
A tornado path we will use is the 2011 Joplin tornado path. The tornado path represents the wind speed within the vortex (multi-vortex in the case of Joplin) that was estimated to have EF5 wind speeds of more than 200 mph, reducing to EF4 wind speeds as the areas move outward from the vortex, and eventually reaching EF1 zone. (Attary et al. 2018)
# Import modules
from pyincore import IncoreClient, HazardService
from pyincore import FragilityService, MappingSet, Dataset, DataService
from pyincore_viz.plotutil import PlotUtil as frag_plot
from pyincore_viz.geoutil import GeoUtil as viz
from pyincore.models.fragilitycurveset import FragilityCurveSet
import wsviz
# Connect to IN-CORE serivce by creating IncoreClient
client = IncoreClient()
# The 2011 Joplin Tornado is stored at IN-CORE Hazard service
tornado_id = "5d07cbeab9219c065b080930"
viz.plot_tornado(tornado_id, client, basemap=True)
# show the tornado path on interactive map
wsviz.show_tornado_by_id(client, tornado_id)
2.2. Building Archetypes#
The 19 archetype buildings are used herein to represent the Joplin community. The selected building archetypes consider building characteristics such as footprint area, roof structure, number of stories, and so on. (Attary et al. 2018, Memari et al. 2018)
Joplin building archetypes
Archetype |
Building Type |
---|---|
1, 5 |
Residential wood building |
6 |
Business and retail building |
7 |
Light industrial building |
8 |
Heavy industrial building |
9 |
Elementary/middle school |
10 |
High school |
11 |
Fire/police station |
12 |
Hospital |
13 |
Community center/church |
14 |
Government building |
15 |
Large big-box |
16 |
Small big-box |
17 |
Mobile home |
18 |
Shopping center |
19 |
Office building |
DataService: once the
IncoreClient
has been initiated, theDataService
module uses the information from that client to administer existing information about known or new dataset.DataService
can:Retrieve existing metadata for various dataset
Search datasets per specific information
Request creation of a new dataset
Request deletion of a new dataset
Data Viewer: https://incore.ncsa.illinois.edu/DataViewer
Let’s browse the Joplin building inventory
5dbc8478b9219c06dd242c0d
on the Data Veiwer
# Load the Joplin building dataset.
bldg_dataset_id = "5dbc8478b9219c06dd242c0d"
bldg_dataset = Dataset.from_data_service(bldg_dataset_id, DataService(client))
viz.plot_map(bldg_dataset, column="archetype", category="True")
# Display the buldiing inventory on the interactive map by using Geopandas
bldg_gdf = bldg_dataset.get_dataframe_from_shapefile()
bldg_gdf.explore(
column="archetype", # column for archetype info
tooltip="archetype",
popup=True,
cmap="Set1" # use "Set1" matplotlib colormap)
)
2.3. Fragility Curves#
Damage analyses use fragility curve sets to calculate the limit state (LS) and damage stsate (DS). Mapping then determines which fragility curve set applying to which inventory. For example, a building that has certain number of stories could be mapped to a specific set of fragility curves.
In IN-CORE, A user can both upload a set of fragility curves and mapping to the DFR3 services, or use local curve and mapping set object DFR3 stands for Damage, Functionality, Repair, Reconstruction, and Recovery.
DFR3 Service is the IN-CORE web service to manage curves (equations) of fragility, repair, damage, etc.s.
This section briefly shows how to access fragility curves. A detailed instrcution of how to create your own fragility entities and mapping entities can be found in our Mannual: Toutorial - Create DFR3 Ob
Note that you will learn detail about this matter at Session 4.ject
User can easily browsing and searching Fragility Curves and Mappings via DFR3
DFR3 Viewer: https://incore.ncsa.illinois.edu/DFR3Viewer
Let’s browse the building fragility curve (tornado) 5d8942dfb9219c068ea795ea
on the DFR3 Veiwer.mvices)
There 19 fragility curves set for these 19 building types in the four damage states, which covers the entire range of wind speeds associated with tornadoes (Attary et al. 2018, Memari et al. 2018). Below we selected 3 types to plot as examples.
# Using FragilityService to show the fragility curve
fragility_service = FragilityService(client)
mapping_id = "5d8942dfb9219c068ea795ea"
mapping_set = MappingSet(fragility_service.get_mapping(mapping_id))
# plot fragility for the first 3 archetypes using pyincore viz method
for mapping in mapping_set.mappings[:3]:
fragility_id = mapping.entry["Non-Retrofit Fragility ID Code"]
fragility_set = FragilityCurveSet(fragility_service.get_dfr3_set(fragility_id))
plt = frag_plot.get_fragility_plot(fragility_set, start=20, end=80)
plt.show()
3. Building Damage Analysis#
The models in this section implement the estimation of community-level damage to the buildings for the 2011 Joplin tornado.
3.1. Executing Building Damage Analysis#
This analysis computes building damage based on a particular hazard. Currently supported hazards are: earthquake, tsunami, tornado, hurricane and flood.
The process for computing the structural damage is similar to other parts of the built environment.
First, a fragility is obtained based on the hazard type and attributes of the building.
Based on the fragility, the hazard intensity at the location of the building is computed.
Using this information, the probability of exceeding each limit state is computed, along with the probability of damage state.
The outputs of this analysis are CSV file with probabilities of damage and JSON file with information about hazard and fragilities.
You can Import the BuildingDamage Analysis from pyincore as following:
from pyincore.analyses.buildingdamage import BuildingDamage
Input variables to the BuildingDamage Class are:
key name |
type |
name |
---|---|---|
result_name * |
str |
Result name |
hazard_type * |
str |
Hazard type |
hazard_id * |
str |
Hazard id |
Input Datasets to BuildingDamage Class are:
key name |
type |
name |
---|---|---|
buildings * |
ergo:buildingInventoryVer4 |
Building dataset |
dfr3_mapping_set * |
incore:dfr3MappingSet |
DFR3 Mapping Set |
Output Datasets of BuildingDamage class are:
key name |
type |
parent key |
---|---|---|
ds_result * |
ergo:buildingDamageVer5 |
buildings |
damage_result * |
incore:buildingDamageSupplement |
buildings |
Note: * means it is a required input or an output
Details can be found: https://incore.ncsa.illinois.edu/doc/incore/analyses/building_dmg.html
Lets explore both the Hazard and buildings on an interactive map
# Lets get the tornado Dataset from service
# tornado_id is defined in previous section
tornado_dataset_id = HazardService(client).get_tornado_hazard_metadata(tornado_id)["hazardDatasets"][0]["datasetId"]
tornado_dataset = Dataset.from_data_service(tornado_dataset_id, DataService(client))
# Reusing the merged building damage geodataframe bldg_dmg_gdf to visualize the tornado path and the damage values
viz.get_gdf_wms_map([tornado_dataset], [bldg_dataset])
3.1.1. Setting up and Executing analysis#
The following cell sets up the buiding damage analysis using the variables we defined previous sections:
buiding inventory: bldg_dataset
fragility mapping: mapping_set
hazard_id: tornado_id
# TODO: Please type in the variables and execute this cell
# Import building damage module integrated into pyIncore.
from pyincore.analyses.buildingdamage import BuildingDamage
bldg_dmg = BuildingDamage(client)
# setting input data and parameters
bldg_dmg.set_input_dataset("buildings", bldg_dataset ) # needs a variable
bldg_dmg.set_input_dataset("dfr3_mapping_set", mapping_set ) # needs a variable
bldg_dmg.set_parameter("hazard_type", "tornado")
bldg_dmg.set_parameter("hazard_id", tornado_id ) # needs a variable
bldg_dmg.set_parameter("num_cpu", 4) # Define the result name, etc., and choose the number of CPU locally
# setting output info
result_name = "Joplin_bldg_dmg_result"
bldg_dmg.set_parameter("result_name", result_name)
# to run the model parallelly.
bldg_dmg.run_analysis() # Run the building damage module to get building damage results for Joplin in a .csv file format.
3.1.2. Exploring Results#
# The building damage results herein are referred to fragilities at three damage states (moderate,
# heavy, complete) for 28152 buildings located in Joplin that fall within the tornado path or not.
building_dmg_result = bldg_dmg.get_output_dataset("ds_result")
# Convert dataset to Pandas DataFrame
df_bldg_dmg = building_dmg_result.get_dataframe_from_csv()
df_bldg_dmg.head()
3.1.3. Joining dataset#
Data preparation and data post-processing are common procedures. Prior to using pyIncore, users often encounter situation that they need to reshape their own dataset to make it compliant with the input dataset format of pyIncore.
After acquiring outputs from pyIncore analyses, often time user would need to perform data aggregation to gain statitical insights. The below tutorial gives a few examples on how to join datasets and generate some basic visualizations.
# read building inventory as gdf
bldg_gdf = bldg_dataset.get_dataframe_from_shapefile()
# you can choose columns to be merged
df_bldg_dmg_merged = bldg_gdf[['guid', 'archetype', 'geometry']].merge(df_bldg_dmg, on='guid')
df_bldg_dmg_merged.head()
3.1.4. Show statistical summary on a column#
Try out different columns to see statistical info of other columns
df_bldg_dmg_merged["LS_0"].describe()
3.1.5. Show table group by archetype#
grouped_bldg_dmg = df_bldg_dmg_merged.groupby(by=['archetype'], as_index=True).agg({'DS_0': 'mean', 'DS_1':'mean', 'DS_2': 'mean', 'DS_3': 'mean', 'guid': 'count'})
grouped_bldg_dmg.rename(columns={'guid': 'total_count'}, inplace=True)
grouped_bldg_dmg.head(19)
3.2. Visualizing Building Damage Results#
# Lets look at how many buildings were exposed to the Hazard
count = df_bldg_dmg_merged["haz_expose"].value_counts()
plt = count.plot(kind="bar")
# Plot Damage state by archetype
ax = grouped_bldg_dmg[["DS_0", "DS_1", "DS_2", "DS_3"]].plot.barh(stacked=True)
ax.set_title("Stacked Bar Chart of Damage State Grouped by Archetype Type", fontsize=12)
ax.set_xlabel("complete damage value", fontsize=12)
ax.legend(loc='center left', bbox_to_anchor=(1.0, 0.5)) #here is the magic
Now using Pyincore-viz, Vizualize Damage states
from pyincore_viz.geoutil import GeoUtil as geoviz
In the below cell, modify the string between DS_0 - DS_3 to see buildings in various damage states.
# Plot a map with GeoDataframe
geoviz.plot_gdf_map(df_bldg_dmg_merged, "DS_3", basemap=True)
# Interactive Map
df_bldg_dmg_merged.explore(
column="DS_3",
tooltip="DS_3",
popup=True,
cmap="bwr" # use "Set1" matplotlib colormap)
)
Only the residential buildings
residential_archetypes = [1, 5]
mask = df_bldg_dmg_merged['archetype'].isin(residential_archetypes)
df_residential_bldg_dmg = df_bldg_dmg_merged.loc[mask]
geoviz.plot_gdf_map(df_residential_bldg_dmg, "DS_3", basemap=True)
df_residential_bldg_dmg.explore(
column="DS_3",
tooltip="DS_3",
popup=True,
cmap="bwr" # use "Set1" matplotlib colormap)
)
Only the commercial buildings
commercial_archetypes = [6, 7, 8, 15, 16, 18, 19]
mask = df_bldg_dmg_merged['archetype'].isin(commercial_archetypes)
df_commercial_bldg_dmg = df_bldg_dmg_merged.loc[mask]
geoviz.plot_gdf_map(df_commercial_bldg_dmg, "DS_3", basemap=True)
df_commercial_bldg_dmg.explore(
column="DS_3",
tooltip="DS_3",
popup=True,
cmap="bwr" # use "Set1" matplotlib colormap)
)
## TODO: Please create an interactive map of commercial buildings with column "DS_1" below
4. Monte Carlo Simulation (MCS)#
Researchers can use Monte Carlo Simulation to estimate the probability of each building being in a particular damage state. This example uses 500 iterations to determine the failure probability of buildings reaching damage state 2, damage state 3, and damage state 4. Users can run 10000 samples or even more for a more accurate Monte Carlo Simulation to determine the building failure probabilities. Note that this takes several minutes and we are working on developing a more efficient algorithm.
4.1. Monte Carlo Simulation with Building damages#
This analysis calculates a probability of failure using a stochastic process. Failure probability and Failure state are derived using the dictionary of failed damage states in the input infrastructure dataset. Failure probability is calculated from all stochastic runs, failure state shows all infrastructure standings as a string of failed (0) and not failed (1) states of each individual run.
The output of this analysis are two CSV files;
a failure proability base_name_failure_probability.csv with allocated house units and,
base_name_failure_state.csv.
You can import the MCS class from pyincore as follows:
from pyincore.analyses.montecarlofailureprobability import MonteCarloFailureProbability
The input variables that MCS takes in are:
key name |
type |
name |
---|---|---|
result_name * |
str |
Result name |
num_samples * |
int |
Samples |
damage_interval_keys * |
list[str] |
Damage keys |
failure_state_keys * |
list[str] |
Failure keys |
Input Dataset to MCS Class is:
key name |
type |
name |
---|---|---|
damage * |
ergo:buildingDamageVer4 |
Infrastructure damage |
Schema of the dataset type can be found leveraging Dataset Type Viewer
Output Datasets of MCS class are:
key name |
type |
name |
---|---|---|
failure_probability * |
incore:failureProbability |
Results |
sample_failure_state * |
incore:sampleFailureState |
Results |
Note: * means it is a required input or an output
More details can be found Monte Carlo failure probability analysis
3.2. MCS chaining with Joplin building damage#
The MCS analysis will use the results of building damage analysis in previous section
building damage result: building_dmg_result
# TODO: Please type in the variable and exeucte this cell
# Import Monte Carlo failure probability module integrated into pyIncore.
from pyincore.analyses.montecarlofailureprobability import MonteCarloFailureProbability
mc_bldg = MonteCarloFailureProbability(client)
mc_bldg.set_input_dataset("damage",building_dmg_result ) # needs a varialbe
mc_bldg.set_parameter("num_samples", 500)
mc_bldg.set_parameter("damage_interval_keys", ["DS_0", "DS_1", "DS_2", "DS_3"])
mc_bldg.set_parameter("failure_state_keys", ["DS_1", "DS_2", "DS_3"])
mc_bldg.set_parameter("num_cpu", 8)
# name of csv file with results
mc_bldg.set_parameter("result_name", "tornado_mc_failure_probability_buildings")
# Run the Monte Carlo Simulation module to obtain the building failure probabilities. The building failure
# probabilities herein only consider the physical damage without the interdependency.
mc_bldg.run_analysis()
# get buildings probability of non-functionality
building_failure_probability = mc_bldg.get_output_dataset("failure_probability")
df_bldg_fail = building_failure_probability.get_dataframe_from_csv()
df_bldg_fail.head()
# get buildings failure states
building_failure_mcs_samples = mc_bldg.get_output_dataset("sample_failure_state")
bdmcs = building_failure_mcs_samples.get_dataframe_from_csv()
bdmcs.head()
building_damage_states_mcs_output = mc_bldg.get_output_dataset("sample_damage_states")
building_damage_states_mcs_output.get_dataframe_from_csv().head()
3.3. Visualize probability of non-functionality#
# getting geodataframe of building dataset and merge with output
bldg_gdf = bldg_dataset.get_dataframe_from_shapefile()
bldg_fail_gdf = bldg_gdf.merge(df_bldg_fail, on="guid")
viz.plot_gdf_map(bldg_fail_gdf, column="failure_probability")
bldg_fail_gdf.explore(
column="failure_probability",
tooltip="failure_probability",
popup=True,
cmap="bwr" # use "Set1" matplotlib colormap)
)
5. Commercial Building Recovery Analysis#
This analysis computes the recovery time needed for each commercial building from any damage states to receive the full restoration. Currently, supported hazards are tornadoes.
The methodology incorporates the multi-layer Monte Carlo simulation approach and determines the two-step recovery time that includes delay and repair. The delay model was modified based on the REDi framework and calculated the end-result outcomes resulted from delay impeding factors such as post-disaster inspection, insurance claim, financing and government permit. The repair model followed the FEMA P-58 approach and was controlled by fragility functions.
The outputs of this analysis is a CSV file with time-stepping recovery probabilities at the building level.
You can import the Commercial Recovery Analysis from pyincore as follows:
from pyincore.analyses.commercialbuildingrecovery.commercialbuildingrecovery import CommercialBuildingRecovery
The input variables for Commercial Recovery Analysis are:
key name |
type |
name |
---|---|---|
result_name * |
str |
Result name |
num_samples * |
int |
Samples number |
The input Datasets for Commercial Recovery Analysis are:
key name |
type |
name |
---|---|---|
buildings * |
ergo:buildingInventoryVer4 |
Building dataset |
dfr3_mapping_set * |
incore:dfr3MappingSet |
DFR3 Mapping Set |
sample_damage_states * |
incore:sampleDamageState |
Damage states |
mcs_failure * |
incore:failureProbability |
MCS failure |
delay_factors * |
incore:buildingRecoveryFactors |
Delay factors |
The output dataset for Commercial Recovery Analysis are:
key name |
type |
name |
---|---|---|
time_stepping_recovery * |
incore:buildingRecovery |
Buildings |
recovery * |
incore:buildingRecoveryTime |
buildings |
total_delay * |
incore:buildingRecoveryDelay |
buildings |
Note: * means it is a required input or an output
Details can be found at Commercial Building Recovery
5.1. Repair Curves and Mapping#
In order to perform recovery analysis, we need repair curves and mapping.
DFR3 Viewer: https://incore.ncsa.illinois.edu/DFR3Viewer
Let’s browse the building repair curve (tornado) 60edfa3efc0f3a7af53a21b5
on the DFR3 Veiwer.
from pyincore import RepairService
# Repair mapping
repair_mapping_id = "60edfa3efc0f3a7af53a21b5"
# Create repair service
repair_service = RepairService(client)
repair_mapping_set = MappingSet(repair_service.get_mapping(repair_mapping_id))
5.2. Setting up and Executing Analysis#
The Commercial Building Recovery analysis will use the results of MCS analysis in previous section
building inventory: bldg_dataset
DFR3 mapping: repair_mapping_set
MCS result - sample damage state: building_damage_states_mcs_output
MCS result - failure probability: building_failure_probability
building damage result: building_dmg_result
from pyincore.analyses.commercialbuildingrecovery.commercialbuildingrecovery import CommercialBuildingRecovery
# Create commercial recovery instance
com_recovery = CommercialBuildingRecovery(client)
com_recovery.set_input_dataset("buildings", bldg_dataset)
com_recovery.set_input_dataset("dfr3_mapping_set", repair_mapping_set)
# using the building_damage_mcs_samples output dataset from the MCS analysis
com_recovery.set_input_dataset("sample_damage_states", building_damage_states_mcs_output)
# using the building_failure_probability output dataset from the MCS analysis
com_recovery.set_input_dataset("mcs_failure", building_failure_probability)
delay_factors = "64ee0bcd553ecf0768e21e55"
com_recovery.load_remote_input_dataset("delay_factors", delay_factors)
com_recovery.set_parameter("num_samples", 50)
# using the building damage results from the building damage analysis
com_recovery.set_input_dataset("building_dmg", building_dmg_result)
# Specify the result name
result_name = "joplin_commercial_test"
com_recovery.set_parameter("result_name", result_name)
# Run the analysis
com_recovery.run_analysis()
6.2 Visualizing Results#
# Retrieve result dataset
com_result = com_recovery.get_output_dataset("time_stepping_recovery")
# Convert dataset to Pandas DataFrame
com_df = com_result.get_dataframe_from_csv()
# Display top 5 rows of output data
com_df.head()
# we need to first get archetypes from the building inventory dataset
# and merge with the recovery results to group by archetypes
bldg_gdf = bldg_dataset.get_dataframe_from_shapefile()
# merge/join two dataframe
# you can choose columns to be merged
# we need guid to merge the two dataframes
# we can then drop it for the final dataframe
bldg_com_rec_gdf = bldg_gdf[["geometry","guid", "archetype"]].merge(com_df, on="guid").drop(["guid"], axis=1)
# visualizing the repair status at Quarter 6
bldg_com_rec_gdf.explore(
column="quarter_6",
tooltip="quarter_6",
popup=True,
cmap="RdBu" # use "Set1" matplotlib colormap)
)
# visualizing the repair status at Quarter 10
bldg_com_rec_gdf.explore(
column="quarter_10",
tooltip="quarter_10",
popup=True,
cmap="RdBu" # use "Set1" matplotlib colormap)
)
# Unique Archetypes
bldg_com_rec_gdf["archetype"].unique()
bldg_mean_com_rec_df = bldg_com_rec_gdf.drop(["geometry"], axis=1).groupby("archetype").agg("mean")
bldg_mean_com_rec_df.head(10)
# plot the recovery curves
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(10, 6))
# we need to transpose the dataframe to change axes
bldg_mean_com_rec_df.T.plot(kind="line", ax=ax)
ax.set_xlabel("Recovery Time (Quarters)")
ax.set_ylabel("Recovery State") # 0 means damaged completely and 1 means fully recovered
ax.set_title("Joplin Commercial Building Recovery Curves by Archetype")