diff --git a/docsrc/contents/smt.rst b/docsrc/contents/smt.rst index 2dd645e3a..99c58d657 100644 --- a/docsrc/contents/smt.rst +++ b/docsrc/contents/smt.rst @@ -9,7 +9,7 @@ Here, we will demonstrate how each of these components can be implemented, in th Please note that this documentation assumes an elementary knowledge of GMPEs, residual analysis and ground-motion characterisation. Therefore, this documentation's purpose is to facilitate the application of the smt by user who is already familiar with the underlying theory. References are provided throughout for useful overviews of such theory! -Performing a Residual Analysis within the smt +Performing a Residual Analysis ********************************************* The smt provides capabilities (parsers) for the parsing of an inputted dataset into metadata for the performing of a residual analysis, so as to evaluate GMPE performance against the inputted dataset. @@ -94,7 +94,7 @@ We can specify the inputs to perform a residual analysis with as follows: > gmpe_list = ['AkkarEtAlRjb2014', 'BooreEtAl2014', 'BooreEtAl2020', 'CauzziEtAl2014', 'KothaEtAl2020regional', 'LanzanoEtAl2019_RJB_OMO'] > imt_list = ['PGA','SA(0.1)', 'SA(0.2)', 'SA(0.5)', 'SA(1.0)'] -3. We can also specify the GMPEs and intensity measures within a ``.toml`` file. The ``.toml`` file method is required for specifying the inputs of GMPEs with user-specifiable input parameters e.g. regionalisation parameter or logic tree branch parameters. Note that here the GMPEs listed in the first ``.toml`` file are not appropriate for our target region, but have been selected to demonstrate how GMPEs with additional inputs can be specified within a ``.toml`` file. The second ``.toml`` file provides the GMPEs and intensity measures we use for running this demonstration analysis. +3. We can also specify the GMPEs and intensity measures within a ``.toml`` file. The ``.toml`` file method is required for the use of GMPEs with user-specifiable input parameters. Note that here the GMPEs listed in this example ``.toml`` file are not appropriate for our target region, but have been selected to demonstrate how GMPEs with additional inputs can be specified. The additional input parameters which are specifiable for certain GMPEs are available within their corresponding GSIM files (found in ``oq-engine.openquake.hazardlib.gsim``, or for ModifiableGMPE features in ``oq-engine.openquake.hazardlib.gsim.mgmpe.modifiable_gmpe``). Note also that a GMPE sigma model must be provided by the GMPE for the computation of residuals. If a sigma model is not provided by the GMPE, it can be specified as demonstrated below. @@ -104,21 +104,9 @@ We can specify the inputs to perform a residual analysis with as follows: [models] - [models.1-AbrahamsonGulerce2020SInter] - region = "GLO" - - [models.2-AbrahamsonGulerce2020SInter] + [models.AbrahamsonGulerce2020SInter] region = "CAS" - [models.AbrahamsonEtAl2014] - - [models.AbrahamsonEtAl2014RegJPN] - region = "JPN" # nb currently a bug for specifically this gmm in the SMT where the user must still specify the region param despite the class name differentiating as regionalised variant (will be fixed!) - - [models.BooreEtAl2014] - - [models.BooreEtAl2014LowQ] - [models.YenierAtkinson2015BSSA] sigma_model = 'al_atik_2015_sigma' # use Al Atik (2015) sigma model @@ -131,10 +119,10 @@ We can specify the inputs to perform a residual analysis with as follows: [models.3-CampbellBozorgnia2014] set_between_epsilon = 0.5 # Shift the mean with formula mean --> mean + epsilon_tau * between event - [models.1-AbrahamsonEtAl2014] + [models.1-ChiouYoungs2014] median_scaling_scalar = 1.4 # scale median by factor of 1.4 over all imts - [models.2-AbrahamsonEtAl2014] + [models.2-ChiouYoungs2014] median_scaling_vector = "{'PGA': 1.10, 'SA(0.1)': 1.15, 'SA(0.5)': 1.20}" # scale median by imt-dependent factor [models.1-KothaEtAl2020] @@ -151,57 +139,32 @@ We can specify the inputs to perform a residual analysis with as follows: [models.3-BooreEtAl2014] site_term = 'NRCan15SiteTermLinear' # use NRCan15 linear site term - - [models.NGAEastGMPE] - gmpe_table = 'NGAEast_FRANKEL_J15.hdf5' # use a gmpe table [models.HassaniAtkinson2018] - d_sigma = 100 # gmpe specific param + d_sigma = 100 kappa0 = 0.04 [models.KothaEtAl2020ESHM20] # ESHM20 model sigma_mu_epsilon = 2.85697 c3_epsilon = 1.72 region = 4 # Note that within the residuals toml we specify the region here, whereas in the comparison module toml (below) we specify the region for all ESHM20 GMMs uniformly using the eshm20_region param - - [imts] - imt_list = ['PGA', 'SA(0.2)', 'SA(0.5)', 'SA(1.0'] - - Adhering to this formatting, we here provide the GMPEs and intensity measures we consider within the subsequent analysis: - - .. code-block:: ini - - [models] - - [models.AbrahamsonEtAl2014] - - [models.AkkarEtAlRjb2014] - - [models.AmeriEtAl2017Rjb] - - [models.BindiEtAl2014Rjb] - - [models.BooreEtAl2014] - - [models.BooreEtAl2020] - - [models.CauzziEtAl2014] - [models.CampbellBozorgnia2014] - - [models.ChiouYoungs2014] - - [models.HassaniAtkinson2020Asc] - - [models.KaleEtAl2015Turkey] - - [models.KothaEtAl2020regional] - - [models.LanzanoEtAl2019_RJB_OMO] + [models.NGAEastGMPE] + gmpe_table = 'NGAEast_FRANKEL_J15.hdf5' # use a gmpe table + # Note: currently a bug for GMMs which use add_alias to specify gsim + # class (will be fixed - current workarounds demonstrated below) + + [models.AbrahamsonEtAl2014RegJPN] + region = "JPN" # add_alias bug means must still specify 'JPN' region param + + [models.NGAEastUSGSGMPE] + gmpe_table = 'usgs17.hdf5' # another example of add_alias bug + + [imts] - imt_list = ["PGA","SA(0.1)","SA(0.2)","SA(0.5)","SA(1.0)","SA(2.0)"] - + imt_list = ['PGA', 'SA(0.2)', 'SA(0.5)', 'SA(1.0'] + 4. Following specification of the GMPEs and intensity measures, we can now compute the ground-motion residuals using the Residuals module. We first need to get the metadata from the parsed ``.pkl`` file (stored within the metadata folder): @@ -338,7 +301,7 @@ Single Station Residual Analysis Finally, the standard deviation of the :math:`\delta W_{o,es}` term (:math:`\phi_{SS}`) is representative of the single-station standard deviation of the GMPE, and is an estimate of the non-ergodic standard deviation of the model. - As previously, we can specify the GMPEs and intensity measures to compute the residuals per site for using either a GMPE list and intensity measure list, or from a .toml file. + As previously, we can specify the GMPEs and intensity measures to compute the residuals per site for using either a GMPE list and intensity measure list, or from a ``.toml`` file. .. code-block:: ini @@ -493,7 +456,7 @@ Comparing GMPEs ztor = 'None' # Set to string of 'None' to NOT consider otherwise specify as array matching number of mag and depth values strike = -999 dip = 60 - rake = 90 # (+90 for compression, -90 for extension) + rake = 90 # Must be provided. Strike and dip can be approximated if either set to -999 aratio = 2 # If set to -999 the user-provided trt string will be used to assign a trt-dependent aratio trellis_and_rs_mag_list = [5, 6, 7] # mags used only for trellis and response spectra trellis_and_rs_depths = [20, 20, 20] # depth per magnitude for trellis and response spectra @@ -572,7 +535,7 @@ Comparing GMPEs 4. Spectra Plots - We can also plot response spectra. Note that a spectra computed from a recorded ground-motion and the corresponding ground-motions predicted by the considered GMPEs can be plotted (instead of iterating through the provided magnitudes and distances) by specifying the path to a ``.csv`` of the spectra using the ``obs_spectra`` variable (see the example spectra file in openquake.smt.tests.file_samples, and the functions within openquake.smt.comparison for more details): + We can also plot response spectra: .. code-block:: ini @@ -581,11 +544,23 @@ Comparing GMPEs Response spectra plots for input parameters specified in toml file: .. image:: /contents/smt_images/ResponseSpectra.png - - GMPE sigma spectra plots for input parameters specified in toml file: - .. image:: /contents/smt_images/sigma.png - -5. Sammon's Maps + +5. Plot of Spectra from a Record + + The spectra of a processed record can also be plotted along with predictions by the selected GMMs for the same ground-shaking scenario. An example of the input for the record spectra is provided in the demo files: + + .. code-block:: ini + + > # Generate plot of observed spectra and predictions by GMMs + > # Note we use spectra from a record for the 1991 Chamoli EQ in this + > # example rather than from a record from an earthquake in/near Albania + > comp.plot_spectra(filename, output_directory, obs_spectra = 'spectra_chamoli_1991_station_UKHI.csv') + + Response spectra plots for input parameters specified in toml file: + .. image:: /contents/smt_images/ObsSpectra.png + + +6. Sammon's Maps We can plot Sammon's Maps to examine how similar the medians (and 16th and 84th percentiles) of predicted ground-motion of each GMPE are (see Sammon, 1969 and Scherbaum et al. 2010 for more details on the Sammon's mapping procedure). @@ -601,7 +576,7 @@ Comparing GMPEs Sammon's Maps (median predicted ground-motion) for input parameters specified in toml file: .. image:: /contents/smt_images/Median_SammonMaps.png -6. Hierarchical Clustering +7. Hierarchical Clustering Dendrograms can be plotted as an alternative tool to evaluate how similarly the predicted ground-motion is by each GMPE. @@ -617,7 +592,7 @@ Comparing GMPEs Dendrograms (median predicted ground-motion) for input parameters specified in toml file: .. image:: /contents/smt_images/Median_Clustering.png -7. Matrix Plots of Euclidean Distance +8. Matrix Plots of Euclidean Distance In addition to Sammon's Maps and hierarchical clustering, we can also plot the Euclidean distance between the predicted ground-motions by each GMPE in a matrix plot. diff --git a/docsrc/contents/smt_images/ObsSpectra.png b/docsrc/contents/smt_images/ObsSpectra.png new file mode 100644 index 000000000..2ff6ee014 Binary files /dev/null and b/docsrc/contents/smt_images/ObsSpectra.png differ diff --git a/docsrc/contents/smt_images/sigma.png b/docsrc/contents/smt_images/sigma.png deleted file mode 100644 index ce9339a4c..000000000 Binary files a/docsrc/contents/smt_images/sigma.png and /dev/null differ diff --git a/openquake/smt/comparison/compare_gmpes.py b/openquake/smt/comparison/compare_gmpes.py index f8eaf62e4..8a1e44da5 100644 --- a/openquake/smt/comparison/compare_gmpes.py +++ b/openquake/smt/comparison/compare_gmpes.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2023 GEM Foundation +# Copyright (C) 2014-2024 GEM Foundation # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -201,7 +201,7 @@ def plot_trellis(filename, output_directory): config.up_or_down_dip) -def plot_spectra(filename, output_directory, obs_spectra = None): +def plot_spectra(filename, output_directory, obs_spectra=None): """ Plot response spectra and GMPE sigma wrt spectral period for given run configuration diff --git a/openquake/smt/comparison/utils_compare_gmpes.py b/openquake/smt/comparison/utils_compare_gmpes.py index e1e2effe7..2bc80e752 100644 --- a/openquake/smt/comparison/utils_compare_gmpes.py +++ b/openquake/smt/comparison/utils_compare_gmpes.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2023 GEM Foundation +# Copyright (C) 2014-2024 GEM Foundation # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -290,7 +290,7 @@ def compute_matrix_gmpes(trt, ztor, imt_list, mag_list, gmpe_list, rake, strike, mtxs_median = {} d_step = 1 Z1, Z25 = get_z1_z25(Z1, Z25, Vs30, region) - for n, i in enumerate(imt_list): # Iterate though imt_list + for n, i in enumerate(imt_list): # Iterate through imt_list matrix_medians=np.zeros((len(gmpe_list), (len(mag_list)*int((maxR-minR)/d_step)))) @@ -317,7 +317,7 @@ def compute_matrix_gmpes(trt, ztor, imt_list, mag_list, gmpe_list, rake, strike, dist_type, trt, up_or_down_dip) # Get means further than minR - idx = np.argwhere(r_vals>minR).flatten() + idx = np.argwhere(r_vals>=minR).flatten() mean = [mean[0][0][idx]] std = [std[0][0][idx]] tau = [tau[0][0][idx]] @@ -331,7 +331,7 @@ def compute_matrix_gmpes(trt, ztor, imt_list, mag_list, gmpe_list, rake, strike, if mtxs_type == '16th_perc': Nstd = 1 # Median - 1std = ~16th percentile medians = np.append(medians, (np.exp(mean-Nstd*std[0]))) - sigmas = np.append(sigmas,std[0]) + sigmas = np.append(sigmas, std[0]) matrix_medians[:][g]= medians mtxs_median[n] = matrix_medians @@ -971,7 +971,7 @@ def update_spec_plots(ax1, ax2, m, i, n, l, dist_list): ax1.set_xlabel('Period (s)', fontsize=16) ax2.set_xlabel('Period (s)', fontsize=16) if l == 0: # left row only - ax1.set_ylabel('Sa (g)', fontsize=16) + ax1.set_ylabel('SA (g)', fontsize=16) ax2.set_ylabel(r'$\sigma$', fontsize=16) diff --git a/openquake/smt/comparison/utils_gmpes.py b/openquake/smt/comparison/utils_gmpes.py index 59bb72435..697c53a54 100644 --- a/openquake/smt/comparison/utils_gmpes.py +++ b/openquake/smt/comparison/utils_gmpes.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2023 GEM Foundation +# Copyright (C) 2014-2024 GEM Foundation # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -242,8 +242,8 @@ def _get_z25(Vs30, region): def _param_gmpes(strike, dip, depth, aratio, rake, trt): """ - Get (crude) proxies for strike, dip, depth and aspect ratio if not provided - by the user. + Get (crude) proxies for strike, dip, rake, depth and aspect ratio if not + provided by the user. """ # Strike if strike == -999: @@ -253,32 +253,32 @@ def _param_gmpes(strike, dip, depth, aratio, rake, trt): # Dip if dip == -999: - if rake == 0: - dip_s = 90 # strike slip + if rake == 0 or rake == 180: + dip_s = 90 # Strike slip else: - dip_s = 45 # reverse or normal fault + dip_s = 45 # Reverse or normal fault else: dip_s = dip # Depth if depth == -999: if trt == 'Interface': - depth_s = 30 + depth_s = 40 if trt == 'InSlab': - depth_s = 50 + depth_s = 150 else: - depth_s = 15 + depth_s = 15 # Crustal else: depth_s = depth - # a-ratio + # Aspect ratio if aratio > -999.0 and np.isfinite(aratio): aratio_s = aratio else: - if trt == 'InSlab' or trt == 'Interface': + if trt in ['InSlab', 'Interface']: aratio_s = 5 else: - aratio_s = 2 + aratio_s = 2 # Crustal return strike_s, dip_s, depth_s, aratio_s @@ -292,12 +292,12 @@ def mgmpe_check(gmpe): # Preserve original GMPE prior and create base version of GMPE orig_gmpe = gmpe base_gsim = gmpe.__class__.__name__ - + # Get the additional params if specified inputs = pd.Series(str(gmpe).splitlines()[1:], dtype='object') add_inputs = {} add_as_int = ['eshm20_region'] - add_as_str = ['region', 'gmpe_table', 'volc_arc_file'] + add_as_str = ['region', 'saturation_region', 'gmpe_table', 'volc_arc_file'] if len(inputs) > 0: # If greater than 0 must add required gsim inputs idx_to_drop = [] @@ -340,14 +340,14 @@ def mgmpe_check(gmpe): val = float(par.split('=')[1]) add_inputs[key] = val - # reconstruct the gmpe as kwargs + # Reconstruct the gmpe as kwargs kwargs = {'gmpe': {base_gsim: add_inputs}} # Al Atik 2015 sigma model if 'al_atik_2015_sigma' in str(orig_gmpe): - kwargs['sigma_model_alatik2015'] = {"tau_model": "global", - "ergodic": False} - + kwargs['sigma_model_alatik2015'] = { + "tau_model": "global", "ergodic": False} + # Fix total sigma per imt if 'fix_total_sigma' in str(orig_gmpe): kwargs['set_fixed_total_sigma'] = {'total_sigma': fixed_sigma_vector} @@ -392,6 +392,6 @@ def mgmpe_check(gmpe): if 'NRCan15SiteTermLinear' in str(orig_gmpe): kwargs['nrcan15_site_term'] = {'kind': 'linear'} - gmpe = mgmpe.ModifiableGMPE(**kwargs) # remake gmpe using mgmpe + gmpe = mgmpe.ModifiableGMPE(**kwargs) # Remake gmpe using mgmpe return gmpe diff --git a/openquake/smt/data_default.py b/openquake/smt/data_default.py index 5ffa67b38..e7e338ec6 100644 --- a/openquake/smt/data_default.py +++ b/openquake/smt/data_default.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -15,7 +15,6 @@ # # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . - """ IMS IMS/V diff --git a/openquake/smt/database_visualiser.py b/openquake/smt/database_visualiser.py index f7fc8e214..a8d2f0157 100755 --- a/openquake/smt/database_visualiser.py +++ b/openquake/smt/database_visualiser.py @@ -1,8 +1,7 @@ -#!/usr/bin/env python # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,7 +15,6 @@ # # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . - """ Tool for creating visualisation of database information """ diff --git a/openquake/smt/demos/demo_comparison_script.py b/openquake/smt/demos/demo_comparison_script.py index 6cf6a029d..49be29d7f 100644 --- a/openquake/smt/demos/demo_comparison_script.py +++ b/openquake/smt/demos/demo_comparison_script.py @@ -1,13 +1,14 @@ """ -This notebook provides a simple and efficient script for comparing candidate -GMPEs in terms of median predicted ground-motion using trellis plots, -hierarchical clustering plots, Euclidean distance matrix plots and Sammons Maps -plotting functions available within the SMT's comparison module. +This notebook provides a simple script for comparing candidate GMMs using +trellis plots, hierarchical clustering plots, Euclidean distance matrix plots +and Sammons Maps plotting functions available within the SMT's comparison module. """ import os import toml from openquake.smt.comparison import compare_gmpes as comp +import warnings +warnings.filterwarnings("ignore") # User input (can add more input tomls to run multiple analyses if required) file_list = ['demo_comparison_analysis_inputs.toml'] diff --git a/openquake/smt/demos/demo_residuals_script.py b/openquake/smt/demos/demo_residuals_script.py index 2268901a5..b924e6766 100644 --- a/openquake/smt/demos/demo_residuals_script.py +++ b/openquake/smt/demos/demo_residuals_script.py @@ -1,19 +1,19 @@ """ -This demo script runs a residual analysis using a subset of the ESM flatfile -filtered geographically to Albania + a 100 km buffer. - -The results generated by this workflow can be used to deduce which of the -preselected GMMs perform best for Albania. +This demo script runs a residual analysis using a subset of the URL-searchable +ESM database (https://esm-db.eu/esmws/flatfile/1/) """ import os import shutil import pickle from openquake.baselib import sap -from openquake.smt.parsers.esm23_flatfile_parser import ESM23FlatfileParser +from openquake.smt.parsers.esm_url_flatfile_parser import ESMFlatfileParserURL from openquake.smt.residuals import gmpe_residuals as res from openquake.smt.residuals import residual_plotter as rspl +import warnings +warnings.filterwarnings("ignore") + """USER INPUTS""" #Specify absolute path @@ -37,7 +37,7 @@ def parse_into_metadata(): shutil.rmtree(metadata_dir) # Parse the metadata - ESM23FlatfileParser.autobuild("000", 'db', metadata_dir, db) + ESMFlatfileParserURL.autobuild("000", 'db', metadata_dir, db) def get_residuals(): @@ -61,7 +61,7 @@ def get_residuals(): for gmm in residuals.gmpe_list: # Create output directory - gmm_dir = residuals.gmpe_list[gmm].__class__.__name__ + gmm_dir = residuals.gmpe_list[gmm]._toml.split('\n')[0] out = os.path.join('residuals', gmm_dir) if not os.path.exists(out): os.makedirs(out) @@ -96,17 +96,18 @@ def get_residuals(): rspl.plot_residual_pdf_with_spectral_period(residuals, fi_pdf) rspl.pdf_table(residuals, fi_pdf_table) + return residuals + def main(): """ Run the demo workflow """ - - # Parse each flatfile into metadata + # Parse flatfile into metadata parse_into_metadata() # Get the residuals per trt - res_per_trt = get_residuals() + res = get_residuals() if __name__ == '__main__': diff --git a/openquake/smt/demos/demo_single_station_analysis.py b/openquake/smt/demos/demo_single_station_analysis.py new file mode 100644 index 000000000..0cdfc1fac --- /dev/null +++ b/openquake/smt/demos/demo_single_station_analysis.py @@ -0,0 +1,132 @@ +""" +This demo script runs a single stationresidual analysis using a subset of the +ESM flatfile filtered geographically to Albania + an approximately 100 km buffer. +""" +import os +import pickle +import shutil + +from openquake.baselib import sap +from openquake.smt.parsers.esm_url_flatfile_parser import ESMFlatfileParserURL +import openquake.smt.residuals.gmpe_residuals as res +import openquake.smt.residuals.residual_plotter as rspl +from openquake.smt.strong_motion_selector import rank_sites_by_record_count + +import warnings +warnings.filterwarnings("ignore") + +"""USER INPUTS""" + +# Flatfile to use +db = 'demo_flatfile.csv' + +# Specify .toml file with GMPEs and imts to use +gmms_imts = 'demo_residual_analysis_inputs.toml' + +# Specify results folder name +run_folder = 'results_single_station_analysis' + +# Minimum number of records for a site to be considered in SSA +threshold = 45 + + +def get_residuals(): + """ + Compute the residuals from the example flatfile, GMMs and imts + """ + # Create metadata directory + metadata_dir = 'metadata' + if os.path.exists(metadata_dir): + shutil.rmtree(metadata_dir) + + # Parse the metadata + ESMFlatfileParserURL.autobuild("000", 'db', metadata_dir, db) + + # Get inputs + metadata = os.path.join('metadata', 'metadatafile.pkl') + sm_database = pickle.load(open(metadata,"rb")) + + # If output directory for residuals exists remove and remake + if os.path.exists('residuals'): + shutil.rmtree('residuals') + + # Get residuals + residuals = res.Residuals.from_toml(gmms_imts) + residuals.get_residuals(sm_database) + + # Create results folder for single station analysis + if os.path.exists(run_folder): + shutil.rmtree(run_folder) + os.mkdir(run_folder) + + return sm_database + + +def single_station_analysis(sm_database): + """ + Perform the analysis using the demo files + """ + # Find sites with threshold minimum for number of records + top_sites = rank_sites_by_record_count(sm_database, threshold) + + # For each station print some info + msg = 'Sites with required threshold of at least %s records' %(threshold) + print(msg) + for idx, site_id in enumerate(top_sites.keys()): + print(" Site ID: %s Name: %s, Number of Records: %s" %( + site_id, top_sites[site_id]["Name"], top_sites [site_id]["Count"])) + + # Create SingleStationAnalysis object + ssa1 = res.SingleStationAnalysis.from_toml(top_sites.keys(), gmms_imts) + + # Compute the total, inter-event and intra-event residuals for each site + ssa1.get_site_residuals(sm_database) + + # Output for summary csv + csv_output = os.path.join(run_folder, 'ssa_results.csv') + + # Get summary of statistics and output them + ssa1.residual_statistics(True, csv_output) + + # Output plots + for gmpe in ssa1.gmpe_list: + for imt in ssa1.imts: + + # Create folder for the GMM + gmpe_folder_name = str(ssa1.gmpe_list[gmpe]).split( + '(')[0].replace('\n','_').replace(' ','').replace('"','') + + gmpe_folder = os.path.join(run_folder, gmpe_folder_name) + + if not os.path.exists(gmpe_folder): + os.mkdir(gmpe_folder) + + # Filenames + output_all_res_with_site = os.path.join( + gmpe_folder, gmpe_folder_name + '_' + imt + '_' + + 'AllResPerSite.jpg') + output_intra_res_components_with_site = os.path.join( + gmpe_folder, gmpe_folder_name + '_' + imt + '_' + + 'IntraResCompPerSite.jpg') + + # Create the plots and save + rspl.ResidualWithSite(ssa1, gmpe, imt, + output_all_res_with_site, filetype='jpg') + rspl.IntraEventResidualWithSite( + ssa1, gmpe, imt, output_intra_res_components_with_site, + iletype='jpg') + + +def main(): + """ + Run the demo for single station residual analysis + """ + # Get residuals + sm_database = get_residuals() + + # Run the single station analysis + single_station_analysis(sm_database) + + +if __name__ == '__main__': + sap.run(main) \ No newline at end of file diff --git a/openquake/smt/intensity_measures.py b/openquake/smt/intensity_measures.py index 987631393..46a75cdbd 100755 --- a/openquake/smt/intensity_measures.py +++ b/openquake/smt/intensity_measures.py @@ -1,8 +1,7 @@ -#!/usr/bin/env python # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,7 +15,6 @@ # # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . - """ General Class for extracting Ground Motion Intensity Measures (IMs) from a set of acceleration time series diff --git a/openquake/smt/parsers/asa_database_parser.py b/openquake/smt/parsers/asa_database_parser.py index 734671705..828349791 100755 --- a/openquake/smt/parsers/asa_database_parser.py +++ b/openquake/smt/parsers/asa_database_parser.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -66,7 +66,6 @@ def _get_metadata_from_file(file_str): version as an ordered dictionary. Note that every file contains the metadata corresponding to all 3 components. """ - metadata = {} for i in range(7, 81): # Exclude lines with "==", websites, and with lenghts < 42 @@ -94,7 +93,6 @@ def _get_metadata_from_file(file_str): class ASADatabaseMetadataReader(SMDatabaseReader): - """ Reader for the metadata database of the UNAM and CICESE files (ASA format) """ @@ -102,7 +100,7 @@ class ASADatabaseMetadataReader(SMDatabaseReader): def parse(self): """ - Parses the record + Parse the record """ self.database = GroundMotionDatabase(self.id, self.name) self._sort_files() @@ -140,7 +138,6 @@ def parse_metadata(self, metadata, file_dict): """ Parses the metadata dictionary """ - file_str = metadata["NOMBRE DEL ARCHIVO"] file_info = _get_info_from_archive_name(file_str) @@ -178,7 +175,6 @@ def _parse_event(self, metadata, file_str): openquake.smt.sm_database.Earthquake. Coordinates in western hemisphere are returned as negative values. """ - months = {'ENERO': 1, 'FEBRERO': 2, 'MARZO': 3, 'ABRIL': 4, 'MAYO': 5, 'JUNIO': 6, 'JULIO': 7, 'AGOSTO': 8, 'SEPTIEMBRE': 9, 'OCTUBURE': 10, 'NOVIEMBRE': 11, 'DICIEMBRE': 12} @@ -328,7 +324,6 @@ def _parse_distance_data(self, metadata, file_str, eqk): openquake.smt.sm_database.RecordDistance. Coordinates in western hemisphere are converted to negative values. """ - epi_lon = -get_float( metadata["COORDENADAS DEL EPICENTRO"].split(" ")[3]) epi_lat = get_float( @@ -431,7 +426,6 @@ def parse_records(self, record=None): """ Parses the time series """ - time_series = OrderedDict([ ("X", {"Original": {}, "SDOF": {}}), ("Y", {"Original": {}, "SDOF": {}}), @@ -454,7 +448,6 @@ def _parse_time_history(self, ifile, component2parse): components are defined with various names, and are not always given in the same order """ - # The components are definied using the following names comp_names = {'X': ['ENE', 'N90E', 'N90E;', 'N90W', 'N90W;', 'S90E', 'S90W', 'E--W', 'S9OE'], diff --git a/openquake/smt/parsers/base_database_parser.py b/openquake/smt/parsers/base_database_parser.py index 220dc638a..7e0beec36 100755 --- a/openquake/smt/parsers/base_database_parser.py +++ b/openquake/smt/parsers/base_database_parser.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/parsers/esm_database_parser.py b/openquake/smt/parsers/esm_database_parser.py index dfbb4a685..9a7518bba 100755 --- a/openquake/smt/parsers/esm_database_parser.py +++ b/openquake/smt/parsers/esm_database_parser.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -431,8 +431,7 @@ def _parse_component_data(self, wfid, metadata): class ESMTimeSeriesParser(SMTimeSeriesReader): """ Parses time series in the European Strong Motion Database Format - """ - + """ def parse_records(self, record=None): """ Parses the time series diff --git a/openquake/smt/parsers/esm_dictionaries.py b/openquake/smt/parsers/esm_dictionaries.py index 82a89959f..284dcfe92 100644 --- a/openquake/smt/parsers/esm_dictionaries.py +++ b/openquake/smt/parsers/esm_dictionaries.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/parsers/esm_flatfile_parser.py b/openquake/smt/parsers/esm_flatfile_parser.py index 6e95d2dca..242394f9b 100644 --- a/openquake/smt/parsers/esm_flatfile_parser.py +++ b/openquake/smt/parsers/esm_flatfile_parser.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -17,7 +17,7 @@ # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . """ -Parser from the ESM Flatfile to SMT +Parse ESM format flatfile into SMT metadata """ import os, sys import csv @@ -82,14 +82,14 @@ class ESMFlatfileParser(SMDatabaseReader): """ - Parses the ESM metadata from the flatfile to a set of metadata objects + Parses the data from the flatfile to a set of metadata objects """ M_PRECEDENCE = ["EMEC_Mw", "Mw", "Ms", "ML"] BUILD_FINITE_DISTANCES = False def parse(self, location="./"): """ - + Parse the flatfile """ assert os.path.isfile(self.filename) headers = getline(self.filename, 1).rstrip("\n").split(";") @@ -99,24 +99,22 @@ def parse(self, location="./"): % hdr) # Read in csv reader = csv.DictReader(open(self.filename, "r"), delimiter=";") - metadata = [] self.database = GroundMotionDatabase(self.id, self.name) counter = 0 for row in reader: - if self._sanitise(row, reader): - # Build the metadata - record = self._parse_record(row) - if record: - # Parse the strong motion - record = self._parse_ground_motion( - os.path.join(location, "records"), - row, record, headers) - self.database.records.append(record) + # Build the metadata + record = self._parse_record(row) + if record: + # Parse the strong motion + record = self._parse_ground_motion( + os.path.join(location, "records"), + row, record, headers) + self.database.records.append(record) - else: - print("Record with sequence number %s is null/invalid" - % "{:s}-{:s}".format(row["event_id"], - row["station_code"])) + else: + print("Record with sequence number %s is null/invalid" + % "{:s}-{:s}".format(row["event_id"], + row["station_code"])) if (counter % 100) == 0: print("Processed record %s - %s" % (str(counter), record.id)) @@ -146,13 +144,10 @@ def autobuild(cls, dbid, dbname, output_location, flatfile_location): pickle.dump(database.database, f) return database - def _sanitise(self, row, reader): + def _parse_record(self, metadata): """ - TODO - Not implemented yet! + Parse a record """ - return True - - def _parse_record(self, metadata): # Waveform ID not provided in file so concatenate Event and Station ID wfid = "_".join([metadata["event_id"], metadata["network_code"], metadata["station_code"], metadata["location_code"]]) @@ -239,7 +234,6 @@ def _parse_rupture_mechanism(self, metadata, eq_id, eq_name, mag, depth): """ If rupture data is available - parse it, otherwise return None """ - sof = metadata["fm_type_code"] if not metadata["event_source_id"].strip(): # No rupture model available. Mechanism is limited to a style @@ -345,7 +339,6 @@ def _parse_site_data(self, metadata): network_code = metadata["network_code"].strip() station_code = metadata["station_code"].strip() site_id = "{:s}-{:s}".format(network_code, station_code) - location_code = metadata["location_code"].strip() site_lon = valid.longitude(metadata["st_longitude"]) site_lat = valid.latitude(metadata["st_latitude"]) elevation = valid.vfloat(metadata["st_elevation"], "st_elevation") @@ -497,10 +490,9 @@ def _parse_ground_motion(self, location, row, record, headers): record.datafile = filename return record - def _retreive_ground_motion_from_row(self, row, header_list): """ - + Get the ground motion data from a row (record) in the database """ imts = ["U", "V", "W", "rotD00", "rotD100", "rotD50"] spectra = [] @@ -528,7 +520,6 @@ def _retreive_ground_motion_from_row(self, row, header_list): # Not a spectral period but T90 continue iky = header.replace(key, "").replace("_", ".") - #print imt, key, header, iky periods.append(float(iky)) value = row[header].strip() if value: @@ -554,5 +545,4 @@ def _retreive_ground_motion_from_row(self, row, header_list): if scalars["U"][key] and scalars["V"][key]: scalars["Geometric"][key] = np.sqrt( scalars["U"][key] * scalars["V"][key]) - return scalars, spectra - + return scalars, spectra \ No newline at end of file diff --git a/openquake/smt/parsers/esm23_flatfile_parser.py b/openquake/smt/parsers/esm_url_flatfile_parser.py similarity index 59% rename from openquake/smt/parsers/esm23_flatfile_parser.py rename to openquake/smt/parsers/esm_url_flatfile_parser.py index c60f77bd5..14036f390 100644 --- a/openquake/smt/parsers/esm23_flatfile_parser.py +++ b/openquake/smt/parsers/esm_url_flatfile_parser.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,8 +16,8 @@ # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . """ -Parser for the ESM23 flatfile format (i.e. flatfile downloaded from custom -header URL) to SMT +Parser for a flatfile downloaded from the ESM custom url database +--> (https://esm-db.eu/esmws/flatfile/1/) This parser assumes you have selected all available headers in your URL search when downloading the flatfile @@ -49,6 +49,9 @@ # Import the ESM dictionaries from .esm_dictionaries import * +# Define base path +DATA = os.path.abspath('') + SCALAR_LIST = ["PGA", "PGV", "PGD", "CAV", "CAV5", "Ia", "D5-95"] HEADER_STR = "event_id;event_time;ISC_ev_id;USGS_ev_id;INGV_ev_id;"\ @@ -69,39 +72,16 @@ HEADERS = set(HEADER_STR.split(";")) -COUNTRY_CODES = {"AL": "Albania", "AM": "Armenia", "AT": "Austria", - "AR": "Argentina", "AZ": "Azerbaijan", - "BA": "Bosnia and Herzegowina", "BG": "Bulgaria", - "CH": "Switzerland", "CL": "Chile", "CN": "China", - "CR": "Costa Rica", "CY": "Cyprus", "CZ": "Czech Republic", - "DE": "Germany", "DJ": "Djibouti", "DZ": "Algeria", - "ES": "Spain", "FR": "France", "GE": "Georgia", "GH": "Ghana", - "GR": "Greece", "HR": "Croatia", "HU": "Hungary", - "IL": "Israel", "ID": "Indonesia", "IR": "Iran", - "IS": "Iceland", "IT": "Italy", "JO": "Jordan", "KE":"Kenya", - "KG": "Kyrgyzstan", "KZ": "Kazakhstan", "LI": "Lichtenstein", - "MA": "Morocco", "MC": "Monaco", "MD": "Moldova", - "ME": "Montenegro", "MK": "Macedonia", "MM": "Myanmar", - "MT": "Malta", "MX": "Mexico", "NI": "Nicaragua", - "NO": "Norway", "PA": "Panama", "PG": "Papa New Guinea", - "PL": "Poland", "PT": "Portugal", "PS": "Palestine", - "RO": "Romania", "RS": "Serbia", "RU": "Russia", - "SI": "Slovenia", "SM": "San Marino", "SY": "Syria", - "TM": "Turkmenistan", "TR": "Turkey", "TW": "Taiwan", - "UA": "Ukraine", "US": "United States", "UZ": "Uzbekistan", - "VU": "Vanuatu", "XK": "Kosovo", "YE": "Yemen"} - -class ESM23FlatfileParser(SMDatabaseReader): - +class ESMFlatfileParserURL(SMDatabaseReader): """ - Parses the ESM metadata from the flatfile to a set of metadata objects + Parses the data from the flatfile to a set of metadata objects """ - M_PRECEDENCE = ["EMEC_Mw", "Mw", "Ms", "ML"] BUILD_FINITE_DISTANCES = False def parse(self, location='./'): """ + Parse the flatfile """ assert os.path.isfile(self.filename) headers = getline(self.filename, 1).rstrip("\n").split(";") @@ -111,24 +91,22 @@ def parse(self, location='./'): % hdr) # Read in csv reader = csv.DictReader(open(self.filename, "r"), delimiter=";") - metadata = [] self.database = GroundMotionDatabase(self.id, self.name) counter = 0 for row in reader: - if self._sanitise(row, reader): - # Build the metadata - record = self._parse_record(row) - if record: - # Parse the strong motion - record = self._parse_ground_motion( - os.path.join(location, "records"), - row, record, headers) - self.database.records.append(record) + # Build the metadata + record = self._parse_record(row) + if record: + # Parse the strong motion + record = self._parse_ground_motion( + os.path.join(location, "records"), + row, record, headers) + self.database.records.append(record) - else: - print("Record with sequence number %s is null/invalid" - % "{:s}-{:s}".format(row["event_id"], - row["station_code"])) + else: + print("Record with sequence number %s is null/invalid" + % "{:s}-{:s}".format(row["event_id"], + row["station_code"])) if (counter % 100) == 0: print("Processed record %s - %s" % (str(counter), record.id)) @@ -136,25 +114,15 @@ def parse(self, location='./'): counter += 1 @classmethod - def autobuild(cls, dbid, dbname, output_location, - ESM23_flatfile_directory): + def autobuild(cls, dbid, dbname, output_location, ESM_flatfile_directory): """ Quick and dirty full database builder! """ - # Import ESM 2023 format strong-motion flatfile - ESM23 = pd.read_csv(ESM23_flatfile_directory) - - # Create default values for headers not considered in ESM23 format - default_string = pd.Series(np.full(np.size(ESM23.esm_event_id), "")) - - # Assign strike-slip to unknown faulting mechanism - r_fm_type = ESM23.fm_type_code.fillna('SS') + # Import ESM URL format strong-motion flatfile + ESM = pd.read_csv(ESM_flatfile_directory) - # Reformat datetime - r_datetime = ESM23.event_time.str.replace('T',' ') - - converted_base_data_path=_get_ESM18_headers( - ESM23, default_string, r_fm_type, r_datetime) + # Get path to tmp csv once modified dataframe + converted_base_data_path=_get_ESM18_headers(ESM) if os.path.exists(output_location): raise IOError("Target database directory %s already exists!" @@ -175,13 +143,10 @@ def autobuild(cls, dbid, dbname, output_location, return database - def _sanitise(self, row, reader): + def _parse_record(self, metadata): """ - TODO - Not implemented yet! + Parse a record """ - return True - - def _parse_record(self, metadata): # Waveform ID not provided in file so concatenate Event and Station ID wfid = "_".join([metadata["event_id"], metadata["network_code"], metadata["station_code"], metadata["location_code"]]) @@ -200,7 +165,6 @@ def _parse_record(self, metadata): xcomp, ycomp, vertical=vertical) - def _parse_event_data(self, metadata): """ Parses the event metadata @@ -208,12 +172,7 @@ def _parse_event_data(self, metadata): # ID and Name (name not in file so use ID again) eq_id = metadata["event_id"] eq_name = metadata["event_id"] - # Country - cntry_code = metadata["ev_nation_code"].strip() - if cntry_code and cntry_code in COUNTRY_CODES: - eq_country = COUNTRY_CODES[cntry_code] - else: - eq_country = None + # Date and time eq_datetime = valid.date_time(metadata["event_time"], "%Y-%m-%d %H:%M:%S") @@ -225,7 +184,7 @@ def _parse_event_data(self, metadata): eq_depth = 0.0 eqk = Earthquake(eq_id, eq_name, eq_datetime, eq_lon, eq_lat, eq_depth, None, # Magnitude not defined yet - eq_country=eq_country) + eq_country=None) # Get preferred magnitude and list pref_mag, magnitude_list = self._parse_magnitudes(metadata) eqk.magnitude = pref_mag @@ -374,7 +333,6 @@ def _parse_site_data(self, metadata): network_code = metadata["network_code"].strip() station_code = metadata["station_code"].strip() site_id = "{:s}-{:s}".format(network_code, station_code) - location_code = metadata["location_code"].strip() site_lon = valid.longitude(metadata["st_longitude"]) site_lat = valid.latitude(metadata["st_latitude"]) elevation = valid.vfloat(metadata["st_elevation"], "st_elevation") @@ -388,15 +346,9 @@ def _parse_site_data(self, metadata): vs30_measured = False else: vs30_measured = False - st_nation_code = metadata["st_nation_code"].strip() - if st_nation_code: - st_country = COUNTRY_CODES[st_nation_code] - else: - st_country = None site = RecordSite(site_id, station_code, station_code, site_lon, site_lat, elevation, vs30, vs30_measured, - network_code=network_code, - country=st_country) + network_code=network_code, country=None) site.slope = valid.vfloat(metadata["slope_deg"], "slope_deg") site.sensor_depth = valid.vfloat(metadata["sensor_depth_m"], "sensor_depth_m") @@ -529,7 +481,7 @@ def _parse_ground_motion(self, location, row, record, headers): def _retreive_ground_motion_from_row(self, row, header_list): """ - + Get the ground-motion data from a row (record) in the database """ imts = ["U", "V", "W", "rotD00", "rotD100", "rotD50"] spectra = [] @@ -557,14 +509,12 @@ def _retreive_ground_motion_from_row(self, row, header_list): # Not a spectral period but T90 continue iky = header.replace(key, "").replace("_", ".") - #print imt, key, header, iky periods.append(float(iky)) value = row[header].strip() if value: values.append(np.fabs(float(value))) else: values.append(np.nan) - #values.append(np.fabs(float(row[header].strip()))) periods = np.array(periods) values = np.array(values) idx = np.argsort(periods) @@ -585,364 +535,372 @@ def _retreive_ground_motion_from_row(self, row, header_list): scalars["U"][key] * scalars["V"][key]) return scalars, spectra -def _get_ESM18_headers(ESM23, default_string, r_fm_type, r_datetime): + +def _get_ESM18_headers(ESM): """ - Convert from ESM23 format flatfile to ESM18 format flatfile + Convert from ESM URL format flatfile to ESM18 format flatfile """ + # Create default values + default_string = pd.Series(np.full(np.size(ESM.esm_event_id), "")) + + # Reformat datetime + r_datetime = ESM.event_time.str.replace('T',' ') + + # Assign unknown to NaN values for faulting mechanism + ESM['fm_type_code'] = ESM.fm_type_code.fillna('U') + # Construct dataframe with original ESM format ESM_original_headers = pd.DataFrame( { # Non-GMIM headers - "event_id":ESM23.esm_event_id, + "event_id":ESM.esm_event_id, "event_time":r_datetime, - "ISC_ev_id":ESM23.isc_event_id, - "USGS_ev_id":ESM23.usgs_event_id, - "INGV_ev_id":ESM23.ingv_event_id, - "EMSC_ev_id":ESM23.emsc_event_id, - "ev_nation_code":ESM23.ev_nation_code, - "ev_latitude":ESM23.ev_latitude, - "ev_longitude":ESM23.ev_longitude, - "ev_depth_km":ESM23.ev_depth_km, + "ISC_ev_id":ESM.isc_event_id, + "USGS_ev_id":ESM.usgs_event_id, + "INGV_ev_id":ESM.ingv_event_id, + "EMSC_ev_id":ESM.emsc_event_id, + "ev_nation_code":ESM.ev_nation_code, + "ev_latitude":ESM.ev_latitude, + "ev_longitude":ESM.ev_longitude, + "ev_depth_km":ESM.ev_depth_km, "ev_hyp_ref":default_string, - "fm_type_code":r_fm_type, - "ML":ESM23.ml, - "ML_ref":ESM23.ml_ref, - "Mw":ESM23.mw, - "Mw_ref":ESM23.mw_ref, - "Ms":ESM23.ms, - "Ms_ref":ESM23.ms_ref, - "EMEC_Mw":ESM23.emec_mw, - "EMEC_Mw_type":ESM23.emec_mw_type, - "EMEC_Mw_ref":ESM23.emec_mw_ref, - "event_source_id":ESM23.event_source_id, - - "es_strike":ESM23.es_strike, - "es_dip":ESM23.es_dip, - "es_rake":ESM23.es_rake, + "fm_type_code":ESM.fm_type_code, + "ML":ESM.ml, + "ML_ref":ESM.ml_ref, + "Mw":ESM.mw, + "Mw_ref":ESM.mw_ref, + "Ms":ESM.ms, + "Ms_ref":ESM.ms_ref, + "EMEC_Mw":ESM.emec_mw, + "EMEC_Mw_type":ESM.emec_mw_type, + "EMEC_Mw_ref":ESM.emec_mw_ref, + "event_source_id":ESM.event_source_id, + + "es_strike":ESM.es_strike, + "es_dip":ESM.es_dip, + "es_rake":ESM.es_rake, "es_strike_dip_rake_ref":default_string, - "es_z_top":ESM23.z_top, - "es_z_top_ref":ESM23.es_z_top_ref, - "es_length":ESM23.es_length, - "es_width":ESM23.es_width, - "es_geometry_ref":ESM23.es_geometry_ref, + "es_z_top":ESM.z_top, + "es_z_top_ref":ESM.es_z_top_ref, + "es_length":ESM.es_length, + "es_width":ESM.es_width, + "es_geometry_ref":ESM.es_geometry_ref, - "network_code":ESM23.network_code, - "station_code":ESM23.station_code, - "location_code":ESM23.location_code, - "instrument_code":ESM23.instrument_type_code, - "sensor_depth_m":ESM23.sensor_depth_m, - "proximity_code":ESM23.proximity, - "housing_code":ESM23.hounsing, # Currently typo in their database header - "installation_code":ESM23.installation, - "st_nation_code":ESM23.st_nation_code, - "st_latitude":ESM23.st_latitude, - "st_longitude":ESM23.st_longitude, - "st_elevation":ESM23.st_elevation, + "network_code":ESM.network_code, + "station_code":ESM.station_code, + "location_code":ESM.location_code, + "instrument_code":ESM.instrument_type_code, + "sensor_depth_m":ESM.sensor_depth_m, + "proximity_code":ESM.proximity, + "housing_code":ESM.hounsing, # Currently typo in their database header + "installation_code":ESM.installation, + "st_nation_code":ESM.st_nation_code, + "st_latitude":ESM.st_latitude, + "st_longitude":ESM.st_longitude, + "st_elevation":ESM.st_elevation, - "ec8_code":ESM23.ec8_code, + "ec8_code":ESM.ec8_code, "ec8_code_method":default_string, "ec8_code_ref":default_string, - "vs30_m_sec":ESM23.vs30_m_s, + "vs30_m_sec":ESM.vs30_m_s, "vs30_ref":default_string, "vs30_calc_method":default_string, - "vs30_meas_type":ESM23.vs30_meas_type, - "slope_deg":ESM23.slope_deg, - "vs30_m_sec_WA":ESM23.vs30_m_s_wa, + "vs30_meas_type":ESM.vs30_meas_type, + "slope_deg":ESM.slope_deg, + "vs30_m_sec_WA":ESM.vs30_m_s_wa, - "epi_dist":ESM23.epi_dist, - "epi_az":ESM23.epi_az, - "JB_dist":ESM23.jb_dist, - "rup_dist":ESM23.rup_dist, - "Rx_dist":ESM23.rx_dist, - "Ry0_dist":ESM23.ry0_dist, + "epi_dist":ESM.epi_dist, + "epi_az":ESM.epi_az, + "JB_dist":ESM.jb_dist, + "rup_dist":ESM.rup_dist, + "Rx_dist":ESM.rx_dist, + "Ry0_dist":ESM.ry0_dist, - "instrument_type_code":ESM23.instrument_type_code, - "late_triggered_flag_01":ESM23.late_triggered_event_01, - "U_channel_code":ESM23.u_channel_code, - "U_azimuth_deg":ESM23.u_azimuth_deg, - "V_channel_code":ESM23.v_channel_code, - "V_azimuth_deg":ESM23.v_azimuth_deg, - "W_channel_code":ESM23.w_channel_code, + "instrument_type_code":ESM.instrument_type_code, + "late_triggered_flag_01":ESM.late_triggered_event_01, + "U_channel_code":ESM.u_channel_code, + "U_azimuth_deg":ESM.u_azimuth_deg, + "V_channel_code":ESM.v_channel_code, + "V_azimuth_deg":ESM.v_azimuth_deg, + "W_channel_code":ESM.w_channel_code, - "U_hp":ESM23.u_hp, - "V_hp":ESM23.v_hp, - "W_hp":ESM23.w_hp, - "U_lp":ESM23.u_lp, - "V_lp":ESM23.v_lp, - "W_lp":ESM23.w_lp, + "U_hp":ESM.u_hp, + "V_hp":ESM.v_hp, + "W_hp":ESM.w_hp, + "U_lp":ESM.u_lp, + "V_lp":ESM.v_lp, + "W_lp":ESM.w_lp, - "U_pga":ESM23.u_pga, - "V_pga":ESM23.v_pga, - "W_pga":ESM23.w_pga, - "rotD50_pga":ESM23.rotd50_pga, - "rotD100_pga":ESM23.rotd100_pga, - "rotD00_pga":ESM23.rotd00_pga, - "U_pgv":ESM23.u_pgv, - "V_pgv":ESM23.v_pgv, - "W_pgv":ESM23.w_pgv, - "rotD50_pgv":ESM23.rotd50_pgv, - "rotD100_pgv":ESM23.rotd100_pgv, - "rotD00_pgv":ESM23.rotd00_pgv, - "U_pgd":ESM23.u_pgd, - "V_pgd":ESM23.v_pgd, - "W_pgd":ESM23.w_pgd, - "rotD50_pgd":ESM23.rotd50_pgd, - "rotD100_pgd":ESM23.rotd100_pgd, - "rotD00_pgd":ESM23.rotd00_pgv, - "U_T90":ESM23.u_t90, - "V_T90":ESM23.v_t90, - "W_T90":ESM23.w_t90, - "rotD50_T90":ESM23.rotd50_t90, - "rotD100_T90":ESM23.rotd100_t90, - "rotD00_T90":ESM23.rot_d00_t90, # This header has typo in current db version - "U_housner":ESM23.u_housner, - "V_housner":ESM23.v_housner, - "W_housner":ESM23.w_housner, - "rotD50_housner":ESM23.rotd50_housner, - "rotD100_housner":ESM23.rotd100_housner, - "rotD00_housner":ESM23.rotd00_housner, - "U_CAV":ESM23.u_cav, - "V_CAV":ESM23.v_cav, - "W_CAV":ESM23.w_cav, - "rotD50_CAV":ESM23.rotd50_cav, - "rotD100_CAV":ESM23.rotd100_cav, - "rotD00_CAV":ESM23.rotd00_cav, - "U_ia":ESM23.u_ia, - "V_ia":ESM23.v_ia, - "W_ia":ESM23.w_ia, - "rotD50_ia":ESM23.rotd50_ia, - "rotD100_ia":ESM23.rotd100_ia, - "rotD00_ia":ESM23.rotd00_ia, + "U_pga":ESM.u_pga, + "V_pga":ESM.v_pga, + "W_pga":ESM.w_pga, + "rotD50_pga":ESM.rotd50_pga, + "rotD100_pga":ESM.rotd100_pga, + "rotD00_pga":ESM.rotd00_pga, + "U_pgv":ESM.u_pgv, + "V_pgv":ESM.v_pgv, + "W_pgv":ESM.w_pgv, + "rotD50_pgv":ESM.rotd50_pgv, + "rotD100_pgv":ESM.rotd100_pgv, + "rotD00_pgv":ESM.rotd00_pgv, + "U_pgd":ESM.u_pgd, + "V_pgd":ESM.v_pgd, + "W_pgd":ESM.w_pgd, + "rotD50_pgd":ESM.rotd50_pgd, + "rotD100_pgd":ESM.rotd100_pgd, + "rotD00_pgd":ESM.rotd00_pgv, + "U_T90":ESM.u_t90, + "V_T90":ESM.v_t90, + "W_T90":ESM.w_t90, + "rotD50_T90":ESM.rotd50_t90, + "rotD100_T90":ESM.rotd100_t90, + "rotD00_T90":ESM.rot_d00_t90, # This header has typo in current db version + "U_housner":ESM.u_housner, + "V_housner":ESM.v_housner, + "W_housner":ESM.w_housner, + "rotD50_housner":ESM.rotd50_housner, + "rotD100_housner":ESM.rotd100_housner, + "rotD00_housner":ESM.rotd00_housner, + "U_CAV":ESM.u_cav, + "V_CAV":ESM.v_cav, + "W_CAV":ESM.w_cav, + "rotD50_CAV":ESM.rotd50_cav, + "rotD100_CAV":ESM.rotd100_cav, + "rotD00_CAV":ESM.rotd00_cav, + "U_ia":ESM.u_ia, + "V_ia":ESM.v_ia, + "W_ia":ESM.w_ia, + "rotD50_ia":ESM.rotd50_ia, + "rotD100_ia":ESM.rotd100_ia, + "rotD00_ia":ESM.rotd00_ia, - "U_T0_010":ESM23.u_t0_010, - "U_T0_025":ESM23.u_t0_025, - "U_T0_040":ESM23.u_t0_040, - "U_T0_050":ESM23.u_t0_050, - "U_T0_070":ESM23.u_t0_070, - "U_T0_100":ESM23.u_t0_100, - "U_T0_150":ESM23.u_t0_150, - "U_T0_200":ESM23.u_t0_200, - "U_T0_250":ESM23.u_t0_250, - "U_T0_300":ESM23.u_t0_300, - "U_T0_350":ESM23.u_t0_350, - "U_T0_400":ESM23.u_t0_400, - "U_T0_450":ESM23.u_t0_450, - "U_T0_500":ESM23.u_t0_500, - "U_T0_600":ESM23.u_t0_600, - "U_T0_700":ESM23.u_t0_700, - "U_T0_750":ESM23.u_t0_750, - "U_T0_800":ESM23.u_t0_800, - "U_T0_900":ESM23.u_t0_900, - "U_T1_000":ESM23.u_t1_000, - "U_T1_200":ESM23.u_t1_200, - "U_T1_400":ESM23.u_t1_400, - "U_T1_600":ESM23.u_t1_600, - "U_T1_800":ESM23.u_t1_800, - "U_T2_000":ESM23.u_t2_000, - "U_T2_500":ESM23.u_t2_500, - "U_T3_000":ESM23.u_t3_000, - "U_T3_500":ESM23.u_t3_500, - "U_T4_000":ESM23.u_t4_000, - "U_T4_500":ESM23.u_t4_500, - "U_T5_000":ESM23.u_t5_000, - "U_T6_000":ESM23.u_t6_000, - "U_T7_000":ESM23.u_t7_000, - "U_T8_000":ESM23.u_t8_000, - "U_T9_000":ESM23.u_t9_000, - "U_T10_000":ESM23.u_t10_000, + "U_T0_010":ESM.u_t0_010, + "U_T0_025":ESM.u_t0_025, + "U_T0_040":ESM.u_t0_040, + "U_T0_050":ESM.u_t0_050, + "U_T0_070":ESM.u_t0_070, + "U_T0_100":ESM.u_t0_100, + "U_T0_150":ESM.u_t0_150, + "U_T0_200":ESM.u_t0_200, + "U_T0_250":ESM.u_t0_250, + "U_T0_300":ESM.u_t0_300, + "U_T0_350":ESM.u_t0_350, + "U_T0_400":ESM.u_t0_400, + "U_T0_450":ESM.u_t0_450, + "U_T0_500":ESM.u_t0_500, + "U_T0_600":ESM.u_t0_600, + "U_T0_700":ESM.u_t0_700, + "U_T0_750":ESM.u_t0_750, + "U_T0_800":ESM.u_t0_800, + "U_T0_900":ESM.u_t0_900, + "U_T1_000":ESM.u_t1_000, + "U_T1_200":ESM.u_t1_200, + "U_T1_400":ESM.u_t1_400, + "U_T1_600":ESM.u_t1_600, + "U_T1_800":ESM.u_t1_800, + "U_T2_000":ESM.u_t2_000, + "U_T2_500":ESM.u_t2_500, + "U_T3_000":ESM.u_t3_000, + "U_T3_500":ESM.u_t3_500, + "U_T4_000":ESM.u_t4_000, + "U_T4_500":ESM.u_t4_500, + "U_T5_000":ESM.u_t5_000, + "U_T6_000":ESM.u_t6_000, + "U_T7_000":ESM.u_t7_000, + "U_T8_000":ESM.u_t8_000, + "U_T9_000":ESM.u_t9_000, + "U_T10_000":ESM.u_t10_000, - "V_T0_010":ESM23.v_t0_010, - "V_T0_025":ESM23.v_t0_025, - "V_T0_040":ESM23.v_t0_040, - "V_T0_050":ESM23.v_t0_050, - "V_T0_070":ESM23.v_t0_070, - "V_T0_100":ESM23.v_t0_100, - "V_T0_150":ESM23.v_t0_150, - "V_T0_200":ESM23.v_t0_200, - "V_T0_250":ESM23.v_t0_250, - "V_T0_300":ESM23.v_t0_300, - "V_T0_350":ESM23.v_t0_350, - "V_T0_400":ESM23.v_t0_400, - "V_T0_450":ESM23.v_t0_450, - "V_T0_500":ESM23.v_t0_500, - "V_T0_600":ESM23.v_t0_600, - "V_T0_700":ESM23.v_t0_700, - "V_T0_750":ESM23.v_t0_750, - "V_T0_800":ESM23.v_t0_800, - "V_T0_900":ESM23.v_t0_900, - "V_T1_000":ESM23.v_t1_000, - "V_T1_200":ESM23.v_t1_200, - "V_T1_400":ESM23.v_t1_400, - "V_T1_600":ESM23.v_t1_600, - "V_T1_800":ESM23.v_t1_800, - "V_T2_000":ESM23.v_t2_000, - "V_T2_500":ESM23.v_t2_500, - "V_T3_000":ESM23.v_t3_000, - "V_T3_500":ESM23.v_t3_500, - "V_T4_000":ESM23.v_t4_000, - "V_T4_500":ESM23.v_t4_500, - "V_T5_000":ESM23.v_t5_000, - "V_T6_000":ESM23.v_t6_000, - "V_T7_000":ESM23.v_t7_000, - "V_T8_000":ESM23.v_t8_000, - "V_T9_000":ESM23.v_t9_000, - "V_T10_000":ESM23.v_t10_000, + "V_T0_010":ESM.v_t0_010, + "V_T0_025":ESM.v_t0_025, + "V_T0_040":ESM.v_t0_040, + "V_T0_050":ESM.v_t0_050, + "V_T0_070":ESM.v_t0_070, + "V_T0_100":ESM.v_t0_100, + "V_T0_150":ESM.v_t0_150, + "V_T0_200":ESM.v_t0_200, + "V_T0_250":ESM.v_t0_250, + "V_T0_300":ESM.v_t0_300, + "V_T0_350":ESM.v_t0_350, + "V_T0_400":ESM.v_t0_400, + "V_T0_450":ESM.v_t0_450, + "V_T0_500":ESM.v_t0_500, + "V_T0_600":ESM.v_t0_600, + "V_T0_700":ESM.v_t0_700, + "V_T0_750":ESM.v_t0_750, + "V_T0_800":ESM.v_t0_800, + "V_T0_900":ESM.v_t0_900, + "V_T1_000":ESM.v_t1_000, + "V_T1_200":ESM.v_t1_200, + "V_T1_400":ESM.v_t1_400, + "V_T1_600":ESM.v_t1_600, + "V_T1_800":ESM.v_t1_800, + "V_T2_000":ESM.v_t2_000, + "V_T2_500":ESM.v_t2_500, + "V_T3_000":ESM.v_t3_000, + "V_T3_500":ESM.v_t3_500, + "V_T4_000":ESM.v_t4_000, + "V_T4_500":ESM.v_t4_500, + "V_T5_000":ESM.v_t5_000, + "V_T6_000":ESM.v_t6_000, + "V_T7_000":ESM.v_t7_000, + "V_T8_000":ESM.v_t8_000, + "V_T9_000":ESM.v_t9_000, + "V_T10_000":ESM.v_t10_000, - "W_T0_010":ESM23.w_t0_010, - "W_T0_025":ESM23.w_t0_025, - "W_T0_040":ESM23.w_t0_040, - "W_T0_050":ESM23.w_t0_050, - "W_T0_070":ESM23.w_t0_070, - "W_T0_100":ESM23.w_t0_100, - "W_T0_150":ESM23.w_t0_150, - "W_T0_200":ESM23.w_t0_200, - "W_T0_250":ESM23.w_t0_250, - "W_T0_300":ESM23.w_t0_300, - "W_T0_350":ESM23.w_t0_350, - "W_T0_400":ESM23.w_t0_400, - "W_T0_450":ESM23.w_t0_450, - "W_T0_500":ESM23.w_t0_500, - "W_T0_600":ESM23.w_t0_600, - "W_T0_700":ESM23.w_t0_700, - "W_T0_750":ESM23.w_t0_750, - "W_T0_800":ESM23.w_t0_800, - "W_T0_900":ESM23.w_t0_900, - "W_T1_000":ESM23.w_t1_000, - "W_T1_200":ESM23.w_t1_200, - "W_T1_400":ESM23.w_t1_400, - "W_T1_600":ESM23.w_t1_600, - "W_T1_800":ESM23.w_t1_800, - "W_T2_000":ESM23.w_t2_000, - "W_T2_500":ESM23.w_t2_500, - "W_T3_000":ESM23.w_t3_000, - "W_T3_500":ESM23.w_t3_500, - "W_T4_000":ESM23.w_t4_000, - "W_T4_500":ESM23.w_t4_500, - "W_T5_000":ESM23.w_t5_000, - "W_T6_000":ESM23.w_t6_000, - "W_T7_000":ESM23.w_t7_000, - "W_T8_000":ESM23.w_t8_000, - "W_T9_000":ESM23.w_t9_000, - "W_T10_000":ESM23.w_t10_000, + "W_T0_010":ESM.w_t0_010, + "W_T0_025":ESM.w_t0_025, + "W_T0_040":ESM.w_t0_040, + "W_T0_050":ESM.w_t0_050, + "W_T0_070":ESM.w_t0_070, + "W_T0_100":ESM.w_t0_100, + "W_T0_150":ESM.w_t0_150, + "W_T0_200":ESM.w_t0_200, + "W_T0_250":ESM.w_t0_250, + "W_T0_300":ESM.w_t0_300, + "W_T0_350":ESM.w_t0_350, + "W_T0_400":ESM.w_t0_400, + "W_T0_450":ESM.w_t0_450, + "W_T0_500":ESM.w_t0_500, + "W_T0_600":ESM.w_t0_600, + "W_T0_700":ESM.w_t0_700, + "W_T0_750":ESM.w_t0_750, + "W_T0_800":ESM.w_t0_800, + "W_T0_900":ESM.w_t0_900, + "W_T1_000":ESM.w_t1_000, + "W_T1_200":ESM.w_t1_200, + "W_T1_400":ESM.w_t1_400, + "W_T1_600":ESM.w_t1_600, + "W_T1_800":ESM.w_t1_800, + "W_T2_000":ESM.w_t2_000, + "W_T2_500":ESM.w_t2_500, + "W_T3_000":ESM.w_t3_000, + "W_T3_500":ESM.w_t3_500, + "W_T4_000":ESM.w_t4_000, + "W_T4_500":ESM.w_t4_500, + "W_T5_000":ESM.w_t5_000, + "W_T6_000":ESM.w_t6_000, + "W_T7_000":ESM.w_t7_000, + "W_T8_000":ESM.w_t8_000, + "W_T9_000":ESM.w_t9_000, + "W_T10_000":ESM.w_t10_000, - "rotD50_T0_010":ESM23.rotd50_t0_010, - "rotD50_T0_025":ESM23.rotd50_t0_025, - "rotD50_T0_040":ESM23.rotd50_t0_040, - "rotD50_T0_050":ESM23.rotd50_t0_050, - "rotD50_T0_070":ESM23.rotd50_t0_070, - "rotD50_T0_100":ESM23.rotd50_t0_100, - "rotD50_T0_150":ESM23.rotd50_t0_150, - "rotD50_T0_200":ESM23.rotd50_t0_200, - "rotD50_T0_250":ESM23.rotd50_t0_250, - "rotD50_T0_300":ESM23.rotd50_t0_300, - "rotD50_T0_350":ESM23.rotd50_t0_350, - "rotD50_T0_400":ESM23.rotd50_t0_400, - "rotD50_T0_450":ESM23.rotd50_t0_450, - "rotD50_T0_500":ESM23.rotd50_t0_500, - "rotD50_T0_600":ESM23.rotd50_t0_600, - "rotD50_T0_700":ESM23.rotd50_t0_700, - "rotD50_T0_750":ESM23.rotd50_t0_750, - "rotD50_T0_800":ESM23.rotd50_t0_800, - "rotD50_T0_900":ESM23.rotd50_t0_900, - "rotD50_T1_000":ESM23.rotd50_t1_000, - "rotD50_T1_200":ESM23.rotd50_t1_200, - "rotD50_T1_400":ESM23.rotd50_t1_400, - "rotD50_T1_600":ESM23.rotd50_t1_600, - "rotD50_T1_800":ESM23.rotd50_t1_800, - "rotD50_T2_000":ESM23.rotd50_t2_000, - "rotD50_T2_500":ESM23.rotd50_t2_500, - "rotD50_T3_000":ESM23.rotd50_t3_000, - "rotD50_T3_500":ESM23.rotd50_t3_500, - "rotD50_T4_000":ESM23.rotd50_t4_000, - "rotD50_T4_500":ESM23.rotd50_t4_500, - "rotD50_T5_000":ESM23.rotd50_t5_000, - "rotD50_T6_000":ESM23.rotd50_t6_000, - "rotD50_T7_000":ESM23.rotd50_t7_000, - "rotD50_T8_000":ESM23.rotd50_t8_000, - "rotD50_T9_000":ESM23.rotd50_t9_000, - "rotD50_T10_000":ESM23.rotd50_t10_000, + "rotD50_T0_010":ESM.rotd50_t0_010, + "rotD50_T0_025":ESM.rotd50_t0_025, + "rotD50_T0_040":ESM.rotd50_t0_040, + "rotD50_T0_050":ESM.rotd50_t0_050, + "rotD50_T0_070":ESM.rotd50_t0_070, + "rotD50_T0_100":ESM.rotd50_t0_100, + "rotD50_T0_150":ESM.rotd50_t0_150, + "rotD50_T0_200":ESM.rotd50_t0_200, + "rotD50_T0_250":ESM.rotd50_t0_250, + "rotD50_T0_300":ESM.rotd50_t0_300, + "rotD50_T0_350":ESM.rotd50_t0_350, + "rotD50_T0_400":ESM.rotd50_t0_400, + "rotD50_T0_450":ESM.rotd50_t0_450, + "rotD50_T0_500":ESM.rotd50_t0_500, + "rotD50_T0_600":ESM.rotd50_t0_600, + "rotD50_T0_700":ESM.rotd50_t0_700, + "rotD50_T0_750":ESM.rotd50_t0_750, + "rotD50_T0_800":ESM.rotd50_t0_800, + "rotD50_T0_900":ESM.rotd50_t0_900, + "rotD50_T1_000":ESM.rotd50_t1_000, + "rotD50_T1_200":ESM.rotd50_t1_200, + "rotD50_T1_400":ESM.rotd50_t1_400, + "rotD50_T1_600":ESM.rotd50_t1_600, + "rotD50_T1_800":ESM.rotd50_t1_800, + "rotD50_T2_000":ESM.rotd50_t2_000, + "rotD50_T2_500":ESM.rotd50_t2_500, + "rotD50_T3_000":ESM.rotd50_t3_000, + "rotD50_T3_500":ESM.rotd50_t3_500, + "rotD50_T4_000":ESM.rotd50_t4_000, + "rotD50_T4_500":ESM.rotd50_t4_500, + "rotD50_T5_000":ESM.rotd50_t5_000, + "rotD50_T6_000":ESM.rotd50_t6_000, + "rotD50_T7_000":ESM.rotd50_t7_000, + "rotD50_T8_000":ESM.rotd50_t8_000, + "rotD50_T9_000":ESM.rotd50_t9_000, + "rotD50_T10_000":ESM.rotd50_t10_000, - "rotD100_T0_010":ESM23.rotd100_t0_010, - "rotD100_T0_025":ESM23.rotd100_t0_025, - "rotD100_T0_040":ESM23.rotd100_t0_040, - "rotD100_T0_050":ESM23.rotd100_t0_050, - "rotD100_T0_070":ESM23.rotd100_t0_070, - "rotD100_T0_100":ESM23.rotd100_t0_100, - "rotD100_T0_150":ESM23.rotd100_t0_150, - "rotD100_T0_200":ESM23.rotd100_t0_200, - "rotD100_T0_250":ESM23.rotd100_t0_250, - "rotD100_T0_300":ESM23.rotd100_t0_300, - "rotD100_T0_350":ESM23.rotd100_t0_350, - "rotD100_T0_400":ESM23.rotd100_t0_400, - "rotD100_T0_450":ESM23.rotd100_t0_450, - "rotD100_T0_500":ESM23.rotd100_t0_500, - "rotD100_T0_600":ESM23.rotd100_t0_600, - "rotD100_T0_700":ESM23.rotd100_t0_700, - "rotD100_T0_750":ESM23.rotd100_t0_750, - "rotD100_T0_800":ESM23.rotd100_t0_800, - "rotD100_T0_900":ESM23.rotd100_t0_900, - "rotD100_T1_000":ESM23.rotd100_t1_000, - "rotD100_T1_200":ESM23.rotd100_t1_200, - "rotD100_T1_400":ESM23.rotd100_t1_400, - "rotD100_T1_600":ESM23.rotd100_t1_600, - "rotD100_T1_800":ESM23.rotd100_t1_800, - "rotD100_T2_000":ESM23.rotd100_t2_000, - "rotD100_T2_500":ESM23.rotd100_t2_500, - "rotD100_T3_000":ESM23.rotd100_t3_000, - "rotD100_T3_500":ESM23.rotd100_t3_500, - "rotD100_T4_000":ESM23.rotd100_t4_000, - "rotD100_T4_500":ESM23.rotd100_t4_500, - "rotD100_T5_000":ESM23.rotd100_t5_000, - "rotD100_T6_000":ESM23.rotd100_t6_000, - "rotD100_T7_000":ESM23.rotd100_t7_000, - "rotD100_T8_000":ESM23.rotd100_t8_000, - "rotD100_T9_000":ESM23.rotd100_t9_000, - "rotD100_T10_000":ESM23.rotd100_t10_000, + "rotD100_T0_010":ESM.rotd100_t0_010, + "rotD100_T0_025":ESM.rotd100_t0_025, + "rotD100_T0_040":ESM.rotd100_t0_040, + "rotD100_T0_050":ESM.rotd100_t0_050, + "rotD100_T0_070":ESM.rotd100_t0_070, + "rotD100_T0_100":ESM.rotd100_t0_100, + "rotD100_T0_150":ESM.rotd100_t0_150, + "rotD100_T0_200":ESM.rotd100_t0_200, + "rotD100_T0_250":ESM.rotd100_t0_250, + "rotD100_T0_300":ESM.rotd100_t0_300, + "rotD100_T0_350":ESM.rotd100_t0_350, + "rotD100_T0_400":ESM.rotd100_t0_400, + "rotD100_T0_450":ESM.rotd100_t0_450, + "rotD100_T0_500":ESM.rotd100_t0_500, + "rotD100_T0_600":ESM.rotd100_t0_600, + "rotD100_T0_700":ESM.rotd100_t0_700, + "rotD100_T0_750":ESM.rotd100_t0_750, + "rotD100_T0_800":ESM.rotd100_t0_800, + "rotD100_T0_900":ESM.rotd100_t0_900, + "rotD100_T1_000":ESM.rotd100_t1_000, + "rotD100_T1_200":ESM.rotd100_t1_200, + "rotD100_T1_400":ESM.rotd100_t1_400, + "rotD100_T1_600":ESM.rotd100_t1_600, + "rotD100_T1_800":ESM.rotd100_t1_800, + "rotD100_T2_000":ESM.rotd100_t2_000, + "rotD100_T2_500":ESM.rotd100_t2_500, + "rotD100_T3_000":ESM.rotd100_t3_000, + "rotD100_T3_500":ESM.rotd100_t3_500, + "rotD100_T4_000":ESM.rotd100_t4_000, + "rotD100_T4_500":ESM.rotd100_t4_500, + "rotD100_T5_000":ESM.rotd100_t5_000, + "rotD100_T6_000":ESM.rotd100_t6_000, + "rotD100_T7_000":ESM.rotd100_t7_000, + "rotD100_T8_000":ESM.rotd100_t8_000, + "rotD100_T9_000":ESM.rotd100_t9_000, + "rotD100_T10_000":ESM.rotd100_t10_000, - "rotD00_T0_010":ESM23.rotd00_t0_010, - "rotD00_T0_025":ESM23.rotd00_t0_025, - "rotD00_T0_040":ESM23.rotd00_t0_040, - "rotD00_T0_050":ESM23.rotd00_t0_050, - "rotD00_T0_070":ESM23.rotd00_t0_070, - "rotD00_T0_100":ESM23.rotd00_t0_100, - "rotD00_T0_150":ESM23.rotd00_t0_150, - "rotD00_T0_200":ESM23.rotd00_t0_200, - "rotD00_T0_250":ESM23.rotd00_t0_250, - "rotD00_T0_300":ESM23.rotd00_t0_300, - "rotD00_T0_350":ESM23.rotd00_t0_350, - "rotD00_T0_400":ESM23.rotd00_t0_400, - "rotD00_T0_450":ESM23.rotd00_t0_450, - "rotD00_T0_500":ESM23.rotd00_t0_500, - "rotD00_T0_600":ESM23.rotd00_t0_600, - "rotD00_T0_700":ESM23.rotd00_t0_700, - "rotD00_T0_750":ESM23.rotd00_t0_750, - "rotD00_T0_800":ESM23.rotd00_t0_800, - "rotD00_T0_900":ESM23.rotd00_t0_900, - "rotD00_T1_000":ESM23.rotd00_t1_000, - "rotD00_T1_200":ESM23.rotd00_t1_200, - "rotD00_T1_400":ESM23.rotd00_t1_400, - "rotD00_T1_600":ESM23.rotd00_t1_600, - "rotD00_T1_800":ESM23.rotd00_t1_800, - "rotD00_T2_000":ESM23.rotd00_t2_000, - "rotD00_T2_500":ESM23.rotd00_t2_500, - "rotD00_T3_000":ESM23.rotd00_t3_000, - "rotD00_T3_500":ESM23.rotd00_t3_500, - "rotD00_T4_000":ESM23.rotd00_t4_000, - "rotD00_T4_500":ESM23.rotd00_t4_500, - "rotD00_T5_000":ESM23.rotd00_t5_000, - "rotD00_T6_000":ESM23.rotd00_t6_000, - "rotD00_T7_000":ESM23.rotd00_t7_000, - "rotD00_T8_000":ESM23.rotd00_t8_000, - "rotD00_T9_000":ESM23.rotd00_t9_000, - "rotD00_T10_000":ESM23.rotd00_t10_000}) + "rotD00_T0_010":ESM.rotd00_t0_010, + "rotD00_T0_025":ESM.rotd00_t0_025, + "rotD00_T0_040":ESM.rotd00_t0_040, + "rotD00_T0_050":ESM.rotd00_t0_050, + "rotD00_T0_070":ESM.rotd00_t0_070, + "rotD00_T0_100":ESM.rotd00_t0_100, + "rotD00_T0_150":ESM.rotd00_t0_150, + "rotD00_T0_200":ESM.rotd00_t0_200, + "rotD00_T0_250":ESM.rotd00_t0_250, + "rotD00_T0_300":ESM.rotd00_t0_300, + "rotD00_T0_350":ESM.rotd00_t0_350, + "rotD00_T0_400":ESM.rotd00_t0_400, + "rotD00_T0_450":ESM.rotd00_t0_450, + "rotD00_T0_500":ESM.rotd00_t0_500, + "rotD00_T0_600":ESM.rotd00_t0_600, + "rotD00_T0_700":ESM.rotd00_t0_700, + "rotD00_T0_750":ESM.rotd00_t0_750, + "rotD00_T0_800":ESM.rotd00_t0_800, + "rotD00_T0_900":ESM.rotd00_t0_900, + "rotD00_T1_000":ESM.rotd00_t1_000, + "rotD00_T1_200":ESM.rotd00_t1_200, + "rotD00_T1_400":ESM.rotd00_t1_400, + "rotD00_T1_600":ESM.rotd00_t1_600, + "rotD00_T1_800":ESM.rotd00_t1_800, + "rotD00_T2_000":ESM.rotd00_t2_000, + "rotD00_T2_500":ESM.rotd00_t2_500, + "rotD00_T3_000":ESM.rotd00_t3_000, + "rotD00_T3_500":ESM.rotd00_t3_500, + "rotD00_T4_000":ESM.rotd00_t4_000, + "rotD00_T4_500":ESM.rotd00_t4_500, + "rotD00_T5_000":ESM.rotd00_t5_000, + "rotD00_T6_000":ESM.rotd00_t6_000, + "rotD00_T7_000":ESM.rotd00_t7_000, + "rotD00_T8_000":ESM.rotd00_t8_000, + "rotD00_T9_000":ESM.rotd00_t9_000, + "rotD00_T10_000":ESM.rotd00_t10_000}) - # Output to folder where converted flatfile read into parser - DATA = os.path.abspath('') + # Output to folder where converted flatfile read into parser tmp = tempfile.mkdtemp() - converted_base_data_path = os.path.join(DATA, tmp, - 'converted_flatfile.csv') + converted_base_data_path = os.path.join(DATA, tmp, 'converted_flatfile.csv') ESM_original_headers.to_csv(converted_base_data_path, sep=';') return converted_base_data_path \ No newline at end of file diff --git a/openquake/smt/parsers/esm22_flatfile_parser.py b/openquake/smt/parsers/esm_ws_flatfile_parser.py similarity index 75% rename from openquake/smt/parsers/esm22_flatfile_parser.py rename to openquake/smt/parsers/esm_ws_flatfile_parser.py index 719e08d1e..8d02dabdb 100644 --- a/openquake/smt/parsers/esm22_flatfile_parser.py +++ b/openquake/smt/parsers/esm_ws_flatfile_parser.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,8 +16,8 @@ # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . """ -Parser for the ESM22 flatfile format (i.e. flatfile downloaded from web -service) to SMT +Parser for a flatfile downloaded from the ESM web service +--> https://esm-db.eu/#/waveform/search """ import pandas as pd import os, sys @@ -45,6 +45,9 @@ # Import the ESM dictionaries from .esm_dictionaries import * +# Define base path +DATA = os.path.abspath('') + SCALAR_LIST = ["PGA", "PGV", "PGD", "CAV", "CAV5", "Ia", "D5-95"] HEADER_STR = "event_id;event_time;ISC_ev_id;USGS_ev_id;INGV_ev_id;"\ @@ -65,33 +68,17 @@ HEADERS = set(HEADER_STR.split(";")) -COUNTRY_CODES = {"AL": "Albania", "AM": "Armenia", "AT": "Austria", - "AZ": "Azerbaijan", "BA": "Bosnia and Herzegowina", - "BG": "Bulgaria", "CH": "Switzerland", "CY": "Cyprus", - "CZ": "Czech Republic", "DE": "Germany", "DZ": "Algeria", - "ES": "Spain", "FR": "France", "GE": "Georgia", - "GR": "Greece", "HR": "Croatia", "HU": "Hungary", - "IL": "Israel", "IR": "Iran", "IS": "Iceland", "IT": "Italy", - "JO": "Jordan", "LI": "Lichtenstein", "MA": "Morocco", - "MC": "Monaco", "MD": "Moldova", "ME": "Montenegro", - "MK": "Macedonia", "MT": "Malta", "PL": "Poland", - "PT": "Portugal", "RO": "Romania", "RS": "Serbia", - "RU": "Russia", "SI": "Slovenia", "SM": "San Marino", - "SY": "Syria", "TM": "Turkmenistan", "TR": "Turkey", - "UA": "Ukraine", "UZ": "Uzbekistan", "XK": "Kosovo"} - - -class ESM22FlatfileParser(SMDatabaseReader): - + +class ESMFlatfileParserWS(SMDatabaseReader): """ - Parses the ESM metadata from the flatfile to a set of metadata objects + Parses the data from the flatfile to a set of metadata objects """ - M_PRECEDENCE = ["EMEC_Mw", "Mw", "Ms", "ML"] BUILD_FINITE_DISTANCES = False def parse(self, location='./'): """ + Parse the flatfile """ assert os.path.isfile(self.filename) headers = getline(self.filename, 1).rstrip("\n").split(";") @@ -101,24 +88,22 @@ def parse(self, location='./'): % hdr) # Read in csv reader = csv.DictReader(open(self.filename, "r"), delimiter=";") - metadata = [] self.database = GroundMotionDatabase(self.id, self.name) counter = 0 for row in reader: - if self._sanitise(row, reader): - # Build the metadata - record = self._parse_record(row) - if record: - # Parse the strong motion - record = self._parse_ground_motion( - os.path.join(location, "records"), - row, record, headers) - self.database.records.append(record) + # Build the metadata + record = self._parse_record(row) + if record: + # Parse the strong motion + record = self._parse_ground_motion( + os.path.join(location, "records"), + row, record, headers) + self.database.records.append(record) - else: - print("Record with sequence number %s is null/invalid" - % "{:s}-{:s}".format(row["event_id"], - row["station_code"])) + else: + print("Record with sequence number %s is null/invalid" + % "{:s}-{:s}".format(row["event_id"], + row["station_code"])) if (counter % 100) == 0: print("Processed record %s - %s" % (str(counter), record.id)) @@ -126,25 +111,15 @@ def parse(self, location='./'): counter += 1 @classmethod - def autobuild(cls, dbid, dbname, output_location, ESM22_flatfile_directory): + def autobuild(cls, dbid, dbname, output_location, ESM_flatfile_directory): """ Quick and dirty full database builder! """ - - # Import ESM 2022 format strong-motion flatfile - ESM22 = pd.read_csv(ESM22_flatfile_directory) #Need to specify ESM22_flatfile_directory and filename + # Import ESM web service format strong-motion flatfile + ESM = pd.read_csv(ESM_flatfile_directory) - # Create default values for headers not considered in ESM22 flatfile format - default_string = pd.Series(np.full(np.size(ESM22.event_id),str(""))) - - # Assign strike-slip to unknown faulting mechanism - r_fm_type = ESM22.fm_type_code.fillna('SS') - - #Reformat datetime - r_datetime = ESM22.event_time.str.replace('T',' ') - - converted_base_data_path=_get_ESM18_headers( - ESM22,default_string,r_fm_type,r_datetime) + # Get path to tmp csv once modified dataframe + converted_base_data_path=_get_ESM18_headers(ESM) if os.path.exists(output_location): raise IOError("Target database directory %s already exists!" @@ -165,12 +140,6 @@ def autobuild(cls, dbid, dbname, output_location, ESM22_flatfile_directory): return database - def _sanitise(self, row, reader): - """ - TODO - Not implemented yet! - """ - return True - def _parse_record(self, metadata): # Waveform ID not provided in file so concatenate Event and Station ID wfid = "_".join([metadata["event_id"], metadata["network_code"], @@ -198,12 +167,7 @@ def _parse_event_data(self, metadata): # ID and Name (name not in file so use ID again) eq_id = metadata["event_id"] eq_name = metadata["event_id"] - # Country - cntry_code = metadata["ev_nation_code"].strip() - if cntry_code and cntry_code in COUNTRY_CODES: - eq_country = COUNTRY_CODES[cntry_code] - else: - eq_country = None + # Date and time eq_datetime = valid.date_time(metadata["event_time"], "%Y-%m-%d %H:%M:%S") @@ -215,7 +179,7 @@ def _parse_event_data(self, metadata): eq_depth = 0.0 eqk = Earthquake(eq_id, eq_name, eq_datetime, eq_lon, eq_lat, eq_depth, None, # Magnitude not defined yet - eq_country=eq_country) + eq_country=None) # Get preferred magnitude and list pref_mag, magnitude_list = self._parse_magnitudes(metadata) eqk.magnitude = pref_mag @@ -364,7 +328,6 @@ def _parse_site_data(self, metadata): network_code = metadata["network_code"].strip() station_code = metadata["station_code"].strip() site_id = "{:s}-{:s}".format(network_code, station_code) - location_code = metadata["location_code"].strip() site_lon = valid.longitude(metadata["st_longitude"]) site_lat = valid.latitude(metadata["st_latitude"]) elevation = valid.vfloat(metadata["st_elevation"], "st_elevation") @@ -378,15 +341,9 @@ def _parse_site_data(self, metadata): vs30_measured = False else: vs30_measured = False - st_nation_code = metadata["st_nation_code"].strip() - if st_nation_code: - st_country = COUNTRY_CODES[st_nation_code] - else: - st_country = None site = RecordSite(site_id, station_code, station_code, site_lon, site_lat, elevation, vs30, vs30_measured, - network_code=network_code, - country=st_country) + network_code=network_code, country=None) site.slope = valid.vfloat(metadata["slope_deg"], "slope_deg") site.sensor_depth = valid.vfloat(metadata["sensor_depth_m"], "sensor_depth_m") @@ -519,7 +476,7 @@ def _parse_ground_motion(self, location, row, record, headers): def _retreive_ground_motion_from_row(self, row, header_list): """ - + Get the ground-motion data from a row (record) in the database """ imts = ["U", "V", "W", "rotD00", "rotD100", "rotD50"] spectra = [] @@ -547,14 +504,12 @@ def _retreive_ground_motion_from_row(self, row, header_list): # Not a spectral period but T90 continue iky = header.replace(key, "").replace("_", ".") - #print imt, key, header, iky periods.append(float(iky)) value = row[header].strip() if value: values.append(np.fabs(float(value))) else: values.append(np.nan) - #values.append(np.fabs(float(row[header].strip()))) periods = np.array(periods) values = np.array(values) idx = np.argsort(periods) @@ -575,40 +530,48 @@ def _retreive_ground_motion_from_row(self, row, header_list): scalars["U"][key] * scalars["V"][key]) return scalars, spectra -def _get_ESM18_headers(ESM22,default_string,r_fm_type,r_datetime): - + +def _get_ESM18_headers(ESM): """ - Convert first from ESM22 format flatfile to ESM18 format flatfile + Convert from ESM web service format flatfile to ESM18 format flatfile """ + # Create default values + default_string = pd.Series(np.full(np.size(ESM.event_id),str(""))) + + #Reformat datetime + r_datetime = ESM.event_time.str.replace('T',' ') + + # Assign unknown to NaN values for faulting mechanism + ESM['fm_type_code'] = ESM.fm_type_code.fillna('U') - #Construct dataframe with original ESM format + # Construct dataframe with original ESM format ESM_original_headers = pd.DataFrame( { - #Non-GMIM headers - "event_id":ESM22.event_id, + # Non-GMIM headers + "event_id":ESM.event_id, "event_time":r_datetime, "ISC_ev_id":default_string, "USGS_ev_id":default_string, "INGV_ev_id":default_string, "EMSC_ev_id":default_string, - "ev_nation_code":ESM22.ev_nation_code, - "ev_latitude":ESM22.ev_latitude, - "ev_longitude":ESM22.ev_longitude, - "ev_depth_km":ESM22.ev_depth_km, + "ev_nation_code":ESM.ev_nation_code, + "ev_latitude":ESM.ev_latitude, + "ev_longitude":ESM.ev_longitude, + "ev_depth_km":ESM.ev_depth_km, "ev_hyp_ref":default_string, - "fm_type_code":r_fm_type, - "ML":ESM22.ML, + "fm_type_code":ESM.fm_type_code, + "ML":ESM.ML, "ML_ref":default_string, - "Mw":ESM22.MW, + "Mw":ESM.MW, "Mw_ref":default_string, "Ms":default_string, "Ms_ref":default_string, - "EMEC_Mw":ESM22.MW, + "EMEC_Mw":ESM.MW, "EMEC_Mw_type":default_string, "EMEC_Mw_ref":default_string, "event_source_id":default_string, - # Rupture plane information not provided in ESM22 format flatfile + # Nodal plane information not provided in raw ESM web service format flatfile "es_strike":default_string, "es_dip":default_string, "es_rake":default_string, @@ -619,78 +582,78 @@ def _get_ESM18_headers(ESM22,default_string,r_fm_type,r_datetime): "es_width":default_string, "es_geometry_ref":default_string, - "network_code":ESM22.network_code, - "station_code":ESM22.station_code, - "location_code":ESM22.location_code, - "instrument_code":ESM22.instrument_type, - "sensor_depth_m":ESM22.sensor_depth_m, - "proximity_code":ESM22.proximity, - "housing_code":ESM22.housing, - "installation_code":ESM22.installation, - "st_nation_code":ESM22.st_nation_code, - "st_latitude":ESM22.st_latitude, - "st_longitude":ESM22.st_longitude, - "st_elevation":ESM22.st_elevation, + "network_code":ESM.network_code, + "station_code":ESM.station_code, + "location_code":ESM.location_code, + "instrument_code":ESM.instrument_type, + "sensor_depth_m":ESM.sensor_depth_m, + "proximity_code":ESM.proximity, + "housing_code":ESM.housing, + "installation_code":ESM.installation, + "st_nation_code":ESM.st_nation_code, + "st_latitude":ESM.st_latitude, + "st_longitude":ESM.st_longitude, + "st_elevation":ESM.st_elevation, - "ec8_code":ESM22.preferred_ec8_code, - "ec8_code_method":ESM22.method_ec8_vs30, + "ec8_code":ESM.preferred_ec8_code, + "ec8_code_method":ESM.method_ec8_vs30, "ec8_code_ref":default_string, - "vs30_m_sec":ESM22.preferred_vs30_m_s, + "vs30_m_sec":ESM.preferred_vs30_m_s, "vs30_ref":default_string, "vs30_calc_method":default_string, "vs30_meas_type":default_string, "slope_deg":default_string, "vs30_m_sec_WA":default_string, - "epi_dist":ESM22.epi_dist, + "epi_dist":ESM.epi_dist, "epi_az":default_string, - "JB_dist":ESM22.JB_dist, - "rup_dist":ESM22.rup_dist, + "JB_dist":ESM.JB_dist, + "rup_dist":ESM.rup_dist, "Rx_dist":default_string, "Ry0_dist":default_string, - "instrument_type_code":ESM22.instrument_type_code, - "late_triggered_flag_01":ESM22.Late_triggered, - "U_channel_code":ESM22.U_channel_code, - "U_azimuth_deg":ESM22.U_azimuth_deg, - "V_channel_code":ESM22.V_channel_code, - "V_azimuth_deg":ESM22.V_azimuth_deg, - "W_channel_code":ESM22.W_channel_code, + "instrument_type_code":ESM.instrument_type_code, + "late_triggered_flag_01":ESM.Late_triggered, + "U_channel_code":ESM.U_channel_code, + "U_azimuth_deg":ESM.U_azimuth_deg, + "V_channel_code":ESM.V_channel_code, + "V_azimuth_deg":ESM.V_azimuth_deg, + "W_channel_code":ESM.W_channel_code, - "U_hp":ESM22.U_hp, - "V_hp":ESM22.V_hp, - "W_hp":ESM22.W_hp, - "U_lp":ESM22.U_lp, - "V_lp":ESM22.V_lp, - "W_lp":ESM22.W_lp, + "U_hp":ESM.U_hp, + "V_hp":ESM.V_hp, + "W_hp":ESM.W_hp, + "U_lp":ESM.U_lp, + "V_lp":ESM.V_lp, + "W_lp":ESM.W_lp, - "U_pga":ESM22.U_pga, - "V_pga":ESM22.V_pga, - "W_pga":ESM22.W_pga, + "U_pga":ESM.U_pga, + "V_pga":ESM.V_pga, + "W_pga":ESM.W_pga, "rotD50_pga":default_string, "rotD100_pga":default_string, "rotD00_pga":default_string, - "U_pgv":ESM22.U_pgv, - "V_pgv":ESM22.V_pgv, - "W_pgv":ESM22.W_pgv, + "U_pgv":ESM.U_pgv, + "V_pgv":ESM.V_pgv, + "W_pgv":ESM.W_pgv, "rotD50_pgv":default_string, "rotD100_pgv":default_string, "rotD00_pgv":default_string, - "U_pgd":ESM22.U_pgd, - "V_pgd":ESM22.V_pgd, - "W_pgd":ESM22.W_pgd, + "U_pgd":ESM.U_pgd, + "V_pgd":ESM.V_pgd, + "W_pgd":ESM.W_pgd, "rotD50_pgd":default_string, "rotD100_pgd":default_string, "rotD00_pgd":default_string, - "U_T90":ESM22.U_T90, - "V_T90":ESM22.V_T90, - "W_T90":ESM22.W_T90, + "U_T90":ESM.U_T90, + "V_T90":ESM.V_T90, + "W_T90":ESM.W_T90, "rotD50_T90":default_string, "rotD100_T90":default_string, "rotD00_T90":default_string, - "U_housner":ESM22.U_housner, - "V_housner":ESM22.V_housner, - "W_housner":ESM22.W_housner, + "U_housner":ESM.U_housner, + "V_housner":ESM.V_housner, + "W_housner":ESM.W_housner, "rotD50_housner":default_string, "rotD100_housner":default_string, "rotD00_housner":default_string, @@ -700,124 +663,125 @@ def _get_ESM18_headers(ESM22,default_string,r_fm_type,r_datetime): "rotD50_CAV":default_string, "rotD100_CAV":default_string, "rotD00_CAV":default_string, - "U_ia":ESM22.U_ia, - "V_ia":ESM22.V_ia, - "W_ia":ESM22.W_ia, - "rotD50_ia":ESM22.U_ia, - "rotD100_ia":ESM22.V_ia, - "rotD00_ia":ESM22.W_ia, + "U_ia":ESM.U_ia, + "V_ia":ESM.V_ia, + "W_ia":ESM.W_ia, + "rotD50_ia":ESM.U_ia, + "rotD100_ia":ESM.V_ia, + "rotD00_ia":ESM.W_ia, - "U_T0_010":ESM22.U_T0_010, - "U_T0_025":ESM22.U_T0_025, - "U_T0_040":ESM22.U_T0_040, - "U_T0_050":ESM22.U_T0_050, - "U_T0_070":ESM22.U_T0_070, - "U_T0_100":ESM22.U_T0_100, - "U_T0_150":ESM22.U_T0_150, - "U_T0_200":ESM22.U_T0_200, - "U_T0_250":ESM22.U_T0_250, - "U_T0_300":ESM22.U_T0_300, - "U_T0_350":ESM22.U_T0_350, - "U_T0_400":ESM22.U_T0_400, - "U_T0_450":ESM22.U_T0_450, - "U_T0_500":ESM22.U_T0_500, - "U_T0_600":ESM22.U_T0_600, - "U_T0_700":ESM22.U_T0_700, - "U_T0_750":ESM22.U_T0_750, - "U_T0_800":ESM22.U_T0_800, - "U_T0_900":ESM22.U_T0_900, - "U_T1_000":ESM22.U_T1_000, - "U_T1_200":ESM22.U_T1_200, - "U_T1_400":ESM22.U_T1_400, - "U_T1_600":ESM22.U_T1_600, - "U_T1_800":ESM22.U_T1_800, - "U_T2_000":ESM22.U_T2_000, - "U_T2_500":ESM22.U_T2_500, - "U_T3_000":ESM22.U_T3_000, - "U_T3_500":ESM22.U_T3_500, - "U_T4_000":ESM22.U_T4_000, - "U_T4_500":ESM22.U_T4_500, - "U_T5_000":ESM22.U_T5_000, - "U_T6_000":ESM22.U_T6_000, - "U_T7_000":ESM22.U_T7_000, - "U_T8_000":ESM22.U_T8_000, - "U_T9_000":ESM22.U_T9_000, - "U_T10_000":ESM22.U_T10_000, + "U_T0_010":ESM.U_T0_010, + "U_T0_025":ESM.U_T0_025, + "U_T0_040":ESM.U_T0_040, + "U_T0_050":ESM.U_T0_050, + "U_T0_070":ESM.U_T0_070, + "U_T0_100":ESM.U_T0_100, + "U_T0_150":ESM.U_T0_150, + "U_T0_200":ESM.U_T0_200, + "U_T0_250":ESM.U_T0_250, + "U_T0_300":ESM.U_T0_300, + "U_T0_350":ESM.U_T0_350, + "U_T0_400":ESM.U_T0_400, + "U_T0_450":ESM.U_T0_450, + "U_T0_500":ESM.U_T0_500, + "U_T0_600":ESM.U_T0_600, + "U_T0_700":ESM.U_T0_700, + "U_T0_750":ESM.U_T0_750, + "U_T0_800":ESM.U_T0_800, + "U_T0_900":ESM.U_T0_900, + "U_T1_000":ESM.U_T1_000, + "U_T1_200":ESM.U_T1_200, + "U_T1_400":ESM.U_T1_400, + "U_T1_600":ESM.U_T1_600, + "U_T1_800":ESM.U_T1_800, + "U_T2_000":ESM.U_T2_000, + "U_T2_500":ESM.U_T2_500, + "U_T3_000":ESM.U_T3_000, + "U_T3_500":ESM.U_T3_500, + "U_T4_000":ESM.U_T4_000, + "U_T4_500":ESM.U_T4_500, + "U_T5_000":ESM.U_T5_000, + "U_T6_000":ESM.U_T6_000, + "U_T7_000":ESM.U_T7_000, + "U_T8_000":ESM.U_T8_000, + "U_T9_000":ESM.U_T9_000, + "U_T10_000":ESM.U_T10_000, - "V_T0_010":ESM22.V_T0_010, - "V_T0_025":ESM22.V_T0_025, - "V_T0_040":ESM22.V_T0_040, - "V_T0_050":ESM22.V_T0_050, - "V_T0_070":ESM22.V_T0_070, - "V_T0_100":ESM22.V_T0_100, - "V_T0_150":ESM22.V_T0_150, - "V_T0_200":ESM22.V_T0_200, - "V_T0_250":ESM22.V_T0_250, - "V_T0_300":ESM22.V_T0_300, - "V_T0_350":ESM22.V_T0_350, - "V_T0_400":ESM22.V_T0_400, - "V_T0_450":ESM22.V_T0_450, - "V_T0_500":ESM22.V_T0_500, - "V_T0_600":ESM22.V_T0_600, - "V_T0_700":ESM22.V_T0_700, - "V_T0_750":ESM22.V_T0_750, - "V_T0_800":ESM22.V_T0_800, - "V_T0_900":ESM22.V_T0_900, - "V_T1_000":ESM22.V_T1_000, - "V_T1_200":ESM22.V_T1_200, - "V_T1_400":ESM22.V_T1_400, - "V_T1_600":ESM22.V_T1_600, - "V_T1_800":ESM22.V_T1_800, - "V_T2_000":ESM22.V_T2_000, - "V_T2_500":ESM22.V_T2_500, - "V_T3_000":ESM22.V_T3_000, - "V_T3_500":ESM22.V_T3_500, - "V_T4_000":ESM22.V_T4_000, - "V_T4_500":ESM22.V_T4_500, - "V_T5_000":ESM22.V_T5_000, - "V_T6_000":ESM22.V_T6_000, - "V_T7_000":ESM22.V_T7_000, - "V_T8_000":ESM22.V_T8_000, - "V_T9_000":ESM22.V_T9_000, - "V_T10_000":ESM22.V_T10_000, + "V_T0_010":ESM.V_T0_010, + "V_T0_025":ESM.V_T0_025, + "V_T0_040":ESM.V_T0_040, + "V_T0_050":ESM.V_T0_050, + "V_T0_070":ESM.V_T0_070, + "V_T0_100":ESM.V_T0_100, + "V_T0_150":ESM.V_T0_150, + "V_T0_200":ESM.V_T0_200, + "V_T0_250":ESM.V_T0_250, + "V_T0_300":ESM.V_T0_300, + "V_T0_350":ESM.V_T0_350, + "V_T0_400":ESM.V_T0_400, + "V_T0_450":ESM.V_T0_450, + "V_T0_500":ESM.V_T0_500, + "V_T0_600":ESM.V_T0_600, + "V_T0_700":ESM.V_T0_700, + "V_T0_750":ESM.V_T0_750, + "V_T0_800":ESM.V_T0_800, + "V_T0_900":ESM.V_T0_900, + "V_T1_000":ESM.V_T1_000, + "V_T1_200":ESM.V_T1_200, + "V_T1_400":ESM.V_T1_400, + "V_T1_600":ESM.V_T1_600, + "V_T1_800":ESM.V_T1_800, + "V_T2_000":ESM.V_T2_000, + "V_T2_500":ESM.V_T2_500, + "V_T3_000":ESM.V_T3_000, + "V_T3_500":ESM.V_T3_500, + "V_T4_000":ESM.V_T4_000, + "V_T4_500":ESM.V_T4_500, + "V_T5_000":ESM.V_T5_000, + "V_T6_000":ESM.V_T6_000, + "V_T7_000":ESM.V_T7_000, + "V_T8_000":ESM.V_T8_000, + "V_T9_000":ESM.V_T9_000, + "V_T10_000":ESM.V_T10_000, - "W_T0_010":ESM22.W_T0_010, - "W_T0_025":ESM22.W_T0_025, - "W_T0_040":ESM22.W_T0_040, - "W_T0_050":ESM22.W_T0_050, - "W_T0_070":ESM22.W_T0_070, - "W_T0_100":ESM22.W_T0_100, - "W_T0_150":ESM22.W_T0_150, - "W_T0_200":ESM22.W_T0_200, - "W_T0_250":ESM22.W_T0_250, - "W_T0_300":ESM22.W_T0_300, - "W_T0_350":ESM22.W_T0_350, - "W_T0_400":ESM22.W_T0_400, - "W_T0_450":ESM22.W_T0_450, - "W_T0_500":ESM22.W_T0_500, - "W_T0_600":ESM22.W_T0_600, - "W_T0_700":ESM22.W_T0_700, - "W_T0_750":ESM22.W_T0_750, - "W_T0_800":ESM22.W_T0_800, - "W_T0_900":ESM22.W_T0_900, - "W_T1_000":ESM22.W_T1_000, - "W_T1_200":ESM22.W_T1_200, - "W_T1_400":ESM22.W_T1_400, - "W_T1_600":ESM22.W_T1_600, - "W_T1_800":ESM22.W_T1_800, - "W_T2_000":ESM22.W_T2_000, - "W_T2_500":ESM22.W_T2_500, - "W_T3_000":ESM22.W_T3_000, - "W_T3_500":ESM22.W_T3_500, - "W_T4_000":ESM22.W_T4_000, - "W_T4_500":ESM22.W_T4_500, - "W_T5_000":ESM22.W_T5_000, - "W_T6_000":ESM22.W_T6_000, - "W_T7_000":ESM22.W_T7_000, - "W_T8_000":ESM22.W_T8_000, - "W_T9_000":ESM22.W_T9_000, - "W_T10_000":ESM22.W_T10_000, + "W_T0_010":ESM.W_T0_010, + "W_T0_025":ESM.W_T0_025, + "W_T0_040":ESM.W_T0_040, + "W_T0_050":ESM.W_T0_050, + "W_T0_070":ESM.W_T0_070, + "W_T0_100":ESM.W_T0_100, + "W_T0_150":ESM.W_T0_150, + "W_T0_200":ESM.W_T0_200, + "W_T0_250":ESM.W_T0_250, + "W_T0_300":ESM.W_T0_300, + "W_T0_350":ESM.W_T0_350, + "W_T0_400":ESM.W_T0_400, + "W_T0_450":ESM.W_T0_450, + "W_T0_500":ESM.W_T0_500, + "W_T0_600":ESM.W_T0_600, + "W_T0_700":ESM.W_T0_700, + "W_T0_750":ESM.W_T0_750, + "W_T0_800":ESM.W_T0_800, + "W_T0_900":ESM.W_T0_900, + "W_T1_000":ESM.W_T1_000, + "W_T1_200":ESM.W_T1_200, + "W_T1_400":ESM.W_T1_400, + "W_T1_600":ESM.W_T1_600, + "W_T1_800":ESM.W_T1_800, + "W_T2_000":ESM.W_T2_000, + "W_T2_500":ESM.W_T2_500, + "W_T3_000":ESM.W_T3_000, + "W_T3_500":ESM.W_T3_500, + "W_T4_000":ESM.W_T4_000, + "W_T4_500":ESM.W_T4_500, + "W_T5_000":ESM.W_T5_000, + "W_T6_000":ESM.W_T6_000, + "W_T7_000":ESM.W_T7_000, + "W_T8_000":ESM.W_T8_000, + "W_T9_000":ESM.W_T9_000, + "W_T10_000":ESM.W_T10_000, + # No RotD intensity measures provided in raw web service format ESM flatfile "rotD50_T0_010":default_string, "rotD50_T0_025":default_string, "rotD50_T0_040":default_string, @@ -930,10 +894,9 @@ def _get_ESM18_headers(ESM22,default_string,r_fm_type,r_datetime): "rotD00_T10_000":default_string}) # Output to folder where converted flatfile read into parser - DATA = os.path.abspath('') temp_folder=tempfile.mkdtemp() - converted_base_data_path = os.path.join(DATA,temp_folder, + converted_base_data_path = os.path.join(DATA,temp_folder, 'converted_flatfile.csv') - ESM_original_headers.to_csv(converted_base_data_path,sep=';') + ESM_original_headers.to_csv(converted_base_data_path, sep=';') return converted_base_data_path \ No newline at end of file diff --git a/openquake/smt/parsers/gem_flatfile_parser.py b/openquake/smt/parsers/gem_flatfile_parser.py new file mode 100644 index 000000000..b241f29bb --- /dev/null +++ b/openquake/smt/parsers/gem_flatfile_parser.py @@ -0,0 +1,592 @@ +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2024 GEM Foundation +# +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with OpenQuake. If not, see . +""" +Parse the GEM globally homogenised flatfile into SMT metadata. +""" +import pandas as pd +import os, sys +import tempfile +import csv +import numpy as np +import copy +import h5py +from math import sqrt +from linecache import getline +from collections import OrderedDict + +from openquake.smt.sm_database import GroundMotionDatabase, GroundMotionRecord,\ + Earthquake, Magnitude, Rupture, FocalMechanism, GCMTNodalPlanes,\ + Component, RecordSite, RecordDistance +from openquake.smt.sm_utils import MECHANISM_TYPE, DIP_TYPE, vs30_to_z1pt0_cy14,\ + vs30_to_z2pt5_cb14 +from openquake.smt.parsers import valid +from openquake.smt.parsers.base_database_parser import SMDatabaseReader + +if sys.version_info[0] >= 3: + import pickle +else: + import cPickle as pickle + +# Import the ESM dictionaries +from .esm_dictionaries import * + +# Base path +DATA = os.path.abspath('') + +SCALAR_LIST = ["PGA", "PGV", "PGD", "CAV", "CAV5", "Ia", "D5-95"] + +HEADER_STR = "event_id;event_time;ISC_ev_id;USGS_ev_id;INGV_ev_id;"\ + "EMSC_ev_id;ev_nation_code;ev_latitude;ev_longitude;"\ + "ev_depth_km;fm_type_code;ML;Mw;Ms;event_source_id;"\ + "es_strike;es_dip;es_rake;es_strike_dip_rake_ref;es_z_top;"\ + "es_length;es_width;network_code;station_code;location_code;"\ + "instrument_code;sensor_depth_m;housing_code;installation_code;"\ + "st_nation_code;st_latitude;st_longitude;st_elevation;vs30_m_sec;"\ + "slope_deg;vs30_meas_type;epi_dist;epi_az;JB_dist;rup_dist;Rx_dist;"\ + "Ry0_dist;late_triggered_flag_01;U_channel_code;U_azimuth_deg;"\ + "V_channel_code;V_azimuth_deg;W_channel_code;U_hp;V_hp;W_hp;U_lp;"\ + "V_lp;W_lp" + +HEADERS = set(HEADER_STR.split(";")) + + +class GEMFlatfileParser(SMDatabaseReader): + """ + Parses the data from the flatfile to a set of metadata objects + """ + M_PRECEDENCE = ["Mw", "Ms", "ML"] + BUILD_FINITE_DISTANCES = False + + def parse(self, location='./'): + """ + Parse the dataset + """ + assert os.path.isfile(self.filename) + headers = getline(self.filename, 1).rstrip("\n").split(";") + for hdr in HEADERS: + if hdr not in headers: + raise ValueError("Required header %s is missing in file" + % hdr) + # Read in csv + reader = csv.DictReader(open(self.filename, "r"), delimiter=";") + self.database = GroundMotionDatabase(self.id, self.name) + counter = 0 + for row in reader: + # Build the metadata + record = self._parse_record(row) + if record: + # Parse the strong motion + record = self._parse_ground_motion( + os.path.join(location, "records"), + row, record, headers) + self.database.records.append(record) + + else: + print("Record with sequence number %s is null/invalid" + % "{:s}-{:s}".format(row["event_id"], + row["station_code"])) + if (counter % 1) == 0: + print("Processed record %s - %s" % (str(counter), + record.id)) + + counter += 1 + + @classmethod + def autobuild(cls, dbid, dbname, output_location, ESM_flatfile_directory, + proxy=None, removal=None): + """ + Quick and dirty full database builder! + """ + # Import GEM strong-motion flatfile + GEM = pd.read_csv(ESM_flatfile_directory) + + # Get path to tmp csv once modified dataframe + converted_base_data_path=_prioritise_rotd50(GEM, proxy, removal) + + if os.path.exists(output_location): + raise IOError("Target database directory %s already exists!" + % output_location) + os.mkdir(output_location) + # Add on the records folder + os.mkdir(os.path.join(output_location, "records")) + # Create an instance of the parser class + database = cls(dbid, dbname, converted_base_data_path) + # Parse the records + print("Parsing Records ...") + database.parse(location=output_location) + # Save itself to file + metadata_file = os.path.join(output_location, "metadatafile.pkl") + print("Storing metadata to file %s" % metadata_file) + with open(metadata_file, "wb+") as f: + pickle.dump(database.database, f) + + return database + + def _parse_record(self, metadata): + """ + Parse a record + """ + # Waveform ID not provided in file so concatenate Event and Station ID + wfid = "_".join([metadata["event_id"], metadata["network_code"], + metadata["station_code"], metadata["location_code"]]) + wfid = wfid.replace("-", "_") + # Parse the event metadata + event = self._parse_event_data(metadata) + # Parse the distance metadata + distances = self._parse_distances(metadata, event.depth) + # Parse the station metadata + site = self._parse_site_data(metadata) + # Parse waveform data + xcomp, ycomp, vertical = self._parse_waveform_data(metadata, wfid) + return GroundMotionRecord(wfid, + [None, None, None], + event, distances, site, + xcomp, ycomp, + vertical=vertical) + + + def _parse_event_data(self, metadata): + """ + Parses the event metadata + """ + # ID and Name (name not in file so use ID again) + eq_id = metadata["event_id"] + eq_name = metadata["event_id"] + # Country + cntry_code = metadata["ev_nation_code"].strip() + # Date and time + eq_datetime = valid.date_time(metadata["event_time"], + "%Y-%m-%d %H:%M:%S") + # Latitude, longitude and depth + eq_lat = valid.latitude(metadata["ev_latitude"]) + eq_lon = valid.longitude(metadata["ev_longitude"]) + eq_depth = valid.positive_float(metadata["ev_depth_km"], "ev_depth_km") + if not eq_depth: + eq_depth = 0.0 + eqk = Earthquake(eq_id, eq_name, eq_datetime, eq_lon, eq_lat, eq_depth, + None, # Magnitude not defined yet + eq_country=None) + # Get preferred magnitude and list + pref_mag, magnitude_list = self._parse_magnitudes(metadata) + eqk.magnitude = pref_mag + eqk.magnitude_list = magnitude_list + eqk.rupture, eqk.mechanism = self._parse_rupture_mechanism(metadata, + eq_id, + eq_name, + pref_mag, + eq_depth) + return eqk + + def _parse_magnitudes(self, metadata): + """ + An order of precedence is required and the preferred magnitude will be + the highest found + """ + pref_mag = None + mag_list = [] + for key in self.M_PRECEDENCE: + mvalue = metadata[key].strip() + if mvalue: + mtype = key + msource = metadata[key + "_ref"].strip() + mag = Magnitude(float(mvalue), + mtype, + source=msource) + if not pref_mag: + pref_mag = copy.deepcopy(mag) + mag_list.append(mag) + return pref_mag, mag_list + + def _parse_rupture_mechanism(self, metadata, eq_id, eq_name, mag, depth): + """ + If rupture data is available - parse it, otherwise return None + """ + + sof = metadata["fm_type_code"] + if not metadata["event_source_id"].strip(): + # No rupture model available. Mechanism is limited to a style + # of faulting only + rupture = Rupture(eq_id, eq_name, mag, None, None, depth) + mechanism = FocalMechanism( + eq_id, eq_name, GCMTNodalPlanes(), None, + mechanism_type=sof) + # See if focal mechanism exists + fm_set = [] + for key in ["strike_1", "dip_1", "rake_1"]: + if key in metadata: + fm_param = valid.vfloat(metadata[key], key) + if fm_param is not None: + fm_set.append(fm_param) + if len(fm_set) == 3: + # Have one valid focal mechanism + mechanism.nodal_planes.nodal_plane_1 = {"strike": fm_set[0], + "dip": fm_set[1], + "rake": fm_set[2]} + fm_set = [] + for key in ["strike_2", "dip_2", "rake_2"]: + if key in metadata: + fm_param = valid.vfloat(metadata[key], key) + if fm_param is not None: + fm_set.append(fm_param) + if len(fm_set) == 3: + # Have one valid focal mechanism + mechanism.nodal_planes.nodal_plane_2 = {"strike": fm_set[0], + "dip": fm_set[1], + "rake": fm_set[2]} + + if not mechanism.nodal_planes.nodal_plane_1 and not\ + mechanism.nodal_planes.nodal_plane_2: + # Absolutely no information - base on stye-of-faulting + mechanism.nodal_planes.nodal_plane_1 = { + "strike": 0.0, # Basically unused + "dip": DIP_TYPE[sof], + "rake": MECHANISM_TYPE[sof] + } + return rupture, mechanism + + strike = valid.strike(metadata["es_strike"]) + dip = valid.dip(metadata["es_dip"]) + rake = valid.rake(metadata["es_rake"]) + ztor = valid.positive_float(metadata["es_z_top"], "es_z_top") + length = valid.positive_float(metadata["es_length"], "es_length") + width = valid.positive_float(metadata["es_width"], "es_width") + rupture = Rupture(eq_id, eq_name, mag, length, width, ztor) + + # Get mechanism type and focal mechanism + # No nodal planes, eigenvalues moment tensor initially + mechanism = FocalMechanism( + eq_id, eq_name, GCMTNodalPlanes(), None, + mechanism_type=metadata["fm_type_code"]) + if strike is None: + strike = 0.0 + if dip is None: + dip = DIP_TYPE[sof] + if rake is None: + rake = MECHANISM_TYPE[sof] + # if strike is not None and dip is not None and rake is not None: + mechanism.nodal_planes.nodal_plane_1 = {"strike": strike, + "dip": dip, + "rake": rake} + return rupture, mechanism + + def _parse_distances(self, metadata, hypo_depth): + """ + Parse the distances + """ + repi = valid.positive_float(metadata["epi_dist"], "epi_dist") + razim = valid.positive_float(metadata["epi_az"], "epi_az") + rjb = valid.positive_float(metadata["JB_dist"], "JB_dist") + rrup = valid.positive_float(metadata["rup_dist"], "rup_dist") + r_x = valid.vfloat(metadata["Rx_dist"], "Rx_dist") + ry0 = valid.positive_float(metadata["Ry0_dist"], "Ry0_dist") + rhypo = sqrt(repi ** 2. + hypo_depth ** 2.) + if not isinstance(rjb, float): + # In the first case Rjb == Repi + rjb = copy.copy(repi) + + if not isinstance(rrup, float): + # In the first case Rrup == Rhypo + rrup = copy.copy(rhypo) + + if not isinstance(r_x, float): + # In the first case Rx == -Repi (collapse to point and turn off + # any hanging wall effect) + r_x = copy.copy(-repi) + + if not isinstance(ry0, float): + # In the first case Ry0 == Repi + ry0 = copy.copy(repi) + distances = RecordDistance(repi, rhypo, rjb, rrup, r_x, ry0) + distances.azimuth = razim + return distances + + def _parse_site_data(self, metadata): + """ + Parses the site information + """ + network_code = metadata["network_code"].strip() + station_code = metadata["station_code"].strip() + site_id = "{:s}-{:s}".format(network_code, station_code) + site_lon = valid.longitude(metadata["st_longitude"]) + site_lat = valid.latitude(metadata["st_latitude"]) + elevation = valid.vfloat(metadata["st_elevation"], "st_elevation") + + vs30 = valid.vfloat(metadata["vs30_m_sec"], "vs30_m_sec") + vs30_measured = None # User should check selected records (many diff. + # flatfiles --> see vs30_meas_type column for if + # vs30 if available was measured of inferred) + + site = RecordSite(site_id, station_code, station_code, site_lon, + site_lat, elevation, vs30, vs30_measured, + network_code=network_code, country=None) + site.slope = valid.vfloat(metadata["slope_deg"], "slope_deg") + site.sensor_depth = valid.vfloat(metadata["sensor_depth_m"], + "sensor_depth_m") + site.instrument_type = metadata["instrument_code"].strip() + if site.vs30: + site.z1pt0 = vs30_to_z1pt0_cy14(vs30) + site.z2pt5 = vs30_to_z2pt5_cb14(vs30) + housing_code = metadata["housing_code"].strip() + if housing_code and (housing_code in HOUSING): + site.building_structure = HOUSING[housing_code] + return site + + def _parse_waveform_data(self, metadata, wfid): + """ + Parse the waveform data + """ + late_trigger = valid.vint(metadata["late_triggered_flag_01"], + "late_triggered_flag_01") + # U channel - usually east + xorientation = metadata["U_channel_code"].strip() + xazimuth = valid.vfloat(metadata["U_azimuth_deg"], "U_azimuth_deg") + xfilter = {"Low-Cut": valid.vfloat(metadata["U_hp"], "U_hp"), + "High-Cut": valid.vfloat(metadata["U_lp"], "U_lp")} + xcomp = Component(wfid, xazimuth, waveform_filter=xfilter, + units="cm/s/s") + xcomp.late_trigger = late_trigger + # V channel - usually North + vorientation = metadata["V_channel_code"].strip() + vazimuth = valid.vfloat(metadata["V_azimuth_deg"], "V_azimuth_deg") + vfilter = {"Low-Cut": valid.vfloat(metadata["V_hp"], "V_hp"), + "High-Cut": valid.vfloat(metadata["V_lp"], "V_lp")} + vcomp = Component(wfid, vazimuth, waveform_filter=vfilter, + units="cm/s/s") + vcomp.late_trigger = late_trigger + zorientation = metadata["W_channel_code"].strip() + if zorientation: + zfilter = {"Low-Cut": valid.vfloat(metadata["W_hp"], "W_hp"), + "High-Cut": valid.vfloat(metadata["W_lp"], "W_lp")} + zcomp = Component(wfid, None, waveform_filter=zfilter, + units="cm/s/s") + zcomp.late_trigger = late_trigger + else: + zcomp = None + + return xcomp, vcomp, zcomp + + def _parse_ground_motion(self, location, row, record, headers): + """ + In this case we parse the information from the flatfile directly + to hdf5 at the metadata stage + """ + # Get the data + scalars, spectra = self._retreive_ground_motion_from_row(row, headers) + # Build the hdf5 files + filename = os.path.join(location, "{:s}.hdf5".format(record.id)) + fle = h5py.File(filename, "w-") + ims_grp = fle.create_group("IMS") + for comp, key in [("X", "U"), ("Y", "V"), ("V", "W")]: + comp_grp = ims_grp.create_group(comp) + # Add on the scalars + scalar_grp = comp_grp.create_group("Scalar") + for imt in scalars[key]: + if imt in ["ia", "housner"]: + # In the smt convention it is "Ia" and "Housner" + ikey = imt[0].upper() + imt[1:] + else: + # Everything else to upper case (PGA, PGV, PGD, T90, CAV) + ikey = imt.upper() + dset = scalar_grp.create_dataset(ikey, (1,), dtype="f") + dset[:] = scalars[key][imt] + # Add on the spectra + spectra_grp = comp_grp.create_group("Spectra") + response = spectra_grp.create_group("Response") + accel = response.create_group("Acceleration") + accel.attrs["Units"] = "cm/s/s" + # Add on the periods + pers = spectra[key]["Periods"] + periods = response.create_dataset("Periods", pers.shape, dtype="f") + periods[:] = pers + periods.attrs["Low Period"] = np.min(pers) + periods.attrs["High Period"] = np.max(pers) + periods.attrs["Number Periods"] = len(pers) + + # Add on the values + values = spectra[key]["Values"] + spectra_dset = accel.create_dataset("damping_05", values.shape, + dtype="f") + spectra_dset[:] = np.copy(values) + spectra_dset.attrs["Damping"] = 5.0 + # Add on the horizontal values + hcomp = ims_grp.create_group("H") + # Scalars - just geometric mean for now + hscalar = hcomp.create_group("Scalar") + for imt in scalars["Geometric"]: + if imt in ["ia", "housner"]: + # In the smt convention it is "Ia" and "Housner" + key = imt[0].upper() + imt[1:] + else: + # Everything else to upper case (PGA, PGV, PGD, T90, CAV) + key = imt.upper() + dset = hscalar.create_dataset(key, (1,), dtype="f") + dset[:] = scalars["Geometric"][imt] + # For Spectra - can support multiple components + hspectra = hcomp.create_group("Spectra") + hresponse = hspectra.create_group("Response") + pers = spectra["Geometric"]["Periods"] + hpers_dset = hresponse.create_dataset("Periods", pers.shape, dtype="f") + hpers_dset[:] = np.copy(pers) + hpers_dset.attrs["Low Period"] = np.min(pers) + hpers_dset.attrs["High Period"] = np.max(pers) + hpers_dset.attrs["Number Periods"] = len(pers) + haccel = hresponse.create_group("Acceleration") + for htype in ["Geometric", "rotD00", "rotD50", "rotD100"]: + if np.all(np.isnan(spectra[htype]["Values"])): + # Component not determined + continue + if not (htype == "Geometric"): + key = htype[0].upper() + htype[1:] + else: + key = copy.deepcopy(htype) + htype_grp = haccel.create_group(htype) + hvals = spectra[htype]["Values"] + hspec_dset = htype_grp.create_dataset("damping_05", hvals.shape, + dtype="f") + hspec_dset[:] = hvals + hspec_dset.attrs["Units"] = "cm/s/s" + record.datafile = filename + return record + + + def _retreive_ground_motion_from_row(self, row, header_list): + """ + Get the ground-motion data from a row (record) in the database + """ + imts = ["U", "V", "W", "rotD00", "rotD100", "rotD50"] + spectra = [] + scalar_imts = ["pga", "pgv", "pgd", "T90", "housner", "ia", "CAV"] + scalars = [] + for imt in imts: + periods = [] + values = [] + key = "{:s}_T".format(imt) + scalar_dict = {} + for header in header_list: + # Deal with the scalar case + for scalar in scalar_imts: + if header == "{:s}_{:s}".format(imt, scalar): + # The value is a scalar + value = row[header].strip() + if value: + scalar_dict[scalar] = np.fabs(float(value)) + else: + scalar_dict[scalar] = None + scalars.append((imt, scalar_dict)) + for header in header_list: + if key in header: + if header == "{:s}90".format(key): + # Not a spectral period but T90 + continue + iky = header.replace(key, "").replace("_", ".") + periods.append(float(iky)) + value = row[header].strip() + if value: + values.append(np.fabs(float(value))) + else: + values.append(np.nan) + periods = np.array(periods) + values = np.array(values) + idx = np.argsort(periods) + spectra.append((imt, {"Periods": periods[idx], + "Values": values[idx]})) + # Add on the as-recorded geometric mean + spectra = OrderedDict(spectra) + scalars = OrderedDict(scalars) + spectra["Geometric"] = { + "Values": np.sqrt(spectra["U"]["Values"] * + spectra["V"]["Values"]), + "Periods": np.copy(spectra["U"]["Periods"]) + } + scalars["Geometric"] = dict([(key, None) for key in scalars["U"]]) + for key in scalars["U"]: + if scalars["U"][key] and scalars["V"][key]: + scalars["Geometric"][key] = np.sqrt( + scalars["U"][key] * scalars["V"][key]) + return scalars, spectra + + +def _prioritise_rotd50(df, proxy=None, removal=None): + """ + Assign RotD50 values to accelerations for computation of residuals. RotD50 + is available for the vast majority of the records in the GEM flatfile for + PGA to 10 s (KikNet subset of records is limited to up to 5 s because we + use Beyes and Bommer 2006 to convert from horizontals to RotD50). + + If no RotD50 use the geometric mean if available (if specified) as a proxy + for RotD50. + + Records lacking acceleration values for any of the required spectral periods + can also be removed (this information can alternatively just be printed) + + :param proxy: + If set to true, if a record is missing RotD50 try and use the geometric + mean of the horizontal components + + :param removal: + If set to true records without complete RotD50 (or complete RotD50 with + use of geometric mean as proxy - if proxy is True) for any of the required + spectral periods are removed. + """ + # Manage RotD50 vs horizontal components + log, cols = [], [] + for idx, rec in df.iterrows(): + for col in rec.index: + if 'rotD' in col: + cols.append(col) + if 'U_T' in col or 'V_T' in col or 'U_pga' in col or 'V_pga' in col: + if 'T90' not in col: + if 'U_' in col: + rotd50_col = col.replace('U_', 'rotD50_') + if 'V_' in col: + rotd50_col = col.replace('V_', 'rotD50_') + + # If RotD50... + if not pd.isnull(rec[rotd50_col]): + df[col].iloc[idx] = rec[rotd50_col] # Assign to h1, h2 + + # Otherwise... + else: + if proxy is True: # Use geo. mean as proxy + if not pd.isnull(rec[col]): + pass # Can use geo. mean from h1, h2 as proxy + else: + log.append(idx) # Log rec as incomplete RotD50 vals + + # Tidy dataframe + cols = pd.Series(cols).unique() + df = df.drop(columns=cols) + + # Drop if req. or else just inform number of recs missing acceleration values + no_vals = len(pd.Series(log).unique()) + if removal is True and log!= []: + df = df.drop(log).reset_index() + msg = 'Records without RotD50 acc. values at all required periods' + msg += ' have been removed from flatfile (%s records)' % no_vals + print(msg) + if len(df) == 0: + raise ValueError('All records have been removed from the flatfile') + elif log != []: + print('%s records do not have RotD50 for all req. periods' % no_vals) + + # Output to folder where converted flatfile read into parser + tmp = tempfile.mkdtemp() + converted_base_data_path = os.path.join(DATA, tmp, 'converted_flatfile.csv') + df.to_csv(converted_base_data_path, sep=';') + + return converted_base_data_path \ No newline at end of file diff --git a/openquake/smt/parsers/general_flatfile_parser.py b/openquake/smt/parsers/general_flatfile_parser.py deleted file mode 100755 index d7229195e..000000000 --- a/openquake/smt/parsers/general_flatfile_parser.py +++ /dev/null @@ -1,391 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- -# vim: tabstop=4 shiftwidth=4 softtabstop=4 -# -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill -# -# OpenQuake is free software: you can redistribute it and/or modify it -# under the terms of the GNU Affero General Public License as published -# by the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# OpenQuake is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with OpenQuake. If not, see . - -""" -General Flatfile Only Parser - to handle the cases in which the only strong -motion input is in the form of a flatfile -""" -import csv -import numpy as np -import copy -from linecache import getline - -from openquake.hazardlib.scalerel.wc1994 import WC1994 -from openquake.hazardlib.geo.mesh import Mesh -from openquake.hazardlib.geo.point import Point -from openquake.hazardlib.geo.line import Line -from openquake.hazardlib.geo.surface.simple_fault import SimpleFaultSurface -from openquake.smt.sm_database import * -from openquake.smt.parsers.base_database_parser import (get_float, - get_positive_float) -from openquake.smt.parsers.simple_flatfile_parser_sara import SimpleFlatfileParserV9 - -from openquake.smt.sm_utils import create_planar_surface, get_hypocentre_on_planar_surface - -SCALAR_LIST = ["PGA", "PGV", "PGD", "CAV", "CAV5", "Ia", "D5-95"] - -HEADER_LIST = set([ - 'Record Sequence Number', 'EQID', 'Earthquake Name', 'Country', 'Year', - 'Month', 'Day', 'Hour', 'Minute', 'Second', - 'Epicenter Latitude (deg; positive N)', - 'Epicenter Longitude (deg; positive E)', 'Hypocenter Depth (km)', - 'Magnitude', 'Magnitude type', 'Magnitude uncertainty', 'Mo (dyne.cm)', - 'Tectonic environment (Crustal; Inslab; Interface; Stable; Geothermal; Volcanic; Oceanic_crust)', - 'Earthquake in Extensional Regime: 1=Yes; 0=No', - 'Nodal Plane 1 Strike (deg)', - 'Nodal Plane 1 Dip (deg)', 'Nodal Plane 1 Rake Angle (deg)', - 'Nodal Plane 2 Strike (deg)', 'Nodal Plane 2 Dip (deg)', - 'Nodal Plane 2 Rake Angle (deg)', 'Fault Plane (1; 2; X)', - 'Style-of-Faulting (S; R; N; U)', 'Fault Name', - 'Depth to Top Of Fault Rupture Model', - 'Fault Rupture Length (km)', 'Fault Rupture Width (km)', - 'Along-strike Hypocenter location on the fault (fraction between 0 and 1)', - 'Along-width Hypocenter location on the fault (fraction between 0 and 1)', - 'Static Stress Drop (bars)', 'Type Static Stress Drop', - 'Directivity flag (Y; N; U)', 'Station ID', 'Station Code', - 'Station Latitude (deg positive N)', 'Station Longitude (deg positive E)', - 'Station Elevation (m)', - 'Site Class (Hard Rock; Rock; Stiff Soil; Soft Soil)', - 'Preferred NEHRP Based on Vs30', 'Preferred Vs30 (m/s)', - 'Measured(1)/Inferred(2) Class', 'Sigma of Vs30 (in natural log Units)', - 'Depth to Basement Rock', 'Z1 (m)','Z2.5 (m)', 'Epicentral Distance (km)', - 'Hypocentral Distance (km)', 'Joyner-Boore Distance (km)', - 'Rupture Distance (km)', 'Rx (km)', 'Ry0 (km)', - 'Source to Site Azimuth (deg)', 'FW/HW Indicator', - 'Forearc/Backarc for subduction events', 'File Name (Horizontal 1)', - 'File Name (Horizontal 2)','File Name (Vertical)', - 'Digital (D)/Analog (A) Recording', 'Acceleration (A)/Velocity (V)', - 'Unit (cm/s/s; m/s/s; g)', 'File format', 'Processing flag', - 'Type of Filter','npass','nroll','HP-H1 (Hz)','HP-H2 (Hz)', - 'LP-H1 (Hz)', 'LP-H2 (Hz)', 'Factor', 'Lowest Usable Freq - H1 (Hz)', - 'Lowest Usable Freq - H2 (Hz)', 'Lowest Usable Freq - Ave. Component (Hz)', - 'Maximum Usable Freq - Ave. Component (Hz)', 'HP-V (Hz)', 'LP-V (Hz)', - 'Lowest Usable Freq - V (Hz)','Comments']) - - -def valid_positive(value): - """ - Returns True if the value is positive or zero, false otherwise - """ - if value and value >= 0.0: - return True - print("Positive value (or 0) is needed - %s is given" % str(value)) - return False - -def valid_longitude(lon): - """ - Returns True if the longitude is valid, False otherwise - """ - if not lon: - return False - if (lon >= -180.0) and (lon <= 180.0): - return True - print("Longitude %s is outside of range -180 <= lon <= 180" % str(lon)) - return False - -def valid_latitude(lat): - """ - Returns True if the latitude is valid, False otherwise - """ - if not lat: - print("Latitude is missing") - return False - if (lat >= -90.0) and (lat <= 90.0): - return True - print("Latitude %s is outside of range -90 <= lat <= 90" % str(lat)) - return False - -def valid_date(year, month, day): - """ - Checks that the year is given and greater than 0, that month is in the - range 1 - 12, and day is in the range 1 - 31 - """ - if all([year > 0, month > 0, month <= 12, day > 0, day <= 31]): - return True - print("Date %s/%s/%s is not valid" % (str(year), str(month), str(day))) - return False - - - -class GeneralFlatfileParser(SimpleFlatfileParserV9): - """ - Operates in the same manner as for SARA Simple Flatfile parser, albeit - with Rx and Ry0 added as explicit columns - """ - def parse(self): - """ - Parses the database - """ - HEADER_LIST1 = copy.deepcopy(HEADER_LIST) - self._header_check(HEADER_LIST1) - # Read in csv - reader = csv.DictReader(open(self.filename, "r")) - metadata = [] - self.database = GroundMotionDatabase(self.id, self.name) - self._get_site_id = self.database._get_site_id - counter = 0 - for row in reader: - #print(row["Record Sequence Number"]) - #if (counter % 100) == 0: - # print("%s - %s" % (str(counter), row["Record Sequence Number"])) - - if self._sanitise(row, reader): - record = self._parse_record(row) - if record: - self.database.records.append(record) - else: - print("Record with sequence number %s is null/invalid" - % str(row["Record Sequence Number"])) - counter += 1 - return self.database - - - def _sanitise(self, row, reader): - """ - If all of the strong motion values are negative the record is null - and should be removed - """ - iml_vals = [] - for fname in reader.fieldnames: - if fname.startswith("SA(") or fname in SCALAR_LIST: - # Ground motion value - iml = get_float(row[fname]) - if iml: - iml_vals.append(iml) - else: - iml_vals.append(-999.0) - # If all ground motion values are negative then the record is null - # return false - if np.all(np.array(iml_vals) < 0.0): - return False - else: - return True - - def _parse_record(self, metadata): - """ - Parses the record information and returns an instance of the - :class: openquake.smt.sm_database.GroundMotionRecord - """ - # Waveform ID - wfid = metadata["Record Sequence Number"] - # Event information - # Verify datetime - event = self._parse_event_data(metadata) - if not self._verify_event(event, wfid): - print("Record Number %s has invalid event!" % wfid) - return False - # Site information - site = self._parse_site_data(metadata) - # Distance Information - distances = self._parse_distance_data(event, site, metadata) - # Components - x_comp, y_comp, vertical = self._parse_processing_data(wfid, metadata) - # Return record metadata - lup = get_float(metadata["Lowest Usable Freq - Ave. Component (Hz)"]) - if lup: - lup = 1. / lup - sup = get_float(metadata["Maximum Usable Freq - Ave. Component (Hz)"]) - if sup: - sup = 1. / sup - return GroundMotionRecord(wfid, - [metadata["File Name (Horizontal 1)"], - metadata["File Name (Horizontal 2)"], - metadata["File Name (Vertical)"]], - event, - distances, - site, - x_comp, - y_comp, - longest_period=lup, - shortest_period=sup) - - def _verify_event(self, event, wfid): - """ - Some basic checks are needed to ensure that the event is usable! - 1. Year, month and day should be valid - 2. Magnitude should be present and valid > -10.0! - 3. Epicentre should be in valid range (-180.0 <= lon <= 180, - -90.0 <= lat <= 180) - 4. Depth should be in valid range (>= 0) - """ - is_valid = valid_date(event.datetime.year, event.datetime.month, - event.datetime.day) and\ - valid_longitude(event.longitude) and\ - valid_latitude(event.latitude) and\ - valid_positive(event.depth) - if not is_valid: - return False - - # Magnitude check - if not event.magnitude.value or event.magnitude.value < -10.0: - # Invalid or missing event - print("Record Number %s has invalid magnitude value %s" - % (str(wfid), str(event.magnitude.value))) - return False - return True - - - def _header_check(self, headerslist): - """ - Checks to see if any of the headers are missing, raises error if so. - Also informs the user of redundent headers in the input file. - """ - # Check which of the pre-defined headers are missing - headers = set((getline(self.filename, 1).rstrip("\n")).split(",")) - missing_headers = headerslist.difference(headers) - if len(missing_headers) > 0: - output_string = ", ".join([value for value in missing_headers]) - raise IOError("The following headers are missing from the input " - "file: %s" % output_string) - - additional_headers = headers.difference(headerslist) - if len(additional_headers) > 0: - for header in additional_headers: - if ("SA(" in header) or ("PGA" in header) or ("PGV" in header)\ - or ("PGD" in header): - continue - else: - print("Header %s not recognised - ignoring this data!" %\ - header) - -# @staticmethod -# def _validate_datetime(metadata): -# """ -# NGA West 2 flatfile is badly formatted w.r.t. date and time - this -# bit of defensive coding should prevent it from crashing out with -# errors -# """ -# year = get_int(metadata["Year"]) -# month = get_int(metadata["Month"]) -# # Month values given incorrectly for many records -# if month > 12: -# month = (month % 12) -# if month == 0: -# month = 12 -# -# day = get_int(metadata["Day"]) -# hour = get_int(metadata["Hour"]) -# minute = get_int(metadata["Minute"]) -# second = get_int(metadata["Second"]) -# return year, month, day, hour, minute, second - - def _parse_distance_data(self, event, site, metadata): - """ - Read in the distance related metadata and return an instance of the - :class: openquake.smt.sm_database.RecordDistance - """ - # Compute various distance metrics - # Add calculation of Repi, Rhypo from event and station localizations - # (latitudes, longitudes, depth, elevation)? - target_site = Mesh(np.array([site.longitude]), - np.array([site.latitude]), - np.array([-site.altitude / 1000.0])) - # Warning ratio fixed to 1.5 - ratio=1.5 - if not event.rupture.area: - event.rupture.area = WC1994().get_median_area(event.magnitude.value, - None) - surface_modeled = create_planar_surface( - Point(event.longitude, event.latitude, event.depth), - event.mechanism.nodal_planes.nodal_plane_1['strike'], - event.mechanism.nodal_planes.nodal_plane_1['dip'], - event.rupture.area, - ratio) - hypocenter = get_hypocentre_on_planar_surface(surface_modeled, - event.rupture.hypo_loc) - try: - surface_modeled._create_mesh() - except: - dip = surface_modeled.get_dip() - dip_dir = (surface_modeled.get_strike() - 90.) % 360. - ztor = surface_modeled.top_left.depth - d_x = ztor * np.tan(np.radians(90.0 - dip)) - top_left_surface = surface_modeled.top_left.point_at(d_x, - -ztor, - dip_dir) - top_left_surface.depth = 0. - top_right_surface = surface_modeled.top_right.point_at(d_x, - -ztor, - dip_dir) - top_right_surface.depth = 0. - surface_modeled = SimpleFaultSurface.from_fault_data( - Line([top_left_surface, top_right_surface]), - surface_modeled.top_left.depth, - surface_modeled.bottom_left.depth, - surface_modeled.get_dip(), - 1.0) - - # Rhypo - Rhypo, Repi, Rrup, Rjb, Ry0 = tuple(map( - get_positive_float, [metadata[key] for key in [ - "Hypocentral Distance (km)", "Epicentral Distance (km)", - "Rupture Distance (km)", "Joyner-Boore Distance (km)", - "Ry0 (km)"]])) - Rx = get_float(metadata["Rx (km)"]) # Rx can be negative - - #Rhypo = get_float(metadata["Hypocentral Distance (km)"]) - if Rhypo is None or Rhypo < 0.0: - Rhypo = hypocenter.distance_to_mesh(target_site) - # Repi - #Repi = get_float(metadata["Epicentral Distance (km)"]) - if Repi is None or Repi < 0.0: - Repi= hypocenter.distance_to_mesh(target_site, with_depths=False) - # Rrup - #Rrup = get_float(metadata["Rupture Distance (km)"]) - if Rrup is None or Rrup < 0.0: - Rrup = surface_modeled.get_min_distance(target_site)[0] - # Rjb - #Rjb = get_float(metadata["Joyner-Boore Distance (km)"]) - if Rjb is None or Rjb < 0.0: - Rjb = surface_modeled.get_joyner_boore_distance( - target_site)[0] - # Need to check if Rx and Ry0 are consistant with the other metrics - # when those are coming from the flatfile? - # Rx - #Rx = get_float(metadata["Rx (km)"]) - if Rx is None or Rx < 0.0: - Rx = surface_modeled.get_rx_distance(target_site)[0] - # Ry0 - Ry0 = get_float(metadata["Ry0 (km)"]) - if Ry0 is None or Ry0 < 0.0: - Ry0 = surface_modeled.get_ry0_distance(target_site)[0] - - distance = RecordDistance( - repi = Repi, - rhypo = Rhypo, - rjb = Rjb, - rrup = Rrup, - r_x = Rx, - ry0 = Ry0) - distance.azimuth = get_float(metadata["Source to Site Azimuth (deg)"]) - #distance.hanging_wall = get_float(metadata["FW/HW Indicator"]) - if metadata["FW/HW Indicator"] == "HW": - distance.hanging_wall = True - elif metadata["FW/HW Indicator"] == "FW": - distance.hanging_wall = False - else: - pass - - return distance - - return -#class GeneralFlatfileSpectraReader(SMSpectraReader): -# """ -# This -# """ diff --git a/openquake/smt/parsers/ngaw2_flatfile_parser.py b/openquake/smt/parsers/ngawest2_flatfile_parser.py similarity index 75% rename from openquake/smt/parsers/ngaw2_flatfile_parser.py rename to openquake/smt/parsers/ngawest2_flatfile_parser.py index 1a8567114..9a9a10746 100644 --- a/openquake/smt/parsers/ngaw2_flatfile_parser.py +++ b/openquake/smt/parsers/ngawest2_flatfile_parser.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -45,53 +45,33 @@ # Import the ESM dictionaries from .esm_dictionaries import * -SCALAR_LIST = ["PGA", "PGV", "PGD", "CAV", "CAV5", "Ia", "D5-95"] +SCALAR_LIST = ["PGA", "PGV", "PGD"] -HEADER_STR = "event_id;event_time;ISC_ev_id;USGS_ev_id;INGV_ev_id;"\ - "EMSC_ev_id;ev_nation_code;ev_latitude;ev_longitude;"\ - "ev_depth_km;ev_hyp_ref;fm_type_code;ML;ML_ref;Mw;Mw_ref;Ms;"\ - "Ms_ref;EMEC_Mw;EMEC_Mw_type;EMEC_Mw_ref;event_source_id;"\ - "es_strike;es_dip;es_rake;es_strike_dip_rake_ref;es_z_top;"\ - "es_z_top_ref;es_length;es_width;es_geometry_ref;network_code;"\ - "station_code;location_code;instrument_code;sensor_depth_m;"\ - "proximity_code;housing_code;installation_code;st_nation_code;"\ - "st_latitude;st_longitude;st_elevation;ec8_code;"\ - "ec8_code_method;ec8_code_ref;vs30_m_sec;vs30_ref;"\ - "vs30_calc_method;vs30_meas_type;slope_deg;vs30_m_sec_WA;"\ - "epi_dist;epi_az;JB_dist;rup_dist;Rx_dist;Ry0_dist;"\ +HEADER_STR = "event_id;event_time;ev_nation_code;ev_latitude;ev_longitude;"\ + "ev_depth_km;fm_type_code;ML;ML_ref;Mw;Mw_ref;Ms;Ms_ref;EMEC_Mw;"\ + "EMEC_Mw_type;EMEC_Mw_ref;event_source_id;es_strike;es_dip;"\ + "es_rake;es_strike_dip_rake_ref;es_z_top;es_length;es_width;"\ + "network_code;station_code;location_code;instrument_code;"\ + "sensor_depth_m;proximity_code;housing_code;st_nation_code;"\ + "st_latitude;st_longitude;st_elevation;vs30_m_sec;slope_deg;"\ + "vs30_m_sec_WA;epi_dist;epi_az;JB_dist;rup_dist;Rx_dist;Ry0_dist;"\ "instrument_type_code;late_triggered_flag_01;U_channel_code;"\ "U_azimuth_deg;V_channel_code;V_azimuth_deg;W_channel_code;"\ "U_hp;V_hp;W_hp;U_lp;V_lp;W_lp" HEADERS = set(HEADER_STR.split(";")) -COUNTRY_CODES = {"AL": "Albania", "AM": "Armenia", "AT": "Austria", - "AZ": "Azerbaijan", "BA": "Bosnia and Herzegowina", - "BG": "Bulgaria", "CH": "Switzerland", "CY": "Cyprus", - "CZ": "Czech Republic", "DE": "Germany", "DZ": "Algeria", - "ES": "Spain", "FR": "France", "GE": "Georgia", - "GR": "Greece", "HR": "Croatia", "HU": "Hungary", - "IL": "Israel", "IR": "Iran", "IS": "Iceland", "IT": "Italy", - "JO": "Jordan", "LI": "Lichtenstein", "MA": "Morocco", - "MC": "Monaco", "MD": "Moldova", "ME": "Montenegro", - "MK": "Macedonia", "MT": "Malta", "PL": "Poland", - "PT": "Portugal", "RO": "Romania", "RS": "Serbia", - "RU": "Russia", "SI": "Slovenia", "SM": "San Marino", - "SY": "Syria", "TM": "Turkmenistan", "TR": "Turkey", - "UA": "Ukraine", "UZ": "Uzbekistan", "XK": "Kosovo"} - class NGAWest2FlatfileParser(SMDatabaseReader): - """ - Parses the metadata from the flatfile to a set of metadata objects + Parses the data from flatfile to a set of metadata objects """ - M_PRECEDENCE = ["EMEC_Mw", "Mw", "Ms", "ML"] BUILD_FINITE_DISTANCES = False def parse(self, location='./'): """ + Parse the metadata """ assert os.path.isfile(self.filename) headers = getline(self.filename, 1).rstrip("\n").split(";") @@ -101,24 +81,22 @@ def parse(self, location='./'): % hdr) # Read in csv reader = csv.DictReader(open(self.filename, "r"), delimiter=";") - metadata = [] self.database = GroundMotionDatabase(self.id, self.name) counter = 0 for row in reader: - if self._sanitise(row, reader): - # Build the metadata - record = self._parse_record(row) - if record: - # Parse the strong motion - record = self._parse_ground_motion( - os.path.join(location, "records"), - row, record, headers) - self.database.records.append(record) + # Build the metadata + record = self._parse_record(row) + if record: + # Parse the strong motion + record = self._parse_ground_motion( + os.path.join(location, "records"), + row, record, headers) + self.database.records.append(record) - else: - print("Record with sequence number %s is null/invalid" - % "{:s}-{:s}".format(row["event_id"], - row["station_code"])) + else: + print("Record with sequence number %s is null/invalid" + % "{:s}-{:s}".format(row["event_id"], + row["station_code"])) if (counter % 100) == 0: print("Processed record %s - %s" % (str(counter), record.id)) @@ -206,7 +184,7 @@ def autobuild(cls, dbid, dbname, output_location, NGAWest2_vertical = NGAWest2_vertical.reset_index().drop( columns='index') - #Remove records with no seismic moment to compute moment magnitude from + # Remove records with no seismic moment to compute moment magnitude from Index_to_drop=np.array(NGAWest2.loc[NGAWest2['Mo (dyne.cm)']==-999][ 'Mo (dyne.cm)'].index) NGAWest2=NGAWest2.drop(Index_to_drop) @@ -299,8 +277,9 @@ def autobuild(cls, dbid, dbname, output_location, NGAWest2_vertical['T4.500S']=(NGAWest2_vertical[ 'T4.400S']+NGAWest2_vertical['T4.600S'])/2 + # Get path to tmp csv once modified dataframe converted_base_data_path=_get_ESM18_headers( - NGAWest2,NGAWest2_vertical,Initial_NGAWest2_size) + NGAWest2, NGAWest2_vertical, Initial_NGAWest2_size) if os.path.exists(output_location): raise IOError("Target database directory %s already exists!" @@ -321,18 +300,10 @@ def autobuild(cls, dbid, dbname, output_location, return database - def _sanitise(self, row, reader): + def _parse_record(self, metadata): """ - TODO - Not implemented yet! + Parse a record """ - return True - - def _parse_record(self, metadata): - # Conc. NGAWest2 record info to identify each record in flatfile: - # --> event_id = NGAWest2['Earthquake Name'] - # --> station_id = NGAWest2['Station Name'] - # --> network_code = NGAWest2['Owner'] - # --> location_code = NGAWest2['Station ID No.'] wfid = "_".join([metadata["event_id"], metadata["network_code"], metadata["station_code"], metadata["location_code"]]) wfid = wfid.replace("-", "_") @@ -350,7 +321,6 @@ def _parse_record(self, metadata): xcomp, ycomp, vertical=vertical) - def _parse_event_data(self, metadata): """ Parses the event metadata @@ -358,12 +328,7 @@ def _parse_event_data(self, metadata): # ID and Name (name not in file so use ID again) eq_id = metadata["event_id"] eq_name = metadata["event_id"] - # Country - cntry_code = metadata["ev_nation_code"].strip() - if cntry_code and cntry_code in COUNTRY_CODES: - eq_country = COUNTRY_CODES[cntry_code] - else: - eq_country = None + # Date and time eq_datetime = valid.date_time(metadata["event_time"], "%Y-%m-%d %H:%M:%S") @@ -372,10 +337,10 @@ def _parse_event_data(self, metadata): eq_lon = valid.longitude(metadata["ev_longitude"]) eq_depth = valid.positive_float(metadata["ev_depth_km"], "ev_depth_km") if not eq_depth: - eq_depth = 0.0 + raise ValueError('Depth missing for one or more events in flatfile') eqk = Earthquake(eq_id, eq_name, eq_datetime, eq_lon, eq_lat, eq_depth, - None, # Magnitude not defined yet - eq_country=eq_country) + magnitude=None, eq_country=None) + # Get preferred magnitude and list pref_mag, magnitude_list = self._parse_magnitudes(metadata) eqk.magnitude = pref_mag @@ -524,7 +489,6 @@ def _parse_site_data(self, metadata): network_code = metadata["network_code"].strip() station_code = metadata["station_code"].strip() site_id = "{:s}-{:s}".format(network_code, station_code) - location_code = metadata["location_code"].strip() site_lon = valid.longitude(metadata["st_longitude"]) site_lat = valid.latitude(metadata["st_latitude"]) elevation = valid.vfloat(metadata["st_elevation"], "st_elevation") @@ -538,15 +502,9 @@ def _parse_site_data(self, metadata): vs30_measured = False else: vs30_measured = False - st_nation_code = metadata["st_nation_code"].strip() - if st_nation_code: - st_country = COUNTRY_CODES[st_nation_code] - else: - st_country = None site = RecordSite(site_id, station_code, station_code, site_lon, site_lat, elevation, vs30, vs30_measured, - network_code=network_code, - country=st_country) + network_code=network_code, country=None) site.slope = valid.vfloat(metadata["slope_deg"], "slope_deg") site.sensor_depth = valid.vfloat(metadata["sensor_depth_m"], "sensor_depth_m") @@ -680,7 +638,7 @@ def _parse_ground_motion(self, location, row, record, headers): def _retreive_ground_motion_from_row(self, row, header_list): """ - + Get the ground-motion data from a row (record) in the database """ imts = ["U", "V", "W", "rotD00", "rotD100", "rotD50"] spectra = [] @@ -715,7 +673,6 @@ def _retreive_ground_motion_from_row(self, row, header_list): values.append(np.fabs(float(value))) else: values.append(np.nan) - #values.append(np.fabs(float(row[header].strip()))) periods = np.array(periods) values = np.array(values) idx = np.argsort(periods) @@ -736,6 +693,7 @@ def _retreive_ground_motion_from_row(self, row, header_list): scalars["U"][key] * scalars["V"][key]) return scalars, spectra + def _get_ESM18_headers(NGAWest2, NGAWest2_vertical, Initial_NGAWest2_size): """ @@ -843,15 +801,10 @@ def _get_ESM18_headers(NGAWest2, NGAWest2_vertical, Initial_NGAWest2_size): # Non-GMIM headers "event_id":NGAWest2['event_id_reformatted'], "event_time":NGAWest2['event_time_reformatted'], - "ISC_ev_id":default_string, - "USGS_ev_id":default_string, - "INGV_ev_id":default_string, - "EMSC_ev_id":default_string, "ev_nation_code":default_nation_code, "ev_latitude":NGAWest2['Hypocenter Latitude (deg)'], "ev_longitude":NGAWest2['Hypocenter Longitude (deg)'], "ev_depth_km":NGAWest2['Hypocenter Depth (km)'], - "ev_hyp_ref":default_string, "fm_type_code":NGAWest2['fm_type'], "ML":default_string, "ML_ref":default_string, @@ -869,10 +822,8 @@ def _get_ESM18_headers(NGAWest2, NGAWest2_vertical, Initial_NGAWest2_size): "es_rake":NGAWest2['Rake Angle (deg)'], "es_strike_dip_rake_ref":default_string, "es_z_top":NGAWest2['Depth to Top Of Fault Rupture Model'], - "es_z_top_ref":default_string, - "es_length":default_string, - "es_width":default_string, - "es_geometry_ref":default_string, + "es_length":NGAWest2['Fault Rupture Length for Calculation of Ry (km)'], + "es_width":NGAWest2['Fault Rupture Width (km)'], "network_code": NGAWest2['Owner'], "station_code":NGAWest2['station_id'], @@ -881,19 +832,12 @@ def _get_ESM18_headers(NGAWest2, NGAWest2_vertical, Initial_NGAWest2_size): "sensor_depth_m":default_string, "proximity_code":default_string, "housing_code":default_string, - "installation_code":default_string, "st_nation_code":default_string, "st_latitude":NGAWest2['Station Latitude'], "st_longitude":NGAWest2['Station Longitude'], "st_elevation":default_string, - "ec8_code":default_string, - "ec8_code_method":default_string, - "ec8_code_ref":default_string, "vs30_m_sec":NGAWest2['Vs30 (m/s) selected for analysis'], - "vs30_ref":default_string, - "vs30_calc_method":default_string, - "vs30_meas_type":default_string, "slope_deg":default_string, "vs30_m_sec_WA":default_string, @@ -912,130 +856,97 @@ def _get_ESM18_headers(NGAWest2, NGAWest2_vertical, Initial_NGAWest2_size): "V_azimuth_deg":NGAWest2['H2 azimith (degrees)'], "W_channel_code":default_V_string, - "U_hp":default_string, - "V_hp":default_string, - "W_hp":default_string, - "U_lp":default_string, - "V_lp":default_string, - "W_lp":default_string, + "U_hp":NGAWest2['HP-H1 (Hz)'], + "V_hp":NGAWest2['HP-H2 (Hz)'], + "W_hp":NGAWest2_vertical['HP-V (Hz)'], + "U_lp":NGAWest2['LP-H1 (Hz)'], + "V_lp":NGAWest2['LP-H2 (Hz)'], + "W_lp":NGAWest2_vertical['LP-V (Hz)'], - "U_pga":default_string, - "V_pga":default_string, + # SMT uses GM of two horizontal components so place RotD50 here + "U_pga":NGAWest2['PGA (g)']*981, + "V_pga":NGAWest2['PGA (g)']*981, "W_pga":NGAWest2_vertical['PGA (g)']*981, - "rotD50_pga":NGAWest2['PGA (g)']*981, - "rotD100_pga":default_string, - "rotD00_pga":default_string, - "U_pgv":default_string, - "V_pgv":default_string, + "U_pgv":NGAWest2['PGV (cm/sec)'], + "V_pgv":NGAWest2['PGV (cm/sec)'], "W_pgv":NGAWest2_vertical['PGV (cm/sec)'], - "rotD50_pgv":NGAWest2['PGV (cm/sec)'], - "rotD100_pgv":default_string, - "rotD00_pgv":default_string, - "U_pgd":default_string, - "V_pgd":default_string, + "U_pgd":NGAWest2['PGD (cm)'], + "V_pgd":NGAWest2['PGD (cm)'], "W_pgd":NGAWest2_vertical['PGD (cm)'], - "rotD50_pgd":NGAWest2['PGD (cm)'], - "rotD100_pgd":default_string, - "rotD00_pgd":default_string, - "U_T90":default_string, - "V_T90":default_string, - "W_T90":default_string, - "rotD50_T90":default_string, - "rotD100_T90":default_string, - "rotD00_T90":default_string, - "U_housner":default_string, - "V_housner":default_string, - "W_housner":default_string, - "rotD50_housner":default_string, - "rotD100_housner":default_string, - "rotD00_housner":default_string, - "U_CAV":default_string, - "V_CAV":default_string, - "W_CAV":default_string, - "rotD50_CAV":default_string, - "rotD100_CAV":default_string, - "rotD00_CAV":default_string, - "U_ia":default_string, - "V_ia":default_string, - "W_ia":default_string, - "rotD50_ia":default_string, - "rotD100_ia":default_string, - "rotD00_ia":default_string, + "U_T0_010":NGAWest2['T0.010S']*981, + "U_T0_025":NGAWest2['T0.025S']*981, + "U_T0_040":NGAWest2['T0.040S']*981, + "U_T0_050":NGAWest2['T0.050S']*981, + "U_T0_070":NGAWest2['T0.070S']*981, + "U_T0_100":NGAWest2['T0.100S']*981, + "U_T0_150":NGAWest2['T0.150S']*981, + "U_T0_200":NGAWest2['T0.200S']*981, + "U_T0_250":NGAWest2['T0.250S']*981, + "U_T0_300":NGAWest2['T0.300S']*981, + "U_T0_350":NGAWest2['T0.350S']*981, + "U_T0_400":NGAWest2['T0.400S']*981, + "U_T0_450":NGAWest2['T0.450S']*981, + "U_T0_500":NGAWest2['T0.500S']*981, + "U_T0_600":NGAWest2['T0.600S']*981, + "U_T0_700":NGAWest2['T0.700S']*981, + "U_T0_750":NGAWest2['T0.750S']*981, + "U_T0_800":NGAWest2['T0.800S']*981, + "U_T0_900":NGAWest2['T0.900S']*981, + "U_T1_000":NGAWest2['T1.000S']*981, + "U_T1_200":NGAWest2['T1.200S']*981, + "U_T1_400":NGAWest2['T1.400S']*981, + "U_T1_600":NGAWest2['T1.600S']*981, + "U_T1_800":NGAWest2['T1.800S']*981, + "U_T2_000":NGAWest2['T2.000S']*981, + "U_T2_500":NGAWest2['T2.500S']*981, + "U_T3_000":NGAWest2['T3.000S']*981, + "U_T3_500":NGAWest2['T3.500S']*981, + "U_T4_000":NGAWest2['T4.000S']*981, + "U_T4_500":NGAWest2['T4.500S']*981, + "U_T5_000":NGAWest2['T5.000S']*981, + "U_T6_000":NGAWest2['T6.000S']*981, + "U_T7_000":NGAWest2['T7.000S']*981, + "U_T8_000":NGAWest2['T8.000S']*981, + "U_T9_000":NGAWest2['T9.000S']*981, + "U_T10_000":NGAWest2['T10.000S']*981, - "U_T0_010":default_string, - "U_T0_025":default_string, - "U_T0_040":default_string, - "U_T0_050":default_string, - "U_T0_070":default_string, - "U_T0_100":default_string, - "U_T0_150":default_string, - "U_T0_200":default_string, - "U_T0_250":default_string, - "U_T0_300":default_string, - "U_T0_350":default_string, - "U_T0_400":default_string, - "U_T0_450":default_string, - "U_T0_500":default_string, - "U_T0_600":default_string, - "U_T0_700":default_string, - "U_T0_750":default_string, - "U_T0_800":default_string, - "U_T0_900":default_string, - "U_T1_000":default_string, - "U_T1_200":default_string, - "U_T1_400":default_string, - "U_T1_600":default_string, - "U_T1_800":default_string, - "U_T2_000":default_string, - "U_T2_500":default_string, - "U_T3_000":default_string, - "U_T3_500":default_string, - "U_T4_000":default_string, - "U_T4_500":default_string, - "U_T5_000":default_string, - "U_T6_000":default_string, - "U_T7_000":default_string, - "U_T8_000":default_string, - "U_T9_000":default_string, - "U_T10_000":default_string, - - "V_T0_010":default_string, - "V_T0_025":default_string, - "V_T0_040":default_string, - "V_T0_050":default_string, - "V_T0_070":default_string, - "V_T0_100":default_string, - "V_T0_150":default_string, - "V_T0_200":default_string, - "V_T0_250":default_string, - "V_T0_300":default_string, - "V_T0_350":default_string, - "V_T0_400":default_string, - "V_T0_450":default_string, - "V_T0_500":default_string, - "V_T0_600":default_string, - "V_T0_700":default_string, - "V_T0_750":default_string, - "V_T0_800":default_string, - "V_T0_900":default_string, - "V_T1_000":default_string, - "V_T1_200":default_string, - "V_T1_400":default_string, - "V_T1_600":default_string, - "V_T1_800":default_string, - "V_T2_000":default_string, - "V_T2_500":default_string, - "V_T3_000":default_string, - "V_T3_500":default_string, - "V_T4_000":default_string, - "V_T4_500":default_string, - "V_T5_000":default_string, - "V_T6_000":default_string, - "V_T7_000":default_string, - "V_T8_000":default_string, - "V_T9_000":default_string, - "V_T10_000":default_string, + "V_T0_010":NGAWest2['T0.010S']*981, + "V_T0_025":NGAWest2['T0.025S']*981, + "V_T0_040":NGAWest2['T0.040S']*981, + "V_T0_050":NGAWest2['T0.050S']*981, + "V_T0_070":NGAWest2['T0.070S']*981, + "V_T0_100":NGAWest2['T0.100S']*981, + "V_T0_150":NGAWest2['T0.150S']*981, + "V_T0_200":NGAWest2['T0.200S']*981, + "V_T0_250":NGAWest2['T0.250S']*981, + "V_T0_300":NGAWest2['T0.300S']*981, + "V_T0_350":NGAWest2['T0.350S']*981, + "V_T0_400":NGAWest2['T0.400S']*981, + "V_T0_450":NGAWest2['T0.450S']*981, + "V_T0_500":NGAWest2['T0.500S']*981, + "V_T0_600":NGAWest2['T0.600S']*981, + "V_T0_700":NGAWest2['T0.700S']*981, + "V_T0_750":NGAWest2['T0.750S']*981, + "V_T0_800":NGAWest2['T0.800S']*981, + "V_T0_900":NGAWest2['T0.900S']*981, + "V_T1_000":NGAWest2['T1.000S']*981, + "V_T1_200":NGAWest2['T1.200S']*981, + "V_T1_400":NGAWest2['T1.400S']*981, + "V_T1_600":NGAWest2['T1.600S']*981, + "V_T1_800":NGAWest2['T1.800S']*981, + "V_T2_000":NGAWest2['T2.000S']*981, + "V_T2_500":NGAWest2['T2.500S']*981, + "V_T3_000":NGAWest2['T3.000S']*981, + "V_T3_500":NGAWest2['T3.500S']*981, + "V_T4_000":NGAWest2['T4.000S']*981, + "V_T4_500":NGAWest2['T4.500S']*981, + "V_T5_000":NGAWest2['T5.000S']*981, + "V_T6_000":NGAWest2['T6.000S']*981, + "V_T7_000":NGAWest2['T7.000S']*981, + "V_T8_000":NGAWest2['T8.000S']*981, + "V_T9_000":NGAWest2['T9.000S']*981, + "V_T10_000":NGAWest2['T10.000S']*981, "W_T0_010":NGAWest2_vertical['T0.010S']*981, "W_T0_025":NGAWest2_vertical['T0.025S']*981, @@ -1072,118 +983,7 @@ def _get_ESM18_headers(NGAWest2, NGAWest2_vertical, Initial_NGAWest2_size): "W_T7_000":NGAWest2_vertical['T7.000S']*981, "W_T8_000":NGAWest2_vertical['T8.000S']*981, "W_T9_000":NGAWest2_vertical['T9.000S']*981, - "W_T10_000":NGAWest2_vertical['T10.000S']*981, - - "rotD50_T0_010":NGAWest2['T0.010S']*981, - "rotD50_T0_025":NGAWest2['T0.025S']*981, - "rotD50_T0_040":NGAWest2['T0.040S']*981, - "rotD50_T0_050":NGAWest2['T0.050S']*981, - "rotD50_T0_070":NGAWest2['T0.070S']*981, - "rotD50_T0_100":NGAWest2['T0.100S']*981, - "rotD50_T0_150":NGAWest2['T0.150S']*981, - "rotD50_T0_200":NGAWest2['T0.200S']*981, - "rotD50_T0_250":NGAWest2['T0.250S']*981, - "rotD50_T0_300":NGAWest2['T0.300S']*981, - "rotD50_T0_350":NGAWest2['T0.350S']*981, - "rotD50_T0_400":NGAWest2['T0.400S']*981, - "rotD50_T0_450":NGAWest2['T0.450S']*981, - "rotD50_T0_500":NGAWest2['T0.500S']*981, - "rotD50_T0_600":NGAWest2['T0.600S']*981, - "rotD50_T0_700":NGAWest2['T0.700S']*981, - "rotD50_T0_750":NGAWest2['T0.750S']*981, - "rotD50_T0_800":NGAWest2['T0.800S']*981, - "rotD50_T0_900":NGAWest2['T0.900S']*981, - "rotD50_T1_000":NGAWest2['T1.000S']*981, - "rotD50_T1_200":NGAWest2['T1.200S']*981, - "rotD50_T1_400":NGAWest2['T1.400S']*981, - "rotD50_T1_600":NGAWest2['T1.600S']*981, - "rotD50_T1_800":NGAWest2['T1.800S']*981, - "rotD50_T2_000":NGAWest2['T2.000S']*981, - "rotD50_T2_500":NGAWest2['T2.500S']*981, - "rotD50_T3_000":NGAWest2['T3.000S']*981, - "rotD50_T3_500":NGAWest2['T3.500S']*981, - "rotD50_T4_000":NGAWest2['T4.000S']*981, - "rotD50_T4_500":NGAWest2['T4.500S']*981, - "rotD50_T5_000":NGAWest2['T5.000S']*981, - "rotD50_T6_000":NGAWest2['T6.000S']*981, - "rotD50_T7_000":NGAWest2['T7.000S']*981, - "rotD50_T8_000":NGAWest2['T8.000S']*981, - "rotD50_T9_000":NGAWest2['T9.000S']*981, - "rotD50_T10_000":NGAWest2['T10.000S']*981, - - "rotD00_T0_010":default_string, - "rotD00_T0_025":default_string, - "rotD00_T0_040":default_string, - "rotD00_T0_050":default_string, - "rotD00_T0_070":default_string, - "rotD00_T0_100":default_string, - "rotD00_T0_150":default_string, - "rotD00_T0_200":default_string, - "rotD00_T0_250":default_string, - "rotD00_T0_300":default_string, - "rotD00_T0_350":default_string, - "rotD00_T0_400":default_string, - "rotD00_T0_450":default_string, - "rotD00_T0_500":default_string, - "rotD00_T0_600":default_string, - "rotD00_T0_700":default_string, - "rotD00_T0_750":default_string, - "rotD00_T0_800":default_string, - "rotD00_T0_900":default_string, - "rotD00_T1_000":default_string, - "rotD00_T1_200":default_string, - "rotD00_T1_400":default_string, - "rotD00_T1_600":default_string, - "rotD00_T1_800":default_string, - "rotD00_T2_000":default_string, - "rotD00_T2_500":default_string, - "rotD00_T3_000":default_string, - "rotD00_T3_500":default_string, - "rotD00_T4_000":default_string, - "rotD00_T4_500":default_string, - "rotD00_T5_000":default_string, - "rotD00_T6_000":default_string, - "rotD00_T7_000":default_string, - "rotD00_T8_000":default_string, - "rotD00_T9_000":default_string, - "rotD00_T10_000":default_string, - - "rotD100_T0_010":default_string, - "rotD100_T0_025":default_string, - "rotD100_T0_040":default_string, - "rotD100_T0_050":default_string, - "rotD100_T0_070":default_string, - "rotD100_T0_100":default_string, - "rotD100_T0_150":default_string, - "rotD100_T0_200":default_string, - "rotD100_T0_250":default_string, - "rotD100_T0_300":default_string, - "rotD100_T0_350":default_string, - "rotD100_T0_400":default_string, - "rotD100_T0_450":default_string, - "rotD100_T0_500":default_string, - "rotD100_T0_600":default_string, - "rotD100_T0_700":default_string, - "rotD100_T0_750":default_string, - "rotD100_T0_800":default_string, - "rotD100_T0_900":default_string, - "rotD100_T1_000":default_string, - "rotD100_T1_200":default_string, - "rotD100_T1_400":default_string, - "rotD100_T1_600":default_string, - "rotD100_T1_800":default_string, - "rotD100_T2_000":default_string, - "rotD100_T2_500":default_string, - "rotD100_T3_000":default_string, - "rotD100_T3_500":default_string, - "rotD100_T4_000":default_string, - "rotD100_T4_500":default_string, - "rotD100_T5_000":default_string, - "rotD100_T6_000":default_string, - "rotD100_T7_000":default_string, - "rotD100_T8_000":default_string, - "rotD100_T9_000":default_string, - "rotD100_T10_000":default_string}) + "W_T10_000":NGAWest2_vertical['T10.000S']*981}) # Output to folder where converted flatfile read into parser DATA = os.path.abspath('') diff --git a/openquake/smt/parsers/sigma_database_parser.py b/openquake/smt/parsers/sigma_database_parser.py index 5a297a6db..40549872d 100755 --- a/openquake/smt/parsers/sigma_database_parser.py +++ b/openquake/smt/parsers/sigma_database_parser.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -44,10 +44,9 @@ class SigmaDatabaseMetadataReader(SMDatabaseReader): def parse(self): """ - + Parse the database """ file_list = os.listdir(self.filename) - num_files = len(file_list) self.database = GroundMotionDatabase(self.id, self.name) for file_str in file_list: if "DS_Store" in file_str: @@ -177,7 +176,7 @@ def _parse_focal_mechanism(self, eq_id, eq_name, metadata): def _parse_rupture(self, eq_id, eq_name, magnitude, metadata): """ - + Parse rupture """ return Rupture(eq_id, eq_name, @@ -345,9 +344,8 @@ def parse_records(self, record=None): def _parse_time_history(self, ifile): """ - + Parse time history """ - output = {} acc_hist = [] cases = {0: self._get_datetime, @@ -454,14 +452,11 @@ def _get_pga(self, output, line): def _get_units(self, output, line): """ - + # Get units """ output['Units'] = "cm/s/s" return output def _get_timehist_line(self, line): - """ - - """ idx = np.arange(0, len(line) + 14, 14) - return [float(line[idx[i]:idx[i + 1]]) for i in range(len(idx) - 1)] + return [float(line[idx[i]:idx[i + 1]]) for i in range(len(idx) - 1)] \ No newline at end of file diff --git a/openquake/smt/parsers/simple_flatfile_parser.py b/openquake/smt/parsers/simple_flatfile_parser.py index 048f33109..4e6099743 100755 --- a/openquake/smt/parsers/simple_flatfile_parser.py +++ b/openquake/smt/parsers/simple_flatfile_parser.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -198,9 +198,6 @@ def _parse_event_data(self, metadata): length, width, ztor) - # get_float(metadata["Fault Rupture Length (km)"]), - # get_float(metadata["Fault Rupture Width (km)"]), - # get_float(metadata["Depth to Top Of Fault Rupture Model"])) eqk.rupture.get_area() return eqk @@ -234,7 +231,8 @@ def _parse_distance_data(self, event, site, metadata): :class: openquake.smt.sm_database.RecordDistance """ # Compute various distance metrics - # Add calculation of Repi, Rhypo from event and station localizations (latitudes, longitudes, depth, elevation)? + # Add calculation of Repi, Rhypo from event and station localizations + # (latitudes, longitudes, depth, elevation)? target_site = Mesh(np.array([site.longitude]), np.array([site.latitude]), np.array([0.0])) @@ -281,13 +279,6 @@ def _parse_distance_data(self, event, site, metadata): ry0 = float(Ry0)) distance.azimuth = get_float(metadata["Source to Site Azimuth (deg)"]) distance.hanging_wall = get_float(metadata["FW/HW Indicator"]) -# distance = RecordDistance( -# get_float(metadata["EpiD (km)"]), -# get_float(metadata["HypD (km)"]), -# get_float(metadata["Joyner-Boore Dist. (km)"]), -# get_float(metadata["Campbell R Dist. (km)"])) -# distance.azimuth = get_float(metadata["Source to Site Azimuth (deg)"]) -# distance.hanging_wall = get_float(metadata["FW/HW Indicator"]) return distance def _parse_site_data(self, metadata): @@ -411,4 +402,4 @@ def _parse_time_history(self, ifile): output["Number Steps"] = int(nvals) output["Units"] = "cm/s/s" output["PGA"] = np.max(np.fabs(output["Acceleration"])) - return output + return output \ No newline at end of file diff --git a/openquake/smt/parsers/simple_flatfile_parser_sara.py b/openquake/smt/parsers/simple_flatfile_parser_sara.py index f7f56cb0b..3eb55e1f5 100755 --- a/openquake/smt/parsers/simple_flatfile_parser_sara.py +++ b/openquake/smt/parsers/simple_flatfile_parser_sara.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -574,8 +574,6 @@ def _parse_processing_data(self, wfid, metadata): return xcomp, ycomp, None - - class NearFaultFlatFileParser(SimpleFlatfileParserV9): def parse(self): @@ -731,4 +729,4 @@ def _parse_time_history(self, ifile, units="cm/s/s"): output["Number Steps"] = int(nvals) output["Units"] = units output["PGA"] = np.max(np.fabs(output["Acceleration"])) - return output + return output \ No newline at end of file diff --git a/openquake/smt/parsers/valid.py b/openquake/smt/parsers/valid.py index 345d22a14..f9008b9a2 100644 --- a/openquake/smt/parsers/valid.py +++ b/openquake/smt/parsers/valid.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -153,4 +153,4 @@ def rake(value): if rake and (rake >= -180.0) and (rake <= 180.0): return rake print("Rake %s is not in range -180 - 180" % value) - return None + return None \ No newline at end of file diff --git a/openquake/smt/residuals/context_db.py b/openquake/smt/residuals/context_db.py index 66819753b..643e3d2f0 100644 --- a/openquake/smt/residuals/context_db.py +++ b/openquake/smt/residuals/context_db.py @@ -1,3 +1,20 @@ +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill +# +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with OpenQuake. If not, see . """ Module defining the interface of a Context Database (ContextDB), a database of data capable of yielding Contexts and Observations suitable for Residual analysis diff --git a/openquake/smt/residuals/gmpe_residuals.py b/openquake/smt/residuals/gmpe_residuals.py index 9464ace51..dfcb5698a 100755 --- a/openquake/smt/residuals/gmpe_residuals.py +++ b/openquake/smt/residuals/gmpe_residuals.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -59,7 +59,6 @@ def get_geometric_mean(fle): :param fle: Instance of :class: h5py.File """ - # periods = fle["IMS/X/Spectra/Response/Periods"].value if not ("H" in fle["IMS"].keys()): # Horizontal spectra not in record x_spc = fle["IMS/X/Spectra/Response/Acceleration/damping_05"].values @@ -297,7 +296,6 @@ def __init__(self, gmpe_list, imts): :param imts: A list e.g. ['PGA', 'SA(0.1)', 'SA(1.0)'] """ - # Residuals object self.gmpe_list = check_gsim_list(gmpe_list) self.number_gmpes = len(self.gmpe_list) @@ -312,8 +310,8 @@ def __init__(self, gmpe_list, imts): gmpe_dict_1 = OrderedDict([]) gmpe_dict_2 = OrderedDict([]) self.unique_indices[gmpe] = {} + # Get the period range and the coefficient types - # gmpe_i = GSIM_LIST[gmpe]() gmpe_i = self.gmpe_list[gmpe] for c in dir(gmpe_i): if 'COEFFS' in c: @@ -348,6 +346,7 @@ def __init__(self, gmpe_list, imts): gmpe_dict_2[imtx][res_type] = [] self.types[gmpe][imtx].append(res_type) gmpe_dict_2[imtx]["Mean"] = [] + # For handling of GMPEs with total sigma only else: for res_type in self.gmpe_list[ @@ -369,7 +368,7 @@ def __init__(self, gmpe_list, imts): def from_toml(cls, filename): """ Read in gmpe_list and imts from .toml file. This method allows use of - non-ergodic GMPEs and gmpes with additional input files within SMT + gmpes with additional parameters and input files within the SMT """ # Read in toml file with dict of gmpes and subdict of imts config_file = toml.load(filename) @@ -403,7 +402,7 @@ def get_residuals(self, ctx_database, nodal_plane_index=1, See e.g., :class:`openquake.smt.sm_database.GroundMotionDatabase` for an example """ - + # Get contexts contexts = ctx_database.get_contexts(nodal_plane_index, self.imts, component) @@ -436,8 +435,8 @@ def get_residuals(self, ctx_database, nodal_plane_index=1, continue for res_type in self.residuals[gmpe][imtx].keys(): if res_type == "Inter event": - inter_ev = \ - context["Residual"][gmpe][imtx][res_type] + inter_ev = context["Residual"][gmpe][imtx][ + res_type] if np.all( np.fabs(inter_ev - inter_ev[0]) < 1.0E-12): # Single inter-event residual @@ -480,28 +479,28 @@ def get_expected_motions(self, context): """ Calculate the expected ground motions from the context """ - # TODO Rake hack will be removed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! if not context["Ctx"].rake: - context["Ctx"].rake = 0.0 + context["Ctx"].rake = 0.0 # Assume strike-slip expected = OrderedDict([(gmpe, {}) for gmpe in self.gmpe_list]) # Period range for GSIM for gmpe in self.gmpe_list: expected[gmpe] = OrderedDict([(imtx, {}) for imtx in self.imts]) for imtx in self.imts: gsim = self.gmpe_list[gmpe] - gsim_orig = gsim # If gsim into mgmpe retain pot. region for ctx + gsim_orig = gsim # Retain before using mgmpe if "SA(" in imtx: period = imt.from_string(imtx).period if period < self.gmpe_sa_limits[gmpe][0] or\ period > self.gmpe_sa_limits[gmpe][1]: expected[gmpe][imtx] = None continue - # Check if gsim needs appending with mgmpe + # Check if gsim needs modifying with mgmpe gsim = mgmpe_check(gsim) # Add region parameter to sites context if specified in gsim if 'eshm20_region' in gsim_orig.kwargs: context["Ctx"].region = gsim_orig.kwargs['eshm20_region'] - if 'region' in gsim_orig.kwargs and 'eshm20_region' not in gsim.kwargs: + if ('region' in gsim_orig.kwargs and + 'eshm20_region' not in gsim.kwargs): context["Ctx"].region = gsim_orig.kwargs['region'] # Get expected motions mean, stddev = gsim.get_mean_and_stddevs( @@ -1220,7 +1219,7 @@ def __init__(self, site_id_list, gmpe_list, imts): def from_toml(cls, site_id_list, filename): """ Read in gmpe_list and imts from .toml file. This method allows use of - non-ergodic GMPEs and gmpes with additional input files within SMT + gmpes with additional parameters and input files within the SMT """ # Read in toml file with dict of gmpes and subdict of imts config_file = toml.load(filename) diff --git a/openquake/smt/residuals/residual_plots.py b/openquake/smt/residuals/residual_plots.py index b7187b1fd..e91f2e765 100755 --- a/openquake/smt/residuals/residual_plots.py +++ b/openquake/smt/residuals/residual_plots.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/residuals/residual_plotter.py b/openquake/smt/residuals/residual_plotter.py index 839c6abc2..60b7926b4 100755 --- a/openquake/smt/residuals/residual_plotter.py +++ b/openquake/smt/residuals/residual_plotter.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/response_spectrum.py b/openquake/smt/response_spectrum.py index 23e320299..fdafec747 100755 --- a/openquake/smt/response_spectrum.py +++ b/openquake/smt/response_spectrum.py @@ -1,8 +1,7 @@ -#!/usr/bin/env python # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,7 +15,6 @@ # # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . - """ Simple Python Script to integrate a strong motion record using the Newmark-Beta method diff --git a/openquake/smt/sm_database.py b/openquake/smt/sm_database.py index cd6684221..f73a47041 100755 --- a/openquake/smt/sm_database.py +++ b/openquake/smt/sm_database.py @@ -1,8 +1,7 @@ -#!/usr/bin/env python # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,7 +15,6 @@ # # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . - """ Basic Pseudo-database built on top of hdf5 for a set of processed strong motion records @@ -995,7 +993,8 @@ def get_event_and_records(self): SCALAR_IMTS = ["PGA", "PGV"] def get_observations(self, imtx, records, component="Geometric"): - """Return observed values for the given imt, as numpy array. + """ + Return observed values for the given imt, as numpy array. See superclass docstring for details """ @@ -1236,7 +1235,7 @@ def _get_event_id_list(self): def _get_site_id(self, str_id): """ - TODO + Get site id """ if str_id not in self.site_ids: self.site_ids.append(str_id) diff --git a/openquake/smt/sm_database_builder.py b/openquake/smt/sm_database_builder.py index 627ebbcca..34dab6db9 100755 --- a/openquake/smt/sm_database_builder.py +++ b/openquake/smt/sm_database_builder.py @@ -1,8 +1,7 @@ -#!/usr/bin/env python # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2018 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,7 +15,6 @@ # # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . - """ Constructs the HDF5 database """ diff --git a/openquake/smt/sm_utils.py b/openquake/smt/sm_utils.py index b86f4d933..93c61e090 100755 --- a/openquake/smt/sm_utils.py +++ b/openquake/smt/sm_utils.py @@ -1,8 +1,7 @@ -#!/usr/bin/env python # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/smoothing/base.py b/openquake/smt/smoothing/base.py index 0e621e700..48b59a71d 100755 --- a/openquake/smt/smoothing/base.py +++ b/openquake/smt/smoothing/base.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/smoothing/konno_ohmachi.py b/openquake/smt/smoothing/konno_ohmachi.py index d4078ba89..ae11e45c6 100755 --- a/openquake/smt/smoothing/konno_ohmachi.py +++ b/openquake/smt/smoothing/konno_ohmachi.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/strong_motion_selector.py b/openquake/smt/strong_motion_selector.py index b95a6f704..cb149f360 100755 --- a/openquake/smt/strong_motion_selector.py +++ b/openquake/smt/strong_motion_selector.py @@ -1,8 +1,7 @@ -#!/usr/bin/env python # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,7 +15,6 @@ # # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . - """ Strong motion record selection tools """ diff --git a/openquake/smt/surface_utils.py b/openquake/smt/surface_utils.py index 2f027ecb4..455d4629d 100755 --- a/openquake/smt/surface_utils.py +++ b/openquake/smt/surface_utils.py @@ -1,8 +1,7 @@ -#!/usr/bin/env python # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/tests/comparison/comparison_test.py b/openquake/smt/tests/comparison/comparison_test.py index 19ed2313d..32f207270 100644 --- a/openquake/smt/tests/comparison/comparison_test.py +++ b/openquake/smt/tests/comparison/comparison_test.py @@ -1,30 +1,20 @@ -# ------------------- The OpenQuake Model Building Toolkit -------------------- -# Copyright (C) 2022 GEM Foundation -# _______ _______ __ __ _______ _______ ___ _ -# | || | | |_| || _ || || | | | -# | _ || _ | ____ | || |_| ||_ _|| |_| | -# | | | || | | ||____|| || | | | | _| -# | |_| || |_| | | || _ | | | | |_ -# | || | | ||_|| || |_| | | | | _ | -# |_______||____||_| |_| |_||_______| |___| |___| |_| +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2014 GEM Foundation # -# This program is free software: you can redistribute it and/or modify it under -# the terms of the GNU Affero General Public License as published by the Free -# Software Foundation, either version 3 of the License, or (at your option) any -# later version. +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. # -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS -# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more -# details. +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. # # You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see . -# ----------------------------------------------------------------------------- -# vim: tabstop=4 shiftwidth=4 softtabstop=4 -# coding: utf-8 - - +# along with OpenQuake. If not, see . """ Tests for execution of comparison module """ diff --git a/openquake/smt/tests/comparison/data/compare_gmpe_inputs.toml b/openquake/smt/tests/comparison/data/compare_gmpe_inputs.toml index 968f8335b..c10e25c2f 100644 --- a/openquake/smt/tests/comparison/data/compare_gmpe_inputs.toml +++ b/openquake/smt/tests/comparison/data/compare_gmpe_inputs.toml @@ -7,7 +7,7 @@ minR = 0 # min dist. used in trellis, Sammon's, clusters and matrix plots maxR = 300 # max dist. used in trellis, Sammon's, clusters and matrix plots dist_type = 'rrup' # specify distance metric used in trellis plots dist_list = [10, 50, 200] # distances for use in response spectra + sigma wrt period plots -eshm20_region = 0 # region for ESHM20 GMMs +eshm20_region = 0 # region for ESHM20 GMM Nstd = 2 # number of standard deviations to sample from sigma distribution [site_properties] @@ -22,7 +22,7 @@ trt = 'ASCR' ztor = 'None' # Set to string of 'None' to NOT consider strike = -999 dip = -999 -rake = 60 +rake = 60 # Must be provided. Strike and dip can be approximated if set to -999 aratio = 2 trellis_and_rs_mag_list = [5,6,7] # mags used only for trellis trellis_and_rs_depths = [20,25,30] # depth per magnitude for trellis diff --git a/openquake/smt/tests/comparison/data/mgmpe_inputs.toml b/openquake/smt/tests/comparison/data/mgmpe_inputs.toml index 0975d84ba..b265e2a86 100644 --- a/openquake/smt/tests/comparison/data/mgmpe_inputs.toml +++ b/openquake/smt/tests/comparison/data/mgmpe_inputs.toml @@ -7,7 +7,7 @@ minR = 0 # min dist. used in trellis, Sammon's, clusters and matrix plots maxR = 300 # max dist. used in trellis, Sammon's, clusters and matrix plots dist_type = 'rrup' # specify distance metric used in trellis plots dist_list = [10, 50, 200] # distances for use in response spectra + sigma wrt period plots -eshm20_region = 0 # region for ESHM20 GMMs +eshm20_region = 0 # region for ESHM20 GMM Nstd = 1 # number of standard deviations to sample from sigma distribution [site_properties] @@ -22,7 +22,7 @@ trt = 'ASCR' ztor = 'None' # Set to string of 'None' to NOT consider strike = -999 dip = -999 -rake = 60 +rake = 60 # Must be provided. Strike and dip can be approximated if set to -999 aratio = 2 trellis_and_rs_mag_list = [5,6,7] # mags used only for trellis trellis_and_rs_depths = [20,25,30] # depth per magnitude for trellis diff --git a/openquake/smt/tests/comparison/mgmpe_from_toml_test.py b/openquake/smt/tests/comparison/mgmpe_from_toml_test.py index f566ffffc..43d11a24b 100644 --- a/openquake/smt/tests/comparison/mgmpe_from_toml_test.py +++ b/openquake/smt/tests/comparison/mgmpe_from_toml_test.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2023 GEM Foundation +# Copyright (C) 2014-2024 GEM Foundation # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/tests/database_io_test.py b/openquake/smt/tests/database_io_test.py index 377610105..b6d0c2b12 100644 --- a/openquake/smt/tests/database_io_test.py +++ b/openquake/smt/tests/database_io_test.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/tests/intensity_measures_test.py b/openquake/smt/tests/intensity_measures_test.py index d713eba8e..807853074 100644 --- a/openquake/smt/tests/intensity_measures_test.py +++ b/openquake/smt/tests/intensity_measures_test.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/tests/parsers/data/ESM23_Greece_test.csv b/openquake/smt/tests/parsers/data/ESM_URL_Greece_test.csv similarity index 100% rename from openquake/smt/tests/parsers/data/ESM23_Greece_test.csv rename to openquake/smt/tests/parsers/data/ESM_URL_Greece_test.csv diff --git a/openquake/smt/tests/parsers/data/ESM22_Albania_filtered_test.csv b/openquake/smt/tests/parsers/data/ESM_WS_Albania_filtered_test.csv similarity index 100% rename from openquake/smt/tests/parsers/data/ESM22_Albania_filtered_test.csv rename to openquake/smt/tests/parsers/data/ESM_WS_Albania_filtered_test.csv diff --git a/openquake/smt/tests/parsers/data/GEM_flatfile_test.csv b/openquake/smt/tests/parsers/data/GEM_flatfile_test.csv new file mode 100644 index 000000000..648abe432 --- /dev/null +++ b/openquake/smt/tests/parsers/data/GEM_flatfile_test.csv @@ -0,0 +1,6 @@ +,orig_db,orig_db_license,eq_st_combo,orig_db_rec_identifier,event_id,event_time,esm_ev_id,ISC_ev_id,USGS_ev_id,INGV_ev_id,EMSC_ev_id,ev_nation_code,ev_latitude,ev_longitude,ev_depth_km,fm_type_code,ML,ML_ref,Mw,Mw_ref,Ms,Ms_ref,event_source_id,es_strike,es_dip,es_rake,es_strike_dip_rake_ref,es_z_top,es_length,es_width,network_code,station_code_orig,station_code,location_code,instrument_code,sensor_depth_m,housing_code,installation_code,st_nation_code,st_latitude,st_longitude,st_elevation,vs30_m_sec,vs30_meas_type,slope_deg,epi_dist,epi_az,JB_dist,rup_dist,Rx_dist,Ry0_dist,late_triggered_flag_01,U_channel_code,U_azimuth_deg,V_channel_code,V_azimuth_deg,W_channel_code,U_hp,V_hp,W_hp,U_lp,V_lp,W_lp,rotd50_flag,Shortest_Usable_Period_for_PSA_Ave_Over_Components,Longest_Usable_Period_for_PSA_Ave_Over_Components,U_pga,V_pga,W_pga,rotD50_pga,rotD100_pga,rotD00_pga,U_pgv,V_pgv,W_pgv,rotD50_pgv,rotD100_pgv,rotD00_pgv,U_pgd,V_pgd,W_pgd,rotD50_pgd,rotD100_pgd,rotD00_pgd,U_T90,V_T90,W_T90,rotD50_T90,rotD100_T90,rotD00_T90,U_housner,V_housner,W_housner,rotD50_housner,rotD100_housner,rotD00_housner,U_CAV,V_CAV,W_CAV,rotD50_CAV,rotD100_CAV,rotD00_CAV,U_ia,V_ia,W_ia,rotD50_ia,rotD100_ia,rotD00_ia,U_T0_010,U_T0_025,U_T0_040,U_T0_050,U_T0_070,U_T0_100,U_T0_150,U_T0_200,U_T0_250,U_T0_300,U_T0_350,U_T0_400,U_T0_450,U_T0_500,U_T0_600,U_T0_700,U_T0_750,U_T0_800,U_T0_900,U_T1_000,U_T1_200,U_T1_400,U_T1_600,U_T1_800,U_T2_000,U_T2_500,U_T3_000,U_T3_500,U_T4_000,U_T4_500,U_T5_000,U_T6_000,U_T7_000,U_T8_000,U_T9_000,U_T10_000,V_T0_010,V_T0_025,V_T0_040,V_T0_050,V_T0_070,V_T0_100,V_T0_150,V_T0_200,V_T0_250,V_T0_300,V_T0_350,V_T0_400,V_T0_450,V_T0_500,V_T0_600,V_T0_700,V_T0_750,V_T0_800,V_T0_900,V_T1_000,V_T1_200,V_T1_400,V_T1_600,V_T1_800,V_T2_000,V_T2_500,V_T3_000,V_T3_500,V_T4_000,V_T4_500,V_T5_000,V_T6_000,V_T7_000,V_T8_000,V_T9_000,V_T10_000,W_T0_010,W_T0_025,W_T0_040,W_T0_050,W_T0_070,W_T0_100,W_T0_150,W_T0_200,W_T0_250,W_T0_300,W_T0_350,W_T0_400,W_T0_450,W_T0_500,W_T0_600,W_T0_700,W_T0_750,W_T0_800,W_T0_900,W_T1_000,W_T1_200,W_T1_400,W_T1_600,W_T1_800,W_T2_000,W_T2_500,W_T3_000,W_T3_500,W_T4_000,W_T4_500,W_T5_000,W_T6_000,W_T7_000,W_T8_000,W_T9_000,W_T10_000,rotD50_T0_010,rotD50_T0_025,rotD50_T0_040,rotD50_T0_050,rotD50_T0_070,rotD50_T0_100,rotD50_T0_150,rotD50_T0_200,rotD50_T0_250,rotD50_T0_300,rotD50_T0_350,rotD50_T0_400,rotD50_T0_450,rotD50_T0_500,rotD50_T0_600,rotD50_T0_700,rotD50_T0_750,rotD50_T0_800,rotD50_T0_900,rotD50_T1_000,rotD50_T1_200,rotD50_T1_400,rotD50_T1_600,rotD50_T1_800,rotD50_T2_000,rotD50_T2_500,rotD50_T3_000,rotD50_T3_500,rotD50_T4_000,rotD50_T4_500,rotD50_T5_000,rotD50_T6_000,rotD50_T7_000,rotD50_T8_000,rotD50_T9_000,rotD50_T10_000,rotD100_T0_010,rotD100_T0_025,rotD100_T0_040,rotD100_T0_050,rotD100_T0_070,rotD100_T0_100,rotD100_T0_150,rotD100_T0_200,rotD100_T0_250,rotD100_T0_300,rotD100_T0_350,rotD100_T0_400,rotD100_T0_450,rotD100_T0_500,rotD100_T0_600,rotD100_T0_700,rotD100_T0_750,rotD100_T0_800,rotD100_T0_900,rotD100_T1_000,rotD100_T1_200,rotD100_T1_400,rotD100_T1_600,rotD100_T1_800,rotD100_T2_000,rotD100_T2_500,rotD100_T3_000,rotD100_T3_500,rotD100_T4_000,rotD100_T4_500,rotD100_T5_000,rotD100_T6_000,rotD100_T7_000,rotD100_T8_000,rotD100_T9_000,rotD100_T10_000,rotD00_T0_010,rotD00_T0_025,rotD00_T0_040,rotD00_T0_050,rotD00_T0_070,rotD00_T0_100,rotD00_T0_150,rotD00_T0_200,rotD00_T0_250,rotD00_T0_300,rotD00_T0_350,rotD00_T0_400,rotD00_T0_450,rotD00_T0_500,rotD00_T0_600,rotD00_T0_700,rotD00_T0_750,rotD00_T0_800,rotD00_T0_900,rotD00_T1_000,rotD00_T1_200,rotD00_T1_400,rotD00_T1_600,rotD00_T1_800,rotD00_T2_000,rotD00_T2_500,rotD00_T3_000,rotD00_T3_500,rotD00_T4_000,rotD00_T4_500,rotD00_T5_000,rotD00_T6_000,rotD00_T7_000,rotD00_T8_000,rotD00_T9_000,rotD00_T10_000,Event_trt,model,model_ssm +0,ESM,Unrestricted non-commercial use with acknowledgement/citation (assume Open Data Commons Attribution license),EQ_EMSC-20161026_0000077_&_ST_MZ01,,EQ_EMSC-20161026_0000077,2016-10-26 17:10:37,EMSC-20161026_0000077,611830883,us20007guy,8663031,20161026_0000077,,42.88834,13.13969,3.421,NF,5.23,,5.5,,5.26,,,161,38,-90,,,8.1,5.22396,3A,MZ01,MZ01_ESM,,D,0,,,,42.671575,13.270813,859,432,MASW,2.99,26.36251853,155.9379295,22.54138981,23.31192011,-0.72,22.53,0,E,90,N,0,Z,0.08,0.08,0.08,30,30,30,Both horizontal components and RotD50.,0.026666667,15.625,-108.780061,222.716024,-45.881192,158.0578535,223.527557,100.333954,-6.29355,-11.968429,1.87797,8.49725,11.9684,5.66205,-0.491045,0.816809,0.409598,0.5915245,0.819616,5.66205,5.115,3.5,7.075,4.0375,5.22,3.29,15.8938,22.4358,6.56143,17.28745,22.4361,15.4313,222.3692659,249.1042814,98.69522581,237.2638944,281.3945904,219.5205067,11.72180552,20.44563019,1.706962667,16.08371738,21.20602943,10.96140531,108.814,109.145,113.382,113.855,131.154,157.119,209.572,287.553,322.037,470.346,485.438,366.386,230.659,150.823,147.798,88.6999,74.8389,56.7993,50.7337,45.2617,31.7445,17.8258,10.69,8.3071,5.9755,4.94116,4.19675,2.94533,1.77386,1.43809,1.10865,0.808699,0.646502,0.555276,0.485435,0.430637,222.754,223.874,223.206,233.682,226.536,229.423,367.39,689.011,604.778,614.015,558.911,464.005,384.068,304.436,181.511,116.186,101.368,89.2144,69.9108,54.475,36.2397,25.3938,18.5248,14.0174,10.9633,6.5821,4.62422,3.73092,3.46864,2.7935,1.90001,1.37214,1.1054,0.939974,0.825001,0.738696,46.0036,46.7648,49.8155,55.3152,56.0385,97.4233,87.01,109.309,141.071,128.861,138.416,105.426,73.6946,47.991,41.638,32.189,28.8204,22.6039,23.6324,20.3062,11.4845,8.40455,6.18486,5.77979,5.52651,5.29949,5.60751,4.01614,2.89897,1.92162,1.26359,0.71051,0.451174,0.32469,0.251564,0.210068,158.237,159.057,160.614,166.106,163.9575,172.0525,267.403,541.9695,509.462,477.9505,486.2615,439.2885,321.112,237.947,157.5885,95.1981,76.06495,63.2522,50.7961,44.6243,31.13795,20.54695,13.13655,9.948655,7.798775,6.13877,4.5332,3.48062,2.587275,2.129465,1.746825,1.201655,0.9443365,0.7416165,0.599498,0.5308465,223.582,224.737,224.181,234.909,227.348,229.636,369.917,694.73,614.773,614.015,558.911,474.467,398.221,304.436,183.322,116.499,101.444,89.2144,69.9108,54.475,36.2397,25.396,18.5646,14.0695,11.0188,6.64568,4.70183,4.07226,3.48888,2.82703,2.0778,1.39204,1.11718,0.947651,0.830024,0.742738,100.334,100.603,104.649,104.309,119.811,132.008,195.791,228.186,272.999,345.806,358.876,205.642,200.462,150.823,114.108,76.1539,65.8692,52.0546,39.8062,35.2361,22.6864,14.8157,9.6153,7.65188,5.67826,4.32238,3.16842,2.29032,1.67326,1.24244,0.982716,0.764951,0.629198,0.548478,0.48188,0.427676,,EUR,EUR +1,NGAWest2,Unrestricted non-commercial use with acknowledgement/citation (assume Open Data Commons Attribution license),EQ_HelenaMontana-01_&_ST_CarrollCollege,1,EQ_HelenaMontana-01,1935-10-31 18:38:00,,,,,,,46.61,-111.96,6,U,,,5.999478682,,,,,75,75,160,,,,,USGS,CarrollCollege,CarrollCollege_NGAWest2,,,,,,,46.58,-112.03,,593.35,"('3b_4b_4c (check NGAWest2 flatfile for details)',)",,6.31,,2.07,,-2.07,8.046385723,,H1,180,H2,270,V,,,,,,,Only RotD50.,Shortest usable period is not provided for NGAWest2 db,6.134969325,,,97.170012,154.017,,,,,8.7544,10.1,,,,,3.7782,3.01,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,101.60217,109.30302,160.75647,193.72788,233.45838,262.27035,242.86617,137.94822,195.08166,184.61439,184.60458,165.87729,132.42519,98.65917,52.244136,42.072147,42.649956,43.504407,45.100494,46.611234,54.132561,56.968632,53.050518,44.712018,38.924118,27.213921,18.330966,12.42927,8.7433587,7.00438905,5.6328039,3.740553,2.8117422,2.1854718,1.7450028,1.4245101,158.922,161.865,220.725,263.889,358.065,310.977,416.925,252.117,287.433,294.3,298.224,270.756,172.656,140.283,105.948,97.119,91.0368,91.5273,99.081,99.081,83.0907,63.765,51.5025,43.7526,35.2179,22.1706,15.0093,10.3986,7.83819,5.77809,4.55184,2.98224,2.12877,1.6677,1.34397,1.09872,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,ASCR,USA,USA +2,NGASUB,Unrestricted non-commercial use with acknowledgement/citation (assume Open Data Commons Attribution license),EQ_32_&_ST_0,3001112,EQ_32,1992-05-23 10:30:00,,,,,,,13.475,-89.968,30.3,TF,,,6.01,,,,,266,31,64,,28.871,,,MARN,0,0_NGASUB,,,,,,,13.901,-89.932,,519,0 (check NGASUB flatfile for details),,47.52844,,42.3033488,53.7305448,49.36,-6.15,,H1,0,H2,90,V,,,,,,,Only RotD50.,-999,1.120393132,,,,19.587627,,,,,,1.2827,,,,,,0.12268,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,19.77965775,23.00567625,22.57553718,21.93305085,27.28007964,40.56590979,44.0809407,47.56503087,45.49767147,43.657443,59.85950166,55.47373515,66.88610055,47.69909433,36.10163385,32.12992782,31.02004404,25.57102068,18.32013576,12.56328441,8.49204612,5.340633651,3.360548916,2.373493203,1.814767596,0.935975043,0.565687764,0.477673425,0.308442096,0.258203615,0.206077689,0.151994178,0.115239051,0.102401685,0.0792648,0.0591543,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Crustal (Subduction),CCA,CCA +3,Turkiye_SMD_DB,Unrestricted non-commercial use with acknowledgement/citation (stated Open Data Commons Attribution license on designsafe-ci.org),EQ_1976-08-19 01:12:39_&_ST_2001,,EQ_1976-08-19_01_12_39,1976-08-19 01:12:39,,,,,,,37.751,29.012,15,NF,,,5.3,,,,,164,45,-30,,13.1658,,,TK,2001,2001_Turkiye_SMD,,,,,,,37.76219,29.09222,427,345,measured,,7.2,,5.3,14,-5.3,0,,H1,0,H2,90,V,0.5,0.7,0.6,50,35,35,"Both horizontal components, geometric mean and RotD50.",,1,349.4699685,265.0300011,175.139892,345.196345,397.5286327,,25.71706851,16.74596444,3.691003062,20.92157556,25.96406635,,2.321535194,1.306427358,0.424777783,1.76671965,2.308163168,,,,,,,,,,,,,,42.81945921,39.91230851,23.01366895,41.83484503,43.05897711,,,,,,,,352.5137751,371.7939459,436.9675422,473.7189885,467.1723851,650.984528,566.0361838,637.8539283,928.4857133,644.5311038,585.407876,887.7871752,919.7338768,948.991776,762.4288679,516.614736,453.5151081,371.1364385,245.3134717,185.3005298,114.8505799,87.79503155,58.68839465,41.24137244,29.23518517,17.74726511,11.58969587,7.740172404,5.475992031,4.138949853,3.428248707,2.445621228,1.795600818,1.359768024,1.059524145,0.846344997,269.0489776,315.4147958,446.7169037,502.7303016,580.1022935,601.7686556,669.0149803,513.9840746,605.2320967,689.0293168,741.2015887,761.1543213,687.0242951,634.9311045,441.632002,298.5782685,259.1369202,221.1998158,154.9156735,104.9689787,68.85758976,42.83862094,28.18929398,20.15359827,15.35405675,9.035385723,5.965897545,4.242641553,3.185243235,2.493461164,2.005100235,1.388219967,1.0213191,0.783943587,0.621046575,0.504247734,178.5869426,203.5847865,370.0625309,494.5172372,583.0962035,463.9003341,373.5391027,215.1353444,262.739266,145.3607659,153.2835769,128.5626722,118.3886555,101.3540486,79.6795099,55.25484658,53.83497073,54.65994366,52.58029625,36.43523958,26.55829614,23.75661311,16.95943971,10.26755115,9.107675613,3.5656407,1.966543992,1.321249059,0.994831119,0.781450376,0.630628002,0.439305534,0.324726696,0.250134399,0.198685854,0.16164918,349.601708,369.5689143,435.3062216,471.504741,511.8556784,607.3665359,560.8208788,552.4788943,788.0604967,658.995012,690.0313112,820.4574792,813.3028167,771.7576442,590.6726264,392.3657239,345.7152999,284.6507653,197.1927543,140.7055334,88.60075431,62.86843075,45.85777106,31.90222202,21.49231404,13.26438549,9.207631665,6.516281709,4.700142675,3.722125896,2.745626724,1.82487582,1.340730738,1.029177891,0.814873536,0.660902643,401.4325555,420.6328004,461.3746721,503.7327865,597.825012,810.1050529,675.9317268,646.6043002,963.3097379,713.7606496,918.9786853,961.08658,966.5096431,953.7293576,773.8470487,532.937026,483.8884004,398.3168299,262.4140606,188.1390455,117.2709159,88.76718979,60.81813192,41.28863309,29.30295952,17.87134396,11.67445917,7.796032506,5.664512763,4.452417612,3.444079104,2.45398131,1.801078722,1.363887243,1.062860526,0.849159486,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,ACR_shallow,MIE,MIE +4,kiknet,Unrestricted non-commercial use with acknowledgement/citation (assume Open Data Commons Attribution license),EQ_2017_12_31_071100_&_ST_OITH11,OITH111712310711,EQ_2017_12_31_071100,2017-12-31 07:11:00,,,,,,,33.33,132.197,52,TF,,,3.9,,,,,353,27,-132,,,,,kiknet,OITH11,OITH11_kiknet,,,,,,,33.2844,131.2118,,458.5260518,measured,,91.77380711,,89.8074312,103.2607496,-89.43607088,6.47537781,,H1,0,H2,90,V,,,,,,,Two horizontal components only.,0.025,4,1.148,1.125,,1.14094224,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,1.1455137,1.170333,1.2639204,1.4065578,2.0498976,3.7883277,5.04234,1.9111842,0.979038,0.8370873,0.8371854,0.567018,0.3585555,0.3014613,0.2554524,0.1702035,0.1655928,0.1499949,0.0998658,0.0845622,0.0375723,0.0442431,0.0243288,0.0206991,0.0161865,0.0092214,0.0054936,0.0037278,0.002943,0.00220725,0.001962,0.0014715,0.0008829,0.0008829,0.0008829,0.0008829,0.0011448,0.0011698,0.0012715,0.0013027,0.0018758,0.0035329,0.0045913,0.002345,0.0013091,0.001148,0.0006805,0.0006129,0.0003926,0.0003049,0.0002422,0.0002107,0.0001877,0.0001466,0.0001078,8.49E-05,6.12E-05,4.22E-05,2.70E-05,1.73E-05,1.75E-05,1.04E-05,6.40E-06,5.70E-06,3.90E-06,2.75E-06,2.30E-06,1.20E-06,1.20E-06,9.00E-07,7.00E-07,7.00E-07,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0.036285381,0.037007194,0.039449768,0.043300039,0.062195302,0.115582305,0.155538328,0.068399665,0.036625513,0.03174797,0.024466669,0.019124101,0.012179765,0.009847959,0.008088241,0.006163356,0.005740206,0.004829948,0.003456381,0.002622068,0.001543604,0.001427586,0.000819657,0.000601923,0.000509502,0.000301749,0.000189883,0.000146586,0.000107028,7.63E-05,6.81E-05,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,JPN,JPN diff --git a/openquake/smt/tests/parsers/esm_flatfile_parser_test.py b/openquake/smt/tests/parsers/esm_flatfile_parser_test.py index 0e61fd93f..60e187348 100644 --- a/openquake/smt/tests/parsers/esm_flatfile_parser_test.py +++ b/openquake/smt/tests/parsers/esm_flatfile_parser_test.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2018 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published diff --git a/openquake/smt/tests/parsers/esm23_flatfile_parser_test.py b/openquake/smt/tests/parsers/esm_url_flatfile_parser_test.py similarity index 71% rename from openquake/smt/tests/parsers/esm23_flatfile_parser_test.py rename to openquake/smt/tests/parsers/esm_url_flatfile_parser_test.py index 4b47ab314..213c22e8f 100644 --- a/openquake/smt/tests/parsers/esm23_flatfile_parser_test.py +++ b/openquake/smt/tests/parsers/esm_url_flatfile_parser_test.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2018 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,8 +16,8 @@ # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . """ -Tests parsing of the ESM23 flatfile format (i.e. flatfile downloaded from custom -header URL) in SMT +Tests parsing of a flatfile downloaded from ESM custom URL database +--> (https://esm-db.eu/esmws/flatfile/1/) This parser assumes you have selected all available headers in your URL search when downloading the flatfile @@ -26,7 +26,7 @@ import sys import shutil import unittest -from openquake.smt.parsers.esm23_flatfile_parser import ESM23FlatfileParser +from openquake.smt.parsers.esm_url_flatfile_parser import ESMFlatfileParserURL if sys.version_info[0] >= 3: import pickle @@ -48,21 +48,23 @@ #Specify base directory BASE_DATA_PATH = os.path.join(os.path.dirname(__file__), "data") -class ESM23FlatfileParserTestCase(unittest.TestCase): +class ESMFlatfileParserURLTestCase(unittest.TestCase): """ - Tests the parsing of the reformatted ESM23 flatfile + Tests the parsing of an ESM URL format flatfile """ @classmethod def setUpClass(cls): - cls.ESM23_flatfile_directory = os.path.join(BASE_DATA_PATH,"ESM23_Greece_test.csv") - cls.db_file = os.path.join(BASE_DATA_PATH, "ESM23_conversion_test_metadata") + cls.ESM_flatfile_directory = os.path.join(BASE_DATA_PATH, + "ESM_URL_Greece_test.csv") + cls.db_file = os.path.join(BASE_DATA_PATH, + "ESM_URL_conversion_test_metadata") - def test_esm23_flatfile_parser(self): + def test_esm_url_flatfile_parser(self): """ - Tests the parsing of the reformatted ESM23 flatfile + Tests the parsing of the reformatted ESM flatfile """ - parser = ESM23FlatfileParser.autobuild("000", "ESM23_conversion_test", - self.db_file, self.ESM23_flatfile_directory) + parser = ESMFlatfileParserURL.autobuild("000", "ESM_conversion_test", + self.db_file, self.ESM_flatfile_directory) with open(os.path.join(self.db_file, "metadatafile.pkl"), "rb") as f: db = pickle.load(f) # Should contain 9 records diff --git a/openquake/smt/tests/parsers/esm22_flatfile_parser_test.py b/openquake/smt/tests/parsers/esm_ws_flatfile_parser_test.py similarity index 71% rename from openquake/smt/tests/parsers/esm22_flatfile_parser_test.py rename to openquake/smt/tests/parsers/esm_ws_flatfile_parser_test.py index ceae9ba99..f814fb3b5 100644 --- a/openquake/smt/tests/parsers/esm22_flatfile_parser_test.py +++ b/openquake/smt/tests/parsers/esm_ws_flatfile_parser_test.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2018 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -16,13 +16,15 @@ # You should have received a copy of the GNU Affero General Public License # along with OpenQuake. If not, see . """ -Tests parsing of the ESM22 flatfile format (i.e. flatfile downloaded from web service) in SMT +Tests parsing of the ESM flatfile format (i.e. flatfile downloaded from ESM +web service) +--> https://esm-db.eu/#/waveform/search """ import os import sys import shutil import unittest -from openquake.smt.parsers.esm22_flatfile_parser import ESM22FlatfileParser +from openquake.smt.parsers.esm_ws_flatfile_parser import ESMFlatfileParserWS if sys.version_info[0] >= 3: import pickle @@ -54,22 +56,23 @@ BASE_DATA_PATH = os.path.join(os.path.dirname(__file__), "data") -class ESM22FlatfileParserTestCase(unittest.TestCase): +class ESMFlatfileParserWSTestCase(unittest.TestCase): """ - Tests the parsing of the reformatted ESM22 flatfile + Tests the parsing of a flatfile downloaded from the ESM web service """ @classmethod def setUpClass(cls): - #Specify base directory - cls.ESM22_flatfile_directory = os.path.join(BASE_DATA_PATH,"ESM22_Albania_filtered_test.csv") - cls.db_file = os.path.join(BASE_DATA_PATH, "ESM22_conversion_test_metadata") + # Specify base directory + cls.ESM_flatfile_directory = os.path.join( + BASE_DATA_PATH, "ESM_WS_Albania_filtered_test.csv") + cls.db_file = os.path.join(BASE_DATA_PATH, "ESM_ws_conversion_test_metadata") - def test_esm22_flatfile_parser(self): + def test_esm_ws_flatfile_parser(self): """ - Tests the parsing of the reformatted ESM22 flatfile + Tests the parsing of the reformatted ESM flatfile """ - parser = ESM22FlatfileParser.autobuild("000", "ESM22_conversion_test", - self.db_file, self.ESM22_flatfile_directory) + parser = ESMFlatfileParserWS.autobuild("000", "ESM_conversion_test", + self.db_file, self.ESM_flatfile_directory) with open(os.path.join(self.db_file, "metadatafile.pkl"), "rb") as f: db = pickle.load(f) # Should contain 19 records diff --git a/openquake/smt/tests/parsers/gem_flatfile_parser_test.py b/openquake/smt/tests/parsers/gem_flatfile_parser_test.py new file mode 100644 index 000000000..278e343db --- /dev/null +++ b/openquake/smt/tests/parsers/gem_flatfile_parser_test.py @@ -0,0 +1,81 @@ +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2024 GEM Foundation +# +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with OpenQuake. If not, see . +""" +Tests parsing of the GEM globally homogenised flatfile using the parser +""" +import os +import sys +import shutil +import unittest +from openquake.smt.parsers.gem_flatfile_parser import GEMFlatfileParser + +if sys.version_info[0] >= 3: + import pickle +else: + import cPickle as pickle + +# Defines the record IDs for the target data set +TARGET_IDS = [ +"EQ_EMSC_20161026_0000077_3A_MZ01_ESM_", +"EQ_HelenaMontana_01_USGS_CarrollCollege_NGAWest2_", +"EQ_32_MARN_0_NGASUB_", +"EQ_1976_08_19_01_12_39_TK_2001_Turkiye_SMD_", +"EQ_2017_12_31_071100_kiknet_OITH11_kiknet_"] + +#Specify base directory +BASE_DATA_PATH = os.path.join(os.path.dirname(__file__), "data") + +class GEMFlatfileParserTestCase(unittest.TestCase): + """ + Tests the parsing of the GEM global flatfile + """ + @classmethod + def setUpClass(cls): + cls.GEM_flatfile_directory = os.path.join(BASE_DATA_PATH, + "GEM_flatfile_test.csv") + cls.db_file = os.path.join(BASE_DATA_PATH, + "ESM_conversion_test_metadata") + + def test_gem_flatfile_parser(self): + """ + Tests the parsing of the GEM flatfile. + + Checks the proxy will give the KiKNet record the geometric mean of the + horizontal components as a proxy for the missing RotD50 acc values beyond + 5 s + the removal option will then not discard this record as RotD50 is + now 'complete' for all required spectral periods + """ + parser = GEMFlatfileParser.autobuild("000", "GEM_conversion_test", + self.db_file, self.GEM_flatfile_directory, + removal=True, proxy=True) + with open(os.path.join(self.db_file, "metadatafile.pkl"), "rb") as f: + db = pickle.load(f) + # Should contain 5 records + self.assertEqual(len(db), 5) + # Record IDs should be equal to the specified target IDs + for rec in db: + print(rec.id) + self.assertListEqual([rec.id for rec in db], TARGET_IDS) + del parser + + @classmethod + def tearDownClass(cls): + """ + Remove the database + """ + shutil.rmtree(cls.db_file) \ No newline at end of file diff --git a/openquake/smt/tests/parsers/ngaw2_flatfile_parser_test.py b/openquake/smt/tests/parsers/ngawest2_flatfile_parser_test.py similarity index 95% rename from openquake/smt/tests/parsers/ngaw2_flatfile_parser_test.py rename to openquake/smt/tests/parsers/ngawest2_flatfile_parser_test.py index f1d7e7a2a..d98d90e28 100644 --- a/openquake/smt/tests/parsers/ngaw2_flatfile_parser_test.py +++ b/openquake/smt/tests/parsers/ngawest2_flatfile_parser_test.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2018 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published @@ -22,7 +22,7 @@ import sys import shutil import unittest -from openquake.smt.parsers.ngaw2_flatfile_parser import NGAWest2FlatfileParser +from openquake.smt.parsers.ngawest2_flatfile_parser import NGAWest2FlatfileParser if sys.version_info[0] >= 3: import pickle diff --git a/openquake/smt/tests/residuals/residual_plots_test.py b/openquake/smt/tests/residuals/residual_plots_test.py index bb66ae43f..d7a7e7f7f 100644 --- a/openquake/smt/tests/residuals/residual_plots_test.py +++ b/openquake/smt/tests/residuals/residual_plots_test.py @@ -1,3 +1,20 @@ +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill +# +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with OpenQuake. If not, see . """ Test suite for the `residual_plots` module responsible for calculating the data used for plotting (see `residual_plotter`) diff --git a/openquake/smt/tests/residuals/residual_plotter_test.py b/openquake/smt/tests/residuals/residual_plotter_test.py index 5b76f21a7..aff7c67c1 100644 --- a/openquake/smt/tests/residuals/residual_plotter_test.py +++ b/openquake/smt/tests/residuals/residual_plotter_test.py @@ -1,3 +1,20 @@ +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill +# +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with OpenQuake. If not, see . """ Test suite for the `residual_plotter` module responsible for plotting the plot data defined in `residual_plots` diff --git a/openquake/smt/tests/residuals/residuals_ranking_metrics_test.py b/openquake/smt/tests/residuals/residuals_ranking_metrics_test.py index b5939f5d2..f186fda1e 100644 --- a/openquake/smt/tests/residuals/residuals_ranking_metrics_test.py +++ b/openquake/smt/tests/residuals/residuals_ranking_metrics_test.py @@ -1,3 +1,23 @@ +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill +# +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with OpenQuake. If not, see . +""" +Tests for GMPE ranking metrics in the SMT +""" import os import sys import shutil diff --git a/openquake/smt/tests/residuals/residuals_test.py b/openquake/smt/tests/residuals/residuals_test.py index f0848a6af..f6db1e552 100644 --- a/openquake/smt/tests/residuals/residuals_test.py +++ b/openquake/smt/tests/residuals/residuals_test.py @@ -1,3 +1,20 @@ +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill +# +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with OpenQuake. If not, see . """ Core test suite for the database and residuals construction """ diff --git a/openquake/smt/tests/residuals/residuals_test_table_and_database.py b/openquake/smt/tests/residuals/residuals_test_table_and_database.py index 05fda356a..be8220473 100644 --- a/openquake/smt/tests/residuals/residuals_test_table_and_database.py +++ b/openquake/smt/tests/residuals/residuals_test_table_and_database.py @@ -1,3 +1,20 @@ +# -*- coding: utf-8 -*- +# vim: tabstop=4 shiftwidth=4 softtabstop=4 +# +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill +# +# OpenQuake is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published +# by the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# OpenQuake is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with OpenQuake. If not, see . """ Core test suite for the database and residuals construction when created from sm_database.GroundMotionDatabase and diff --git a/openquake/smt/tests/sm_utils_test.py b/openquake/smt/tests/sm_utils_test.py index f2bc9284d..be56323c6 100644 --- a/openquake/smt/tests/sm_utils_test.py +++ b/openquake/smt/tests/sm_utils_test.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # vim: tabstop=4 shiftwidth=4 softtabstop=4 # -# Copyright (C) 2014-2017 GEM Foundation and G. Weatherill +# Copyright (C) 2014-2024 GEM Foundation and G. Weatherill # # OpenQuake is free software: you can redistribute it and/or modify it # under the terms of the GNU Affero General Public License as published