Antelope Tutorial – Making a Database from IRIS-DMC Seed Data (quick and dirty)

ECR – December 4, 2014
Modified – February 23, 2022

CREATING a database from downloaded IRIS requested data!
This requres seed files (or miniseed) and dataless metadata.

  1. Break up seed files into day files suing miniseed2days command.
    Example, seed files had been saved to raw_data_ftp folder:

/Volumes/raid1/QCF/db -> miniseed2days -v -U -w “day_volumes/%{sta}/%{sta}.%{net}.%{loc}.%{chan}.%Y.%j” raw_data_ftp/*.seed

output…:

/Volumes/raid1/QCF/db -> miniseed2days -v -U -w “day_volumes/%{sta}/%{sta}.%{net}.%{loc}.%{chan}.%Y.%j” raw_data_ftp/*.seed

miniseed2days: miniseed2days Antelope Release 5.4 Mac OS X 10.8.5 2014-07-07 <#>
miniseed2days: 6481a8b2f07fb719f563732829073f23456ab00c <#>
miniseed2days: Mon Jul 7 15:31:43 2014 <#>

–> miniseed2days -v -U -w day_volumes/%{sta}/%{sta}.%{net}.%{loc}.%{chan}.%Y.%j raw_data_ftp/1125_BBB_BCP_BESE.955498.seed raw_data_ftp/1125_CRAG_JIS_LOGN.787588.seed raw_data_ftp/1125_PIN_PNL_SAMH.162748.seed raw_data_ftp/1125_SIT_SKAG_WHY.419216.seed raw_data_ftp/1125_WRAK_YKU2.556714.seed <#>

miniseed2days: limited to 2550 open files

miniseed2days: raw_data_ftp/1125_BBB_BCP_BESE.955498.seed
miniseed2days: failed to read seed data header at offset 0, skipped ahead to offset 126976
miniseed2days: creating ./day_volumes/BBB/BBB.CN..BHE.2013.115
miniseed2days: failed to read seed data header at offset 5309440, skipped ahead to offset 5312512
miniseed2days: creating ./day_volumes/BBB/BBB.CN..BHE.2013.116
miniseed2days: failed
…
miniseed2days: failed to read seed data header at offset 2445621248, found no more data headers in file
miniseed2days: 4764715 blocks (0 dropped) from raw_data_ftp/1125_WRAK_YKU2.556714.seed

miniseed2days: wrote 17380850 total miniseed data blocks into 1872 files
miniseed2days: dropped 0 total data blocks

  1. Run seed2db to creat and ‘populate’ my database (here calling it craig) with your station data.
    Example, note, now referencing dataless files:

/Volumes/raid1/QCF/db -> seed2db raw_data_ftp/CRAIG_Dataless.806918.dataless craig
214 17:51 seed2db raw_data_ftp/CRAIG_Dataless2.980398.dataless craig
215 17:51 seed2db raw_data_ftp/CRAIG_Dataless3.516918.dataless craig
217 17:52 seed2db raw_data_ftp/CRAIG_Dataless4.616362.dataless craig
218 17:52 seed2db raw_data_ftp/CRAIG_Dataless5.925387.dataless craig

*note, have to do this for each dataless file individually.
At this point, it should be clear that there are files in the response directory.
Output like the following is OK:

seed2db: seed2db Antelope Release 5.4 Mac OS X 10.8.5 2014-07-07 <#>
seed2db: 6481a8b2f07fb719f563732829073f23456ab00c <#>
seed2db: Mon Jul 7 15:31:45 2014 <#>

–> seed2db raw_data_ftp/CRAIG_Dataless5.925387.dataless craig <#>

seed2db: EOF detected on raw_data_ftp/CRAIG_Dataless5.925387.dataless.
seed2db: SEEDERRORs: parsing = 0
seed2db: SEEDERRORs: units = 0
seed2db: SEEDERRORs: samprate = 0
seed2db: SEEDERRORs: normalization freq. = 0
seed2db: SEEDERRORs: stage gains = 0
seed2db: SEEDERRORs: fir order = 0
seed2db: SEEDERRORs: fir delay = 0
seed2db: SEEDERRORs: paz unstable = 0
seed2db: SEEDERRORs: paz complex conjugate = 0
seed2db: SEEDERRORs: coordinate = 0
seed2db: SEEDERRORs: no response stages = 0
/Volumes/raid1/QCF/db ->

Also – now my directory looks like this:
/Volumes/raid1/QCF/db -> ls
craig.calibration craig.lastid craig.schanloc craig.site craig.snetsta pf response
craig.instrument craig.network craig.sensor craig.sitechan day_volumes raw_data_ftp

  1. Now run miniseed2db to add waveforms to the database:

/Volumes/raid1/QCF/db -> miniseed2db day_volumes/BBB/* craig
226 17:56 miniseed2db day_volumes/BCP/* craig
228 17:56 miniseed2db day_volumes/BESE/* craig
230 17:56 miniseed2db day_volumes/CRAG/* craig
232 17:56 miniseed2db day_volumes/JIS/* craig
234 17:57 miniseed2db day_volumes/LOGN/* craig
236 17:57 miniseed2db day_volumes/PIN/* craig
238 17:59 miniseed2db day_volumes/PNL/* craig
242 18:02 miniseed2db day_volumes/SAMH/* craig
244 18:02 miniseed2db day_volumes/SIT/* craig
246 18:03 miniseed2db day_volumes/SKAG/* craig
248 18:03 miniseed2db day_volumes/WHY/* craig
250 18:03 miniseed2db day_volumes/WRAK/* craig
252 18:03 miniseed2db day_volumes/YKU2/* craig

Here’s what the output looks like:

/Volumes/raid1/QCF/db -> miniseed2db day_volumes/BBB/* craig

miniseed2db: miniseed2db Antelope Release 5.4 Mac OS X 10.8.5 2014-07-07 <#>
miniseed2db: 6481a8b2f07fb719f563732829073f23456ab00c <#>
miniseed2db: Mon Jul 7 15:31:31 2014 <#>

–> miniseed2db day_volumes/BBB/BBB.CN..BHE.2013.115 day_volumes/BBB/BBB.CN..BHE.2013.116 day_volumes/BBB/BBB.CN..BHE.2013.117 day_volumes/BBB/BBB.CN..BHE.2013.118 day_volumes/BBB/BBB.CN..BHE.2013.119 day_volumes/BBB/BBB.CN..BHE.2013.120 day_volumes/BBB/BBB.CN..BHE.2013.121 day_volumes/BBB/BBB.CN..BHE.2013.122 day_volumes/BBB/BBB.CN..BHE.2013.123 day_volumes/BBB/BBB.CN..BHE.2013.124 day_volumes/BBB/BBB.CN..BHE.2013.125 day_volumes/BBB/BBB.CN..BHE.2013.126 day_volumes/BBB/BBB.CN..BHE.2013.127 day_volumes/BBB/BBB.CN..BHE.2013.128 day_volumes/BBB/BBB.CN..BHE.2013.129 day_volumes/BBB/BBB.CN..BHE.2013.130 day_volumes/BBB/BBB.CN..BHE.2013.131 day_volumes/BBB/BBB.CN..BHE.2013.132 day_volumes/BBB/BBB.CN..BHE.2013.133 day_volumes/BBB/BBB.CN..BHE.2013.134 day_volumes/BBB/BBB.CN..BHE.2013.135 day_volumes/BBB/BBB.CN..BHE.2013.136 day_volumes/BBB/BBB.CN..BHE.2013.137 day_volumes/BBB/BBB.CN..BHE.2013.138 day_volumes/BBB/BBB.CN..BHE.2013.139 day_volumes/BBB/BBB.CN..BHE.2013.140 day_volumes/BBB/BBB.CN..BHE.2013.141 day_volday_volumes/BBB/BBB.CN..BHE.2013.115 5.182 Mbytes
day_volumes/BBB/BBB.CN..BHE.2013.116 5.795 Mbytes
day_volumes/BBB/BBB.CN..BHE.2013.117 6.219 Mbytes
day_volumes/BBB/BBB.CN..BHE.2013.118 5.905 Mbytes
…
day_volumes/BBB/BBB.CN..BHZ.2013.155 4.829 Mbytes
day_volumes/BBB/BBB.CN..BHZ.2013.156 0.229 Mbytes
177 new wfdisc records
/Volumes/raid1/QCF/db -> ls day_volumes/

3.b *Note, I messed this up by actually running the same line of code twice – I ran: –> miniseed2db day_volumes/PNL/ craig
twice.

Ideed, this added duplicate waveforms for that station.
To fix this, I’m trying deleating ALL the PNL stations, and re adding them :

/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘sta==”PNL”‘ | dbdelete –

(no output here).

And indeed, after running that, there are no PNL entries in wfdisc
but luckily there are still in the craig.sitechan and craig.site!

So now, I’ll just re-run the miniseed command (once!):

–> miniseed2db day_volumes/PNL/* craig

and now their back in teh wfdisc

  1. Create database descriptor file.

Not sure why we need to do this… but make a descriptor file that looks like this – database is named craig:

/Volumes/raid1/QCF/db -> more craig

craig descriptor file

#
schema css3.0
dbpath /Volumes/raid1/QCF/db{craig}
dblocks none
/Volumes/raid1/QCF/db ->

  1. Fix the calibration

/Volumes/raid1/QCF/db -> dbfix_calib craig
dbset: 67622 changes in View100
/Volumes/raid1/QCF/db ->

  1. Assign a segtype to each waveforem in teh wfdisc. Not sure this is 100% necessary.

Do this by subsetting each of the channels and then piping that into the dbset command

*Note: when I look in dbe at sitechan, and I sort by chan, I get
BHE
BHN
BHZ
BH1_00
BH1_10
BH2_00
BH2_10
BHZ_00
BHZ_10

(so need to run the command on all of thses… or I might be able to do it once for all of them, but I’ll just follow instructions here.)

/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BHE”‘ | dbset – segtype NULL V
/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BHN”‘ | dbset – segtype NULL V
/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BHZ”‘ | dbset – segtype NULL V
/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BH1_00″‘ | dbset – segtype NULL V
/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BH2_00″‘ | dbset – segtype NULL V
/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BH1_10″‘ | dbset – segtype NULL V
/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BH2_10″‘ | dbset – segtype NULL V
/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BHZ_00″‘ | dbset – segtype NULL V
/Volumes/raid1/QCF/db -> dbsubset craig.wfdisc ‘chan==”BHZ_10″‘ | dbset – segtype NULL V

(no output for any of this)

  1. Now verify the database

6a. verify that waveforms are properly associated with the database:

/Volumes/raid1/QCF/db -> dbversdwf -dtu craig
0 bad records

6b. Verify that all of the tabesl are properly linked together:

Volumes/raid1/QCF/db -> dbverify -tj craig

dbverify: dbverify Antelope Release 5.4 Mac OS X 10.8.5 2014-07-07 <#>
dbverify: 6481a8b2f07fb719f563732829073f23456ab00c <#>
dbverify: Mon Jul 7 15:31:35 2014 <#>

–> dbverify -tj craig <#>

The following records of craig.wfdisc did not have corresponding rows in site:
Record #52976 : sta = WHY chan = BHE time = 4/25/2013 (115) 0:00:00.000 endtime = 4/25/2013 (115) 23:59:59.975
Record #52977 : sta = WHY chan = BHE time = 4/26/2013 (116) 0:00:00.000 endtime = 4/26/2013 (116) 23:59:59.975
Record #52978 : sta = WHY chan = BHE time = 4/27/2013 (117) 0:00:00.000 endtime = 4/27/2013 (117) 23:59:59.975
Record #52979 : sta = WHY chan = BHE time = 4/28/2013 (118) 0:00:00.000 endtime = 4/28/2013 (118) 23:59:59.975

So – they’re NOT properly linked together.

To fix this, run dbfixchanids:

/Volumes/raid1/QCF/db -> dbfixchanids craig
dbfixchanids: No wfdisc match for WHY BHE 4/25/13 (115) 00:00:00.000
dbfixchanids: No wfdisc match for WHY BHE 4/26/13 (116) 00:00:00.000
dbfixchanids: No wfdisc match for WHY BHE 4/27/13 (117) 00:00:00.000

Oh no – looks like I forgot to request dataless seed for WHY?

So – here’s what fixing this looks like (after requesting WHY dataless from IRIS):

/Volumes/raid1/QCF/db -> cp ../data_land/December4_14/CRAIG_Dataless6.52448.dataless raw_data_ftp/.
/Volumes/raid1/QCF/db -> seed2db raw_data_ftp/CRAIG_Dataless6.52448.dataless craig

seed2db: seed2db Antelope Release 5.4 Mac OS X 10.8.5 2014-07-07 <#>
seed2db: 6481a8b2f07fb719f563732829073f23456ab00c <#>
seed2db: Mon Jul 7 15:31:45 2014 <#>

–> seed2db raw_data_ftp/CRAIG_Dataless6.52448.dataless craig <#>

seed2db: EOF detected on raw_data_ftp/CRAIG_Dataless6.52448.dataless.
seed2db: SEEDERRORs: parsing = 0
seed2db: SEEDERRORs: units = 0
seed2db: SEEDERRORs: samprate = 0
seed2db: SEEDERRORs: normalization freq. = 0
seed2db: SEEDERRORs: stage gains = 0
seed2db: SEEDERRORs: fir order = 0
seed2db: SEEDERRORs: fir delay = 0
seed2db: SEEDERRORs: paz unstable = 0
seed2db: SEEDERRORs: paz complex conjugate = 0
seed2db: SEEDERRORs: coordinate = 0
seed2db: SEEDERRORs: no response stages = 0
/Volumes/raid1/QCF/db -> dbfixchanids craig

So… are we good:

/Volumes/raid1/QCF/db -> dbverify -tj craig

dbverify: dbverify Antelope Release 5.4 Mac OS X 10.8.5 2014-07-07 <#>
dbverify: 6481a8b2f07fb719f563732829073f23456ab00c <#>
dbverify: Mon Jul 7 15:31:35 2014 <#>

–> dbverify -tj craig <#>

0 problems

Ah ha!

6c. Check that response files are properly associated with the database (run dbcheck_responses):

/Volumes/raid1/QCF/db -> dbcheck_responses -v craig

instrument table
Peak Peak Low Center High
Freq Magnitude Shoulder Shoulder filename
0.187 1.00 0.0026 0.2 19 response/Guralp_CMG3T_120sec:CNSN_GD2_v.1
20.2 1.80 0.0036 0.3 23 response/Nanometrics_Trillium_120_Sec_R.2
20.2 1.61 0.0016 0.2 23 response/Nanometrics_Trillium_240_Sec_R.3
0.254 1.00 0.0027 0.3 23 response/STS-2:Trident.4
0.198 1.00 0.0053 0.3 23 response/Guralp_CMG3ESP_60sec:Quanterra.5
20.2 1.61 0.0016 0.2 23 response/Nanometrics_Trillium_240_Sec_R.6
0.196 1.00 0.0027 0.2 9.3 response/STS-2:Trident.7
4.64 1.02 0.0027 0.2 18 response/STS2-I=80444=Gen=Q330SR=3790.8
20.9 1.63 0.0017 0.2 23 response/Trillium-240:Trident.9
16.8 1.01 0.0048 0.3 19 response/Guralp_CMG3ESP_NSN:CNSN_GD1_d.10

Cool! Think I’m done now.
Going home for dinner.

12/8/2014
Extra Notes – feel free to ignore. These relate to after detections/associations have been made.

Copying assoc, arrival, etc files from files created like this:
cat dbquakesApr.assoc dbquakesMay.assoc > craig.assoc

/Volumes/raid1/QCF/Supplimentary -> cp craig.arrival ../db/.
overwrite ../db/./craig.arrival? (y/n [n]) y
/Volumes/raid1/QCF/Supplimentary -> cp craig.assoc ../db/.
/Volumes/raid1/QCF/Supplimentary -> cp craig.event ../db/.
/Volumes/raid1/QCF/Supplimentary -> cp craig.orig* ../db/.

Not sure this will work (dont’t have the velocity models set up yet)

Also removed all of the PAL stations from teh assoc adn arrival files… moved the origional files to the orig/ directory within db

877 17:49 awk ‘{if($1 ~= “PAL”) print $0 > “craigtmp.arrival”}’ craig.arrival
878 17:49 awk ‘{if($1 != “PAL”) print $0 > “craigtmp.arrival”}’ craig.arrival

892 17:51 mv craig.arrival orig/.
893 17:51 mv craig.assoc orig/.

/Volumes/raid1/QCF/db -> mv craigtmp.arrival criag.arrival
/Volumes/raid1/QCF/db -> mv craigtmp.assoc craig.assoc