SGY non-standard trace header location and size

[espenraknes]

Is there a way of using segysak and specifying SGY trace header location and byte size manually? We experience more and more often that we get SGYs delivered with information placed in non-standard SGY byte locations which we would like to extract using segysak.

[trhallam]

Are the files you want to load still standard SEG-Y? You can load any of the 4 byte standard trace headers. If the companies are using more than 4 bytes then you'll have to load all the byte headers and post process them to get the combined value.

You can load any header data with your traces by adding the byte locations you want using the extra_byte_fields keyword argument when loading the file.

Check out the Quick start tutorial for more details here.

[espenraknes]

No, the case is that they are not following the SEGY standard, hence the "normal" header_key vs byte_size does not work. As an example, 223-226 is used for a four byte header, and with the table used that is read as two two-byte headers. So the question is if there is a possibility to add such info in the reading of the file to be able to read non-standard header keys?

If not, the workaround is to map the header to another standard key that is not in use and with correct size. But that requires an extra step outside ofPython.

[trhallam]

Afraid not, SEGY-SAK relies on segyio to read data and it's a faithful implementation of the standard.

As you say, you can process the file again, manually reading the bytes you want, or in a post processing step with Xarray, you could convert header ints back to binary, combine them and convert back to int. I agree that neither is ideal but SEGY isn't either 😅.

[espenraknes]

Ok, then header remapping is needed...

[ThomasHertweck]

You could try to read the SEG-Y file using my seisio package, which gives complete control over the trace headers via a JSON configuration file, e.g., seisio.input(segy_file, thdef="my_trhead_def.json"). The only constraint is that the combination of byte locations and types in the definition of the trace header must be continuous (no gaps or overlaps allowed) and amount to 240 bytes in total (although the software would even allow you to read trace headers of other lengths, but this would violate the SEG-Y standard completely ;-)). Reading the data set from disk using "seisio" will give you a Numpy structured array, which you could easily turn into an Xarray for further processing with SEGY-SAK.

[trhallam]

Have you done any performance comparisons with segyio?

[ThomasHertweck]

Although performance was not the primary goal for developing seisio, we did some comparisons with other available packages. As you certainly know, an objective and fair comparison is a tricky business as there are many issues to consider. The following is a basic comparison of reading a single SEG-Y file from disk (SSD) into memory (both the data themselves and the trace headers) without any fancy re-ordering, slicing etc. The file is about 5 GB with 1'550'400 traces (IBM floats, big endian) and follows SEG-Y rev. 1 standard (it's actually a poststack seismic cube). The test was repeated for each package several times, each time clearing the OS cache etc. The test included unpacking of headers to provide comparable runtimes as returning generators would simply defer the actual work to a later time and spoil the comparison; the runtimes also include IBM to IEEE conversion. The individual runtimes, all given in seconds, for each package were averaged and the standard deviation calculated:

1. segfast: 5.8083 +- 0.0290
2. seisio: 7.1055 +- 0.0228
3. segyio: 8.1640 +- 0.0306
4. segy_lite: 34.6420 +- 0.4106
5. obspy: 103.1666 +- 0.5145
6. segpy: I cancelled the test after >10 minutes
7. segypy: ran out of memory (32 GB RAM available on the test machine)

Obviously, the comparison might look different when only reading subsets of the data (for instance, an inline), or when the data have to be provided in a different order compared to how the data are stored on disk, or when only one or two trace headers are required rather than all of them, or when the data are stored in IEEE instead of IBM format (the conversion is a significant portion of the overall runtime), or... If raw speed matters and it's the primary goal, go with segfast. If you need flexible reading of prestack data as often required in seismic data processing (my scientific background), seisio might be a good choice. If you are primarily interested in reading structured data, segyio is pretty good. If you need some fancy access, perhaps simply convert the SEG-Y data to HDF5, NetCDF, Zarr or MDIO first. So it's also a matter of perspective and it's good that we have a choice. :-) We have also run a comparison of reading approx. 10 TB of SEG-Y data and extracting all trace headers into a database between seisio and a commercial processing system of a big seismic service provider and both took about the same time. By the way, we also compared the data values (after IBM to IEEE conversion) and the traces headers for all the Python packages mentioned above and they were identical, i.e., it seems all packages read the data correctly.

[espenraknes]

Thanks for the tip. I'll have a look at it.