Open Data on AWS

The data repositories of the NSF’s National Geophysical Facility (NGF) are managed by the EarthScope Consortium. A portion of these repositories are available as open data sets sponsored by AWS and available in a dedicated sponsored data bucket in the AWS S3 service.

All data sets contain recordings, or derivatives thereof, from networks of geophysical sensors. A description of each data set is below.

AWS S3 bucket and region¶

Bucket: earthscope-geophysical-data

Region: us-east-2

The S3 bucket may be accessed from any location, but it is most efficient to access it from the AWS us-east-2 Region.

miniSEED repository¶

The sponsored data bucket contains a subset of seismic data from selected networks of the NGF holdings. Generally these networks deliver continuous seismological data to the facility, with some stations also including other geophysical sensor types, along with station and sensor state-of-health information. The data can contain gaps.

Data format and identification¶

The data are in miniSEED format, the standard for data exchange in the seismological community defined by the International Federation of Digital Seismograph Networks (FDSN). Data are either in miniSEED version 2, defined in the SEED specification, or miniSEED v3.

Data are identified using a combination of codes defined by the FDSN in the SEED specification and extended by the (FDSN Source Identifier specification)[https://docs.fdsn.org/projects/source-identifiers]. These codes are summarized as:

• Network code – Uniquely identifies the owner and network operator responsible for the data

• Station code – Uniquely identifies a station within a network

• Location code – Uniquely identifies a group of channels within a station

• Channel code – A sequence of codes that identify the band, source and subsource for specific channel

Combined, these codes globally, and uniquely, identify channels of data. The vast majority of data channels are time series of recorded geophysical sensors. Other channels contain State-of-Health and station log information.

Data organization¶

The miniSEED data are organized into objects that contain all of the channels for a given station for a given day. Day boundaries, and the time base for the data, are always in UTC.

The S3 prefix and object naming scheme is as follows:

/miniSEED/NETWORK/YEAR/DAYOFYEAR/STATION.NETWORK.YEAR.DAYOFYEAR

where:

• miniseed is the top level prefix for this part of the repository

• NETWORK and STATION are data identifier codes

• YEAR is always a 4-digit year

• DAYOFYEAR is the day of the year from 001-366

For example: miniseed/TA/2004/365/A04A.TA.2004.365

Metadata¶

Summary or detailed metadata are available from the facility metadata web service at:

https://service.earthscope.org/fdsnws/station/1

Full granularity metadata is commonly accessed in StationXML format with summaries commonly accessed in GeoCSV and text formats.

Data license and citation¶

When using these data please cite both the network owners, identified by network code, and the NSF NGF operated by EarthScope.

Network operators and owners specify the license for their data, and an appropriate citation, in their registration information which is available at: https://fdsn.org/networks/. When no license has been declared by the owner, the facility distributes the data with a license of CC-BY-4.0.

The facility operation can be cited using the instructions here: https://www.earthscope.org/how-to-cite/

Getting started¶

There are a variety of methods available to the sponsored data bucket.

AWS CLI¶

To access the sponsored data bucket without an AWS account or credentials:

aws s3 ls --no-sign-request s3://earthscope-geophysical-data/

A successful command will show the list of the top level prefixes / repositories available (at the time of this writing, currently only ‘miniseed’):

       PRE miniseed/

If you have an AWS account the --no-sign-request option is not required.

Python with Boto3 and ObsPy¶

Boto3 is the AWS SDK for Python supporting access to S3 and ObsPy is a common Python framework for working with seismological data.

import boto3
from botocore import UNSIGNED
from botocore.config import Config
from obspy import read
import io

# Initialize S3 client
s3 = boto3.client(
    's3',
    region_name='us-east-2',
    config=Config(signature_version=UNSIGNED)
    )

# Define bucket and key
bucket_name = 'earthscope-geophysical-data'
object_key = 'miniseed/TA/2004/365/A04A.TA.2004.365'

# Download object to memory
response = s3.get_object(Bucket=bucket_name, Key=object_key)
data_stream = io.BytesIO(response['Body'].read())

# Parse with ObsPy
st = read(data_stream)

# Print the ObsPy Streams
print(st)

When you have an AWS account, the boto3 client setup has fewer options that need to be specified:

import boto3

s3 = boto3.client('s3', region_name='us-east-2')

# Define bucket and key
...