Uploading Data to EarthScope - Tutorial

Uploading Data to EarthScope - Tutorial#

In this tutorial, we demonstrate programmatic interaction with EarthScope’s Dropoff system for data submission.

Note

For command line access, see the CLI Dropoff Tutorial.

Setup#

Create an EarthScopeClient instance and optionally declare a callback function to receive progress updates.

from earthscope_sdk import AsyncEarthScopeClient
from earthscope_sdk.client.dropoff.models import UploadStatus

es = AsyncEarthScopeClient()


def progress_callback(status: UploadStatus):
    print(status)

Upload an Object#

Use the put_object() or put_objects() (plural) methods to upload objects to EarthScope’s Dropoff S3 Bucket.

When using put_object, you must provide:

body: the content to upload to S3. This can be any of the following:
- a str or Path instance is interpreted as a path to a local file
- a bytes instance is an in-memory buffer
- a file-like object (any object with a .read() method) will be read in a streaming fashion
- an iterator or async iterator is expected to yield bytes and will be read in a streaming fashion
key: the S3 object key where the object will be uploaded
category: the dropoff category. This dropoff system is used for many different data submission categories.
progress_cb (optional): a callback function to receive updates throughout the upload process

In the following example, we upload a local miniSEED file.

result = await es.dropoff.put_object(
    "/path/to/data/SAML.IU.2025.303",
    key="demo/SAML.IU.2025.303",
    category="miniseed",
    progress_cb=progress_callback,
)
print(f"Uploaded {result.key} ({result.size} bytes)")

UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=0, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=10485760, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=20971520, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=31457280, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=41409536, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=9952256, bytes_buffered=41409536, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=20438016, bytes_buffered=41409536, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=30923776, bytes_buffered=41409536, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=41409536, bytes_buffered=41409536, total_bytes=41409536, complete=True)
Uploaded demo/SAML.IU.2025.303 (41409536 bytes)

Browsing Submitted Data#

You can use list_objects() and get_object_history() to explore the data you’ve submitted to EarthScope and check on the status of each submission.

The results of both of these methods are paginated to handle large result sets.

Note

Learn more about how objects flow through our automated processing.

Listing Objects#

To browse through your dropoff space, use the list_objects() method. Optionally, provide a prefix= keyword argument to drill down into nested prefixes.

page = await es.dropoff.list_objects(prefix="demo/")
page.items[0]

DropoffObject(path='demo/SAML.IU.2025.303', received_at=datetime.datetime(2025, 12, 31, 21, 46, 31, tzinfo=TzInfo(0)), status='VALIDATING', status_message=None, size=41409536, sha256='c9a668471eda59068de73e3bcf46febd3fa05be025cae19182d1c0846b83f445')

Object History#

When the same S3 key is used multiple times (i.e. to replace an errant submission), you can retrieve the history of each submission using get_object_history()

page = await es.dropoff.get_object_history(key="demo/SAML.IU.2025.303")
page.items

[DropoffObject(path='demo/SAML.IU.2025.303', received_at=datetime.datetime(2025, 12, 31, 21, 46, 31, tzinfo=TzInfo(0)), status='VALIDATING', status_message=None, size=41409536, sha256='c9a668471eda59068de73e3bcf46febd3fa05be025cae19182d1c0846b83f445'),
 DropoffObject(path='demo/SAML.IU.2025.303', received_at=datetime.datetime(2025, 12, 31, 21, 44, 37, tzinfo=TzInfo(0)), status='AUTHORIZED', status_message=None, size=41409536, sha256='c9a668471eda59068de73e3bcf46febd3fa05be025cae19182d1c0846b83f445'),
 DropoffObject(path='demo/SAML.IU.2025.303', received_at=datetime.datetime(2025, 12, 31, 21, 44, 19, tzinfo=TzInfo(0)), status='AUTHORIZED', status_message=None, size=41409536, sha256='c9a668471eda59068de73e3bcf46febd3fa05be025cae19182d1c0846b83f445'),
 DropoffObject(path='demo/SAML.IU.2025.303', received_at=datetime.datetime(2025, 12, 31, 21, 43, 28, tzinfo=TzInfo(0)), status='AUTHORIZED', status_message=None, size=41409536, sha256='c9a668471eda59068de73e3bcf46febd3fa05be025cae19182d1c0846b83f445'),
 DropoffObject(path='demo/SAML.IU.2025.303', received_at=datetime.datetime(2025, 12, 31, 21, 37, 45, tzinfo=TzInfo(0)), status='ACCEPTED', status_message=None, size=41409536, sha256='c9a668471eda59068de73e3bcf46febd3fa05be025cae19182d1c0846b83f445'),
 DropoffObject(path='demo/SAML.IU.2025.303', received_at=datetime.datetime(2025, 12, 31, 21, 37, 13, tzinfo=TzInfo(0)), status='ACCEPTED', status_message=None, size=41409536, sha256='c9a668471eda59068de73e3bcf46febd3fa05be025cae19182d1c0846b83f445'),
 DropoffObject(path='demo/SAML.IU.2025.303', received_at=datetime.datetime(2025, 12, 31, 21, 35, 31, tzinfo=TzInfo(0)), status='ACCEPTED', status_message=None, size=41409536, sha256='c9a668471eda59068de73e3bcf46febd3fa05be025cae19182d1c0846b83f445')]

Bulk Submission#

When you need to upload more than one object, the put_objects() (plural) method should be used to more effectively saturate your upload bandwidth.

Instead of providing a single body and key, the objects argument is used to provide a list of (body, key) tuples.

the body of each object can still be any of the types supported by put_object() (singular)
objects may be an iterable of tuples which will be consumed in a streaming fashion

results = await es.dropoff.put_objects(
    objects=[
        (
            "/path/to/data/SAML.IU.2025.303#2",
            "demo/SAML.IU.2025.303",
        ),
        (
            "/path/to/data/BILL.IU.2025.303#2",
            "demo/BILL.IU.2025.303",
        ),
        # ... more (body, key) tuples ...
    ],
    category="miniseed",
    progress_cb=progress_callback,
)

for result in results:
    print(f"Uploaded {result.key} ({result.size} bytes)")

UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=0, total_bytes=41409536, complete=False)
UploadStatus(key='demo/BILL.IU.2025.303', bytes_done=0, bytes_buffered=0, total_bytes=17494016, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=10485760, total_bytes=41409536, complete=False)
UploadStatus(key='demo/BILL.IU.2025.303', bytes_done=0, bytes_buffered=10485760, total_bytes=17494016, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=20971520, total_bytes=41409536, complete=False)
UploadStatus(key='demo/BILL.IU.2025.303', bytes_done=0, bytes_buffered=17494016, total_bytes=17494016, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=31457280, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=0, bytes_buffered=41409536, total_bytes=41409536, complete=False)
UploadStatus(key='demo/BILL.IU.2025.303', bytes_done=7008256, bytes_buffered=17494016, total_bytes=17494016, complete=False)
UploadStatus(key='demo/BILL.IU.2025.303', bytes_done=17494016, bytes_buffered=17494016, total_bytes=17494016, complete=True)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=10485760, bytes_buffered=41409536, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=20971520, bytes_buffered=41409536, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=30923776, bytes_buffered=41409536, total_bytes=41409536, complete=False)
UploadStatus(key='demo/SAML.IU.2025.303', bytes_done=41409536, bytes_buffered=41409536, total_bytes=41409536, complete=True)
Uploaded demo/SAML.IU.2025.303 (41409536 bytes)
Uploaded demo/BILL.IU.2025.303 (17494016 bytes)

Advanced Use Cases#

Streaming Uploads#

Our SDK will do its best to stream bodies to S3 rather than materialize them in memory in their entirety.

For example, in the following cell, we perform an HTTP request and upload its response to EarthScope’s Dropoff system. Instead of waiting for the response to download completely before we start the upload, the HTTP response is streamed into S3 without ever keeping the entire body in memory at any given time (unless the body is small).

import requests

result = await es.dropoff.put_object(
    body=requests.get(
        "https://service.earthscope.org/fdsnws/dataselect/1/query?net=IU&sta=ANMO&starttime=2025-11-15T00:00:00&endtime=2025-11-16T00:00:00&nodata=404",
        stream=True,
    ),
    key="demo/ANMO.IU.2025.2025.319.mseed",
    category="miniseed",
    progress_cb=progress_callback,
)
print(f"Uploaded {result.key} ({result.size} bytes)")

UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=10485760, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=20971520, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=31457280, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=41943040, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=52428800, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=62914560, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=73400320, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=83886080, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=94371840, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=104857600, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=0, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=10485760, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=20971520, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=31457280, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=31682560, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=42168320, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=52654080, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=63139840, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=73625600, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=84111360, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=94597120, bytes_buffered=105082880, total_bytes=None, complete=False)
UploadStatus(key='demo/ANMO.IU.2025.2025.319.mseed', bytes_done=105082880, bytes_buffered=105082880, total_bytes=None, complete=True)
Uploaded demo/ANMO.IU.2025.2025.319.mseed (105082880 bytes)

Tuning Upload Behavior#

The put_object and put_objects methods use S3 multipart uploads to facilitate efficient and resumable uploads. Multipart uploads break an object into multiple smaller “parts” that can upload independently in order to checkpoint upload progress and also parallelize uploads.

Depending on your upload bandwidth and connection reliability, it may be beneficial to tune the following parameters to better suit your situation.

object_concurrency: the total number of objects to upload at once
part_concurrency: the total number of object parts to upload at once
part_size: the size (in bytes) of each part to upload. the final part’s size may be less than part_size

Tip

Since parts must buffer in memory before upload, the combination of these three parameters also sets a ceiling for how much memory the upload process will occupy.

For example, on a slow or intermittent internet connection, the following settings may help get the whole body to EarthScope’s S3 bucket by not spreading your upload bandwidth across multiple parts or objects at once. If the upload is interrupted, it can be resumed without needing to re-upload any parts that have already successfully been uploaded.

await es.dropoff.put_object(
    body=...,
    key=...,
    category=...,
    object_concurrency=1,  # only upload one object at a time
    part_concurrency=1,  # only upload one part at a time
    part_size=5 * 1024**2,  # 5 MB parts
)

Conversely, a high speed internet connection may benefit from more parallelism and/or larger part sizes.

await es.dropoff.put_object(
    body=...,
    key=...,
    category=...,
    object_concurrency=3,  # upload three objects at a time
    part_concurrency=12,  # upload twelve parts at a time (across all objects)
    part_size=16 * 1024**2,  # 16 MB parts
)

Note

These upload settings can be configired via SDK Settings.

Uploading Data to EarthScope - Tutorial

Contents

Uploading Data to EarthScope - Tutorial#

Setup#

Upload an Object#

Browsing Submitted Data#

Listing Objects#

Object History#

Bulk Submission#

Advanced Use Cases#

Streaming Uploads#

Tuning Upload Behavior#