EarthScope Dropoff System

EarthScope Dropoff System#

Overview#

The EarthScope Dropoff system provides a streamlined way for data contributors to submit their data to EarthScope. It replaces multiple legacy submission pathways with a unified, automated system that makes it easier to share your data with the scientific community.

Think of the Dropoff system as your private upload space where you can organize and submit data files. Once uploaded, your data is automatically validated and processed before being ingested into the EarthScope data repository, where it becomes available to researchers worldwide.

How It Works#

When you upload files to the Dropoff system, you work with a simple, intuitive file-system structure that makes sense to you. For example, you might organize your local directory using experiment names and datetimes.

Behind the scenes, the SDK handles the complexity for you:

  • Your files are stored in a secure S3 bucket

  • Each contributor has their own protected area

  • You don’t need to worry about the underlying storage details—just organize your files in a way that makes sense for your workflow

This abstraction means you can focus on your data rather than managing cloud storage configurations.

Dropoff Categories#

The Dropoff system supports different categories of data, each with its own validation and processing pipeline. This allows EarthScope to handle diverse data types in a unified system.

  • miniseed: For contributors submitting seismic data in the miniSEED format.

Note

Additional dropoff categories will be introduced in the future, expanding the Dropoff system to replace remaining legacy submission pathways.

Uploading Your Data#

Using the SDK, you can upload files to your Dropoff space with simple, straightforward commands. The SDK takes care of:

  • Authentication and authorization

  • Efficient uploads for large files

  • Error handling and retries

  • Tracking upload progress

Organize your files in whatever structure that makes sense for your data collection workflow. The system will preserve your organizational structure while ensuring your data is properly categorized and stored.

Resuming Uploads#

The SDK supports resuming uploads in case they are interrupted for any reason.

The SDK leverages S3 multipart uploads to perform a set of smaller uploads that make up a whole object. Each part that uploads successfully to S3 is a checkpoint from which we are able to resume. Should your upload be interrupted, reinvoking the same method or command will upload only remaining parts and remaining objects.

Note

Resuming uploads is the default behavior for EarthScope SDK. There is nothing required on your end to enable this behavior.

Automated Processing#

Once files are uploaded, EarthScope’s Dropoff system immediately begins processing them. What happens during processing depends on the data category.

If validation & authorization succeed, your data moves forward to ingestion. If issues are detected, you’ll be able to see detailed status information about what went wrong, allowing you to correct and reupload the file.

Dropoff Status#

The status field reflects the current stage of processing a submitted file.

Status

Meaning

RECEIVED

Your file landed in our S3 bucket and automated processing has started.

VALIDATING

Validating the uploaded file.

VALIDATED

Validation succeeded.

AUTHORIZING

Checking submitter permissions.

AUTHORIZED

Authorization succeeded.

FAILED

Validation or authorization has failed.

INTERNAL_ERROR

A system error occurred unrelated to validation/authorization.

ACCEPTED

File has passed all checks and is accepted for archiving.

Tip

An additional status_message field provides more information about failed validation or authorization.

Processing Progression#

The flow chart below shows the progression through automated processing.

        ---
title: File Submission Processing Flow
---
%%{init: {'theme':'base', 'themeVariables': {'primaryColor':'#e8f4f8','primaryTextColor':'#000','primaryBorderColor':'#4a90a4','lineColor':'#81b1bf','secondaryColor':'#f0f0f0','tertiaryColor':'#fff'}}}%%
flowchart LR
    Upload[File Upload] --> Received[RECEIVED]
    Received --> Validating[VALIDATING]
    Validating --> ValidCheck{Validation<br/>Passed?}
    ValidCheck -->|Yes| Validated[VALIDATED]
    ValidCheck -->|No| Failed1[FAILED]
    Validated --> Authorizing[AUTHORIZING]
    Authorizing --> AuthCheck{Authorization<br/>Passed?}
    AuthCheck -->|Yes| Authorized[AUTHORIZED]
    AuthCheck -->|No| Failed2[FAILED]
    Authorized --> Accepted[ACCEPTED]

    Validating -.->|System Error| InternalError1[INTERNAL_ERROR]
    Authorizing -.->|System Error| InternalError2[INTERNAL_ERROR]

    style Accepted fill:#90EE90,stroke:#2d5c2d,color:#000
    style Failed1 fill:#FFB6C1,stroke:#8b2e4a,color:#000
    style Failed2 fill:#FFB6C1,stroke:#8b2e4a,color:#000
    style InternalError1 fill:#FFD700,stroke:#8b7500,color:#000
    style InternalError2 fill:#FFD700,stroke:#8b7500,color:#000

Miniseed Validation#

For miniSEED files, the system:

  1. Verifies file integrity: Ensures the file can be read as valid miniSEED data

  2. Checks authorization: Confirms you’re authorized to submit data for all networks, stations, locations, and channels contained in the file

Checking Upload Status#

You have two ways to monitor your data submissions:

1. Listing Your Files#

View all files you’ve uploaded to your Dropoff space, similar to listing files in a directory. This gives you a quick overview of what’s been submitted and its most recent status.

2. Getting File History#

For any specific file, you can retrieve its complete processing history. This includes:

  • When the file was uploaded

  • Validation results

  • Processing steps completed

  • Any errors encountered

  • Current status

The history feature is especially useful if you need to:

  • Reupload a file to replace erroneous data

  • Understand why a submission failed validation

  • Track multiple uploads to the same file path

Each time you upload to the same path (or if EarthScope operators rerun processing), a new history entry is created, preserving a complete audit trail.

What Happens After Upload#

Once your data passes validation, it enters the EarthScope ingestion pipeline:

  1. Processing: Automated workflows handle your data according to its category

  2. Ingestion: Your data is incorporated into the EarthScope data repository

  3. Availability: After ingestion completes, your data becomes accessible to the scientific community through standard EarthScope data access tools and services

Warning

The Dropoff system is for submission only. You cannot download data from the Dropoff system. Once ingested, your data (and everyone else’s) is available through EarthScope’s data access services.

Best Practices#

  • Organize thoughtfully: While you have flexibility in how you structure your files, consistent organization makes it easier for you to track submissions and for EarthScope staff to navigate your submission area.

  • Monitor status: Check the processing status of your uploads, especially for large submissions.

  • Reupload when needed: If validation fails, review the error details, correct the issue, and reupload the file to the same path.

  • Name files descriptively: Clear file names help you and EarthScope staff track and troubleshoot submissions.

Getting Started#

To start using the Dropoff system, you’ll need:

  1. An EarthScope account

  2. Contributor permission for the networks/stations you’ll be submitting data for

  3. The EarthScope CLI and SDK installed

For detailed examples and tutorials, see the following:

Note

The EarthScope Dropoff system is continuously evolving to better serve our data contributors. We welcome feedback and suggestions—please contact help@earthscope.org.

Contents