EarthScope Dropoff System

Overview¶

The EarthScope Dropoff system provides a streamlined way for data contributors to submit their data to EarthScope. It replaces multiple legacy submission pathways with a unified, automated system that makes it easier to share your data with the scientific community.

Think of the Dropoff system as your private upload space where you can organize and submit data files. Once uploaded, your data is automatically validated and processed before being ingested into the EarthScope data repository, where it becomes available to researchers worldwide.

How It Works¶

When you upload files to the Dropoff system, you work with a simple, intuitive file-system structure that makes sense to you. For example, you might organize your local directory using experiment names and datetimes.

Behind the scenes, the SDK handles the complexity for you:

• Your files are stored in a secure S3 bucket

• Each contributor has their own protected area

• You don’t need to worry about the underlying storage details—just organize your files in a way that makes sense for your workflow

This abstraction means you can focus on your data rather than managing cloud storage configurations.

Dropoff Categories¶

The Dropoff system supports different categories of data, each with its own validation and processing pipeline. This allows EarthScope to handle diverse data types in a unified system.

• miniseed: For contributors submitting seismic data in the miniSEED format.

Uploading Your Data¶

Using the SDK, you can upload files to your Dropoff space with simple, straightforward commands. The SDK takes care of:

• Authentication and authorization

• Efficient uploads for large files

• Error handling and retries

• Tracking upload progress

Organize your files in whatever structure that makes sense for your data collection workflow. The system will preserve your organizational structure while ensuring your data is properly categorized and stored.

Resuming Uploads¶

The SDK supports resuming uploads in case they are interrupted for any reason.

The SDK leverages S3 multipart uploads to perform a set of smaller uploads that make up a whole object. Each part that uploads successfully to S3 is a checkpoint from which we are able to resume. Should your upload be interrupted, reinvoking the same method or command will upload only remaining parts and remaining objects.

Automated Processing¶

Once files are uploaded, EarthScope’s Dropoff system immediately begins processing them. What happens during processing depends on the data category.

If validation & authorization succeed, your data moves forward to ingestion. If issues are detected, you’ll be able to see detailed status information about what went wrong, allowing you to correct and reupload the file.

Dropoff Status¶

The status field reflects the current stage of processing a submitted file.

Processing Progression¶

The flow chart below shows the progression through automated processing.

Miniseed Validation¶

For miniSEED files, the system:

1. Verifies file integrity: Ensures the file can be read as valid miniSEED data

2. Checks authorization: Confirms you’re authorized to submit data for all networks, stations, locations, and channels contained in the file

Checking Upload Status¶

You have three ways to monitor your data submissions:

1. Listing Your Files¶

View all files you’ve uploaded to your Dropoff space, similar to listing files in a directory. This gives you a quick overview of what’s been submitted and its most recent status.

2. Getting File History¶

For any specific file, you can retrieve its complete processing history. This includes:

• When the file was uploaded

• Validation results

• Processing steps completed

• Any errors encountered

• Current status

The history feature is especially useful if you need to:

• Reupload a file to replace erroneous data

• Understand why a submission failed validation

• Track multiple uploads to the same file path

Each time you upload to the same path (or if EarthScope operators rerun processing), a new history entry is created, preserving a complete audit trail.

3. Getting a Summary¶

For a high-level view of your submission area, you can retrieve a count of objects grouped by processing status for any prefix. This is especially useful after a large batch upload when you want to quickly gauge how many files have been accepted, are still processing, or have failed—without paging through individual file listings.

The summary can be scoped by prefix and optionally filtered by submission time (submitted_after / submitted_before) to focus on a particular batch or time window.

What Happens After Upload¶

Once your data passes validation, it enters the EarthScope ingestion pipeline:

1. Processing: Automated workflows handle your data according to its category

2. Ingestion: Your data is incorporated into the EarthScope data repository

3. Availability: After ingestion completes, your data becomes accessible to the scientific community through standard EarthScope data access tools and services

Best Practices¶

• Organize thoughtfully: While you have flexibility in how you structure your files, consistent organization makes it easier for you to track submissions and for EarthScope staff to navigate your submission area.

• Monitor status: Check the processing status of your uploads, especially for large submissions.

• Reupload when needed: If validation fails, review the error details, correct the issue, and reupload the file to the same path.

• Name files descriptively: Clear file names help you and EarthScope staff track and troubleshoot submissions.

Getting Started¶

To start using the Dropoff system, you’ll need:

1. An EarthScope account

2. Contributor permission for the networks/stations you’ll be submitting data for

3. The EarthScope CLI and SDK installed

For detailed examples and tutorials, see the following:

• Programmatic Upload Tutorial - Programmatic access using the SDK.

• CLI Upload Tutorial - Command line access using the CLI.