Uploading Data to EarthScope - CLI Tutorial

Uploading Data to EarthScope - CLI Tutorial#

This tutorial demonstrates how to use the EarthScope CLI to interact with EarthScope’s Dropoff system for data submission.

Note

For programmatic access using Python, see the SDK Dropoff Tutorial.

Setup#

Before uploading, ensure you have:

  1. The EarthScope CLI installed

  2. An EarthScope account with contributor permissions for the data you’ll be submitting

  3. Authenticated with the CLI (see Authentication)

You can verify your setup by checking the available commands:

$ es dropoff --help

 Usage: es dropoff [OPTIONS] COMMAND [ARGS]...

 Use the EarthScope Dropoff system

╭─ Commands ───────────────────────────────────────────────────────────────────────╮
│ get-object-history   Retrieve the upload history for a specific object key.      │
│ list-objects         List objects in your dropoff space.                         │
│ upload               Upload files or directories to EarthScope's dropoff system. │
╰──────────────────────────────────────────────────────────────────────────────────╯

Upload Data#

Use the upload command to upload files to EarthScope’s Dropoff S3 Bucket.

When using upload, you must provide:

  • At least one fully qualified file path or directory path to upload

  • --category: the dropoff category (the Dropoff system is used for many different submission categories)

Tip

--category may also be configured using SDK Settings to avoid specifying it for every command

In the following example, we upload two local miniSEED files:

$ es dropoff upload data/BILL.IU.2025.303 data/SAML.IU.2025.303 --category miniseed
Uploading 2 files... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 • 21.0 MB • 3.0 MB/s • 0:00:03
  data/SAML.IU.2025.303 ━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/41.4 MB •  25%
  data/BILL.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 10.5/17.5 MB •  60%

After all uploads complete, all progress bars for individual files are hidden, leaving just the overall summary:

$ es dropoff upload data/BILL.IU.2025.303 data/SAML.IU.2025.303 --category miniseed
✓ Completed 2 uploads ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 58.9 MB • 5.0 MB/s • 0:00:07

The CLI shows progress bars tracking:

  • Overall progress

    • Files uploaded/total

    • Total bytes uploaded

    • Transfer speed

    • Elapsed time

  • Upload progress for each file

    • Bytes uploaded/total

    • Copmletion percentage

Note

When uploading a large number of files, the progress bars for individual files hide themselves after completion.

Object Keys#

S3 object keys are generated following the rules below:

  • For individual files: The parent directory name is always included in the S3 key to provide context and prevent name collisions.

    Example: uploading project1/data.mseed creates key project1/data.mseed

  • For directories: All files maintain their relative paths from the directory you specify,
    with the directory name as a prefix.

    Example: uploading directory mydata/ containing subdir/file.mseed creates
    key mydata/subdir/file.mseed

  • Multiple files: Each file preserves its parent directory context.

    Example: uploading proj1/data.mseed and proj2/data.mseed creates
    distinct keys proj1/data.mseed and proj2/data.mseed

Bulk Uploads#

To upload many files at once, we recommend you organize your local files into one or more directories and then pass the names of those directories to the upload command instead of individual file names.

The CLI will recursively discover all files in the specified directories.

$ es dropoff upload data/ --category miniseed
Uploading 6 files... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/6 • 109.3 MB • 13.6 MB/s • 0:00:08
  data/2025/303/ANTO.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━ 28.9/39.4 MB •  73%
  data/2025/303/CASY.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━ 42.5/52.9 MB •  80%
  data/2025/303/SAML.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━ 30.9/41.4 MB •  75%
  data/2025/303/BILL.IU.2025.303 ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.0/17.5 MB  •  40%
  data/2025/303/BBSR.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/78.3 MB  •   0%
  data/2025/303/ANMO.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/103.8 MB •   0%

After completion:

$ es dropoff upload data/ --category miniseed
✓ Completed 6 uploads ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6/6 • 333.4 MB • 16.1 MB/s • 0:00:14

Resumable Uploads#

If an upload is interrupted (network issue, cancelled, etc.), simply run the same command again. The CLI automatically resumes from where it left off without re-uploading data that’s already in EarthScope’s system.

Note

Large file uploads use S3 multipart uploads to facilitate efficient and resumable uploads. Each successfully uploaded part is a checkpoint for resuming.

Browsing Submitted Data#

You can use list-objects and get-object-history to explore the data you’ve submitted to EarthScope and check on the status of each submission.

Note

Learn more about how objects flow through our automated processing.

Listing Objects#

To browse through your dropoff space, use the list-objects command. Optionally, provide a --prefix to drill down into nested prefixes:

$ es dropoff list-objects --category miniseed --prefix data

                           miniseed objects - prefix: 'data'
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ Name                           ┃    Size ┃        Last Modified ┃  Status  ┃ Message ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ data/SAML.IU.2025.303          │ 41.4 MB │ 2026-01-13T15:56:03Z │ ACCEPTED │ -       │
│ data/BILL.IU.2025.303          │ 17.5 MB │ 2026-01-13T15:55:57Z │ ACCEPTED │ -       │
└────────────────────────────────┴─────────┴──────────────────────┴──────────┴─────────┘
                                    End of results.

Object History#

When the same S3 key is used multiple times (e.g. to replace an errant submission), you can retrieve the history of each submission using get-object-history:

$ es dropoff get-object-history --key data/SAML.IU.2025.303

                      History for: data/SAML.IU.2025.303
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ Name                    ┃    Size ┃        Last Modified ┃  Status  ┃ Message ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ data/SAML.IU.2025.303   │ 41.4 MB │ 2026-01-13T15:16:03Z │ ACCEPTED │ -       │
│ data/SAML.IU.2025.303   │ 41.1 MB │ 2026-01-09T20:27:21Z │ ACCEPTED │ -       │
│ data/SAML.IU.2025.303   │ 41.2 MB │ 2026-01-05T18:02:53Z │ ACCEPTED │ -       │
└─────────────────────────┴─────────┴──────────────────────┴──────────┴─────────┘
                                 End of results.

Common Workflows#

Replacing a File#

If you need to correct and resubmit a file, simply upload it again using the exact same object key:

The new file replaces the old one. Use get-object-history to view the complete history of all submissions to that path.

Advanced Use Cases#

Tuning Upload Behavior#

Upload performance and memory usage can be tuned using the following optional parameters to the es dropoff upload command:

  • --part-size: the size (in bytes) of each part to upload. The final part’s size may be less than this value

  • --part-concurrency: the total number of object parts to upload at once

  • --object-concurrency: the total number of objects to upload at once

Tip

Since parts must buffer in memory before upload, the combination of these parameters also sets a ceiling for how much memory the upload process will occupy.

For example, on a slow or intermittent internet connection, the following settings may help get the whole file to EarthScope’s S3 bucket by not spreading your upload bandwidth across multiple parts or objects at once. If the upload is interrupted, it can be resumed without needing to re-upload any parts that have already successfully been uploaded.

$ es dropoff upload [...] --object-concurrency 1 --part-concurrency 1 --part-size 5000000

Conversely, a high speed internet connection may benefit from more parallelism and/or larger part sizes.

$ es dropoff upload [...] --object-concurrency 3 --part-concurrency 12 --part-size 16000000

Note

These options can also be configured as SDK settings, see SDK Settings.

Contents