Uploading Data to EarthScope - CLI Tutorial

Uploading Data to EarthScope - CLI Tutorial#

This tutorial demonstrates how to use the EarthScope CLI to interact with EarthScope’s Dropoff system for data submission.

Note

For programmatic access using Python, see the SDK Dropoff Tutorial.

Setup#

Before uploading, ensure you have:

  1. The EarthScope CLI installed

  2. An EarthScope account with contributor permissions for the data you’ll be submitting

  3. Authenticated with the CLI (see Authentication)

You can verify your setup by checking the available commands:

$ es dropoff --help

 Usage: es dropoff [OPTIONS] COMMAND [ARGS]...

 Use the EarthScope Dropoff system

╭─ Commands ───────────────────────────────────────────────────────────────────────╮
│ get-object-history   Retrieve the upload history for a specific object key.      │
│ get-summary          Get a summary of a dropoff prefix.                          │
│ list-objects         List objects in your dropoff space.                         │
│ upload               Upload files or directories to EarthScope's dropoff system. │
╰──────────────────────────────────────────────────────────────────────────────────╯

Upload Data#

Use the upload command to upload files to EarthScope’s Dropoff S3 Bucket.

When using upload, you must provide:

  • --source / -s – one or more local file or directory paths to upload

  • --destination / -d – the remote object key or prefix

  • --category / -c – the dropoff category (the Dropoff system is used for many different submission categories)

Tip

--category may also be configured using SDK Settings to avoid specifying it for every command

Destination: Key vs Prefix#

The --destination parameter controls where files are placed in your remote dropoff space:

  • Exact key (no trailing /): The destination is used as the complete object key, overriding the source filename. Only valid with a single file source.

    $ es dropoff upload -c miniseed -d 2025/303/SAML.IU.2025.303 -s data/SAML.IU.2025.303

    This creates the remote key 2025/303/SAML.IU.2025.303.

  • Prefix (trailing /): The source filename is appended to the prefix. Required when uploading directories or multiple files.

    $ es dropoff upload -c miniseed -d 2025/303/ -s data/SAML.IU.2025.303 -s data/BILL.IU.2025.303

    This creates remote keys 2025/303/SAML.IU.2025.303 and 2025/303/BILL.IU.2025.303.

Uploading Files#

In the following example, we upload two local miniSEED files to a prefix:

$ es dropoff upload -c miniseed -d 2025/303/ -s data/BILL.IU.2025.303 -s data/SAML.IU.2025.303
Uploading 2 files... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 • 21.0 MB • 3.0 MB/s • 0:00:03
  2025/303/SAML.IU.2025.303 ━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/41.4 MB •  25%
  2025/303/BILL.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 10.5/17.5 MB •  60%

After all uploads complete, all progress bars for individual files are hidden, leaving just the overall summary:

$ es dropoff upload -c miniseed -d 2025/303/ -s data/BILL.IU.2025.303 -s data/SAML.IU.2025.303
✓ Completed 2 uploads ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 58.9 MB • 5.0 MB/s • 0:00:07

The CLI shows progress bars tracking:

  • Overall progress

    • Files uploaded/total

    • Total bytes uploaded

    • Transfer speed

    • Elapsed time

  • Upload progress for each file

    • Bytes uploaded/total

    • Completion percentage

Note

When uploading a large number of files, the progress bars for individual files hide themselves after completion.

Bulk Uploads#

To upload many files at once, organize your local files into one or more directories and pass the directory as the source. Use --recursive / -r to include files in subdirectories.

Without -r, only top-level files in the directory are uploaded:

$ es dropoff upload -c miniseed -d 2025/303/ -s data/

With -r, all files in the directory tree are uploaded, preserving their relative paths under the destination prefix:

$ es dropoff upload -c miniseed -d data/ -r -s data/
Uploading 6 files... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/6 • 109.3 MB • 13.6 MB/s • 0:00:08
  data/2025/303/ANTO.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━ 28.9/39.4 MB •  73%
  data/2025/303/CASY.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━ 42.5/52.9 MB •  80%
  data/2025/303/SAML.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━ 30.9/41.4 MB •  75%
  data/2025/303/BILL.IU.2025.303 ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.0/17.5 MB  •  40%
  data/2025/303/BBSR.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/78.3 MB  •   0%
  data/2025/303/ANMO.IU.2025.303 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/103.8 MB •   0%

After completion:

$ es dropoff upload -c miniseed -d data/ -r -s data/
✓ Completed 6 uploads ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6/6 • 333.4 MB • 16.1 MB/s • 0:00:14

Resumable Uploads#

If an upload is interrupted (network issue, cancelled, etc.), simply run the same command again. The CLI automatically resumes from where it left off without re-uploading data that’s already in EarthScope’s system.

Note

Large file uploads use S3 multipart uploads to facilitate efficient and resumable uploads. Each successfully uploaded part is a checkpoint for resuming.

Browsing Submitted Data#

You can use list-objects and get-object-history to explore the data you’ve submitted to EarthScope and check on the status of each submission.

Note

Learn more about how objects flow through our automated processing.

Listing Objects#

To browse through your dropoff space, use the list-objects command. Optionally, provide a --prefix to drill down into nested prefixes:

$ es dropoff list-objects --category miniseed --prefix data

                           miniseed objects - prefix: 'data'
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ Name                           ┃    Size ┃        Last Modified ┃  Status  ┃ Message ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ data/SAML.IU.2025.303          │ 41.4 MB │ 2026-01-13T15:56:03Z │ ACCEPTED │ -       │
│ data/BILL.IU.2025.303          │ 17.5 MB │ 2026-01-13T15:55:57Z │ ACCEPTED │ -       │
└────────────────────────────────┴─────────┴──────────────────────┴──────────┴─────────┘
                                    End of results.

Object History#

When the same S3 key is used multiple times (e.g. to replace an errant submission), you can retrieve the history of each submission using get-object-history:

$ es dropoff get-object-history --key data/SAML.IU.2025.303

                      History for: data/SAML.IU.2025.303
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ Name                    ┃    Size ┃        Last Modified ┃  Status  ┃ Message ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ data/SAML.IU.2025.303   │ 41.4 MB │ 2026-01-13T15:16:03Z │ ACCEPTED │ -       │
│ data/SAML.IU.2025.303   │ 41.1 MB │ 2026-01-09T20:27:21Z │ ACCEPTED │ -       │
│ data/SAML.IU.2025.303   │ 41.2 MB │ 2026-01-05T18:02:53Z │ ACCEPTED │ -       │
└─────────────────────────┴─────────┴──────────────────────┴──────────┴─────────┘
                                 End of results.

Getting a Summary#

For a high-level view of processing status across a prefix, use get-summary. This returns a count of objects grouped by status—useful for quickly gauging the health of a large batch submission without paging through individual file listings.

Provide --prefix to scope the summary to a specific path within your dropoff space. You can further narrow the results with --submitted-after and --submitted-before to filter by submission time.

$ es dropoff get-summary --category miniseed --prefix data/

 Dropoff Summary for data/
┏━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Status         ┃ Count ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ ACCEPTED       │   102 │
│ FAILED         │     3 │
│ VALIDATING     │    42 │
└────────────────┴───────┘

Narrow by date range to track a specific submission window:

$ es dropoff get-summary --category miniseed --prefix data/ --submitted-after 2026-01-01T00:00:00Z --submitted-before 2026-02-00T00:00:00Z

Common Workflows#

Replacing a File#

If you need to correct and resubmit a file, simply upload it again using the exact same object key:

The new file replaces the old one. Use get-object-history to view the complete history of all submissions to that path.

Advanced Use Cases#

Tuning Upload Behavior#

Upload performance and memory usage can be tuned using the following optional parameters to the es dropoff upload command:

  • --part-size: the size (in bytes) of each part to upload. The final part’s size may be less than this value

  • --part-concurrency: the total number of object parts to upload at once

  • --object-concurrency: the total number of objects to upload at once

Tip

Since parts must buffer in memory before upload, the combination of these parameters also sets a ceiling for how much memory the upload process will occupy.

For example, on a slow or intermittent internet connection, the following settings may help get the whole file to EarthScope’s S3 bucket by not spreading your upload bandwidth across multiple parts or objects at once. If the upload is interrupted, it can be resumed without needing to re-upload any parts that have already successfully been uploaded.

$ es dropoff upload [...] --object-concurrency 1 --part-concurrency 1 --part-size 5000000

Conversely, a high speed internet connection may benefit from more parallelism and/or larger part sizes.

$ es dropoff upload [...] --object-concurrency 3 --part-concurrency 12 --part-size 16000000

Note

These options can also be configured as SDK settings, see SDK Settings.

Contents