EarthScope SDK Settings

EarthScope SDK Settings#

EarthScope SDK Settings consist of configuration & tokens.

  • Configuration is used to control behavior of the SDK.

  • Tokens are used to authenticate and authorize requests made by this SDK.

This document details the settings available, and a specification for settings fields, loading, and persistence on the user’s machine.

Note: if you don’t need to make any configuration changes (the most common use case!) then you don’t need to do anything and this page can be ignored.

Settings Fields#

Configuration#

Field

Environment Variable

Description

profile_name

ES_PROFILE

a named configuration profile

dropoff.*

ES_DROPOFF__*

Data dropoff configuration

http.*

ES_HTTP__*

HTTP client configuration

oauth2.*

ES_OAUTH2__*

oauth2 flow configuration

query_plan.*

ES_QUERY_PLAN__*

query plan configuration

resources.*

ES_RESOURCES__*

EarthScope resource references

Oauth2 Settings#

Field

Environment Variable

Description

oauth2.audience

ES_OAUTH2__AUDIENCE

oauth2 audience targeted

oauth2.client_id

ES_OAUTH2__CLIENT_ID

oauth2 Client ID used when requesting tokens

oauth2.client_secret

ES_OAUTH2__CLIENT_SECRET

oauth2 Client Secret used for machine-to-machine auth

oauth2.domain

ES_OAUTH2__DOMAIN

oauth2 domain for Identity Provider

oauth2.scope

ES_OAUTH2__SCOPE

oauth2 scope requested during token retrieval

oauth2.access_token

ES_OAUTH2__ACCESS_TOKEN

oauth2 access token to use to authenticate with the API

oauth2.refresh_token

ES_OAUTH2__REFRESH_TOKEN

oauth2 refresh token used to retrieve new access tokens

oauth2.id_token

ES_OAUTH2__ID_TOKEN

OIDC token

oauth2.allowed_hosts

ES_OAUTH2__ALLOWED_HOSTS

Your access token is automatically sent when requests are made to these hosts. Supports glob-style strings with ? and * characters.

oauth2.retry.*

ES_OAUTH2__RETRY__*

oauth2 retry configuration

Dropoff Settings#

The EarthScope SDK’s S3 upload behavior can be modified to better suit your upload bandwidth.

Field

Environment Variable

Default

Description

dropoff.bucket

ES_DROPOFF__BUCKET

The S3 bucket to upload to.

dropoff.category

ES_DROPOFF__CATEGORY

A Dropoff category may be configured via settings to avoid specifying the category on each call to put_object().

dropoff.part_size

ES_DROPOFF__PART_SIZE

10 MB

Size (in bytes) of each multipart upload part sent to S3. Larger parts mean fewer upload requests but higher memory use.

dropoff.part_concurrency

ES_DROPOFF__PART_CONCURRENCY

8

Maximum number of parts that will be uploaded in parallel (across multiple objects).

dropoff.object_concurrency

ES_DROPOFF__OBJECT_CONCURRENCY

3

Maximum number of objects that will be uploaded in parallel.

dropoff.retry.*

ES_DROPOFF__RETRY__*

S3 part upload retry configuration

HTTP Client Settings#

Configuration for the HTTPx client & connection pool.

Field

Environment Variable

Description

http.extra_headers

ES_HTTP__EXTRA_HEADERS

Additional arbitrary headers to inject into requests

http.user_agent

ES_HTTP__USER_AGENT

The user agent injected into HTTP requests. This value supersedes a user-agent header passed via http.extra_headers.

http.timeout_connect

ES_HTTP__TIMEOUT_CONNECT

HTTP connect timeout (seconds)

http.timeout_read

ES_HTTP__TIMEOUT_READ

HTTP read/write timeout (seconds)

http.keepalive_expiry

ES_HTTP__KEEPALIVE_EXPIRY

Time limit on idle keep-alive connections (seconds)

http.max_connections

ES_HTTP__MAX_CONNECTIONS

The maximum number of concurrent connections that may be established

http.max_keepalive_connections

ES_HTTP__MAX_KEEPALIVE_CONNECTIONS

Allow the connection pool to maintain keep-alive connections below this point. Should be less than or equal to max_connections.

http.rate_limit.*

ES_HTTP__RATE_LIMIT__*

HTTP request rate limit configuration.

http.retry.*

ES_HTTP__RETRY__*

HTTP request retry configuration.

Query Plan Settings#

Field

Environment Variable

Description

query_plan.memory_limit_bytes

ES_QUERY_PLAN__MEMORY_LIMIT_BYTES

Memory limit for query plan execution in bytes. If not set, no limit is enforced. Can be overridden per query.

query_plan.timeout_seconds

ES_QUERY_PLAN__TIMEOUT_SECONDS

Timeout for query plan execution in seconds. If not set, no timeout is enforced. Can be overridden per query.

Warning

See Safety Limits: Memory & Timeout for details on when and why to configure limits.

Rate Limit Settings#

Field

Environment Variable

Description

max_concurrent

*__MAX_CONCURRENT

Maximum number of concurrent requests

max_per_second

*__MAX_PER_SECOND

Maximum number of requests per second

Retry Settings#

Field

Environment Variable

Description

attempts

*__ATTEMPTS

Maximum total number of attempts to perform when retryable errors occur.

timeout

*__TIMEOUT

Maximum total time for all retries when retryable errors occur.

wait_initial

*__WAIT_INITIAL

Minimum backoff before the first retry.

wait_max

*__WAIT_MAX

Maximum backoff time between retries at any time.

wait_jitter

*__WAIT_JITTER

Maximum jitter that is added to retry back-off delays (the actual jitter added is a random number between 0 and wait_jitter)

wait_exp_base

*__WAIT_EXP_BASE

The exponential base used to compute the retry backoff.

Resource References#

References to EarthScope resources.

Field

Environment Variable

Description

resources.api_url

ES_RESOURCES__API_URL

the URL for EarthScope’s API

Settings Loading#

SDK Settings are provided via the following methods (in order of precedence):

  1. initialization arguments (e.g. via class constructors)

  2. environment variables

  3. dotenv file (.env) variables

  4. user’s home directory settings files

    1. ~/.earthscope/config.toml (for configuration)

    2. ~/.earthscope/<profile-name>/tokens.json (for tokens)

  5. legacy EarthScope CLI v0 credentials

  6. bootstrapping environment: the environment variable ES_BOOTSTRAP_SETTINGS may contain a JSON string with any settings in the structure detailed above

  7. default settings

SDK configuration is managed by the SdkSettings class, and calling the constructor performs this settings loading chain.

from earthscope_sdk.config.settings import SdkSettings

settings = SdkSettings()  # loads settings via loading chain

Initialization Arguments#

Arguments to the SdkSettings class constructor take the highest precedence.

For example, the following will configure a custom oauth2 scope to request from the Identity Provider.

from earthscope_sdk.config.settings import SdkSettings

settings = SdkSettings(
  oauth2={"scope": "custom-scope"}
)

Environment Variables#

Environment variables are a convenient way of configuring the SDK. The various settings are available with an ES_ prefix, as detailed in the tables above.

For example, the following will configure a custom oauth2 scope to request from the Identity Provider.

export ES_OAUTH2__SCOPE="custom-scope"

Settings Files#

EarthScope SDK settings files live in the user’s home directory in a “hidden” directory called .earthscope.

  • Mac OS: ~/.earthscope/

  • Unix: ~/.earthscope/

  • Windows: C:\Users\<user>\.earthscope\

Configuration File#

This file is completely optional.

Location: ~/.earthscope/config.toml

The EarthScope SDK can manage tokens for multiple named “profiles”. Profiles are used to manage multiple sets of tokens for interacting with different APIs, or the same API with different tokens.

You use config.toml to enumerate different profiles, and (optionally) unique configuration for each profile.

# ~/.earthscope/config.toml

# (optional) the `default` block contains fallback configuration for _all_ profiles
[default]
oauth2.scope = "default-custom-scope"
# ...etc (all configuration fields are available)

# (optional) `profile.<name>` block(s) define named configuration profile(s)
[profile.foo]
oauth2.scope = "profile-specific-custom-scope"
# ...etc (all configuration fields are available)

Tokens File(s)#

Tokens for individual profiles are stored in their own JSON files:

Location: ~/.earthscope/<profile_name>/tokens.json

The SDK automatically manages these tokens.json files for you. You never need to manually create or edit these files.

{
  "access_token": "<access-token>", // oauth2 Access Token
  "id_token": "<id-token>", // (optional) oauth2 ID Token
  "refresh_token": "<refresh-token>" // (optional) oauth2 Refresh Token
}

Legacy EarthScope CLI v0 Credentials#

The first version of the EarthScope CLI (v0) persisted tokens in a different location and with a different structure. To make upgrading from v0 to v1+ easier, the SDK will look for tokens in the legacy location.

Bootstrapping Environment Settings#

Compute environments such as GeoLab may inject bootstrapping settings via the special ES_BOOTSTRAP_SETTINGS environment variable.

export ES_BOOTSTRAP_SETTINGS='{"oauth2": {"scope": "bootstrap-scope"}}'

Settings provided in this manner are superseded by all settings providers detailed above. This allows the compute environment (e.g. GeoLab) to inject configuration that may be overridden by users.

Default Settings#

The SDK itself provides default values for all configuration fields. These default values are used as a last resort for any field that does not receive a higher-precedence value through the settings loading chain.

These defaults work out of the box for the primary use case of interacting with https://api.earthscope.org.

Contents