Documentation

§ Overview

Harmony provides access to services that can transform data from NASA's Earth Observing Systems Data and Information System (EOSDIS) Distributed Active Archive Centers (DAAC). Transformations can be requested using one of two Open Geospatial Consortium (OGC) inspired APIs.

Data processed by Harmony is staged in AWS S3 buckets owned by NASA or optionally in user owned S3 buckets. Harmony provides signed URLs or temporary access credentials to users for data staged in NASA S3 buckets.

Data transformation requests are executed as jobs in Harmony. Harmony provides the ability for users to monitor and interact with long-running jobs, both programmatically through an API and via a web-based user interface.

This document describes the APIs and endpoints Harmony provided. You can use these APIs and endpoints to directly interact with Harmony, but the recommended official way to interact with Harmony is through Harmony-Py.

This documentation covers the following:

§ Getting Started

All users will need an Earthdata Login (EDL) account in order to access NASA data and services. Once a user has an EDL username and password they will need to use these when accessing Harmony. They can be used directly in a browser request (the browser will prompt for them), in another client like curl, or in code.

For curl or in code the easiest approach is to place your EDL credentials in a .netrc file.

A sample .netrc file looks like this

machine urs.earthdata.nasa.gov login my-edl-user-name password my-edl-password

COPIED
COPY

Example 2 - Sample .netrc file

Make sure that this file is only readable by the current user or you will receive an error stating "netrc access too permissive."

$ chmod 0600 ~/.netrc
COPIED
COPY

Example 3 - Setting permissions on the .netrc file (Unix/macOS)

Alternatively users can generate an EDL bearer token directly and pass this to Harmony using an Authorization: Bearer header.

§ Passing credentials with curl

Use the -n flag to use your .netrc file with curl. You will also need to pass the -L flag (to handle the redirect from Harmony to EDL and back) and the -b and -j flags to properly handle cookies used during the authentication.

curl -Lnbj https://harmony.earthdata.nasa.gov/C1234208438-POCLOUD/ogc-api-coverages/1.0.0/collections/bathymetry/coverage/rangeset

COPIED
COPY

Example 4 - Curl flags to handle EDL authentication when using a .netrc file

To work directly with a bearer token from EDL you can use an Authorization: Bearer my-bearer-token header as follows:

curl -H "Authorization: Bearer <my-bearer-token>" https://harmony.earthdata.nasa.gov/C1234208438-POCLOUD/ogc-api-coverages/1.0.0/collections/bathymetry/coverage/rangeset

COPIED
COPY

Example 5 - Using a bearer token with curl

§ Passing credentials in code

The following Python example uses the netrc, request, and cookiejar libraries to set up authentication with EDL. No error handling is included in this example.

import netrc
from urllib import request, parse
from http.cookiejar import CookieJar

def setup_earthdata_login_auth(endpoint):
    """
    Set up the request library so that it authenticates against the given Earthdata Login
    endpoint and is able to track cookies between requests.  This uses the .netrc file.
    """
    username, _, password = netrc.netrc().authenticators(endpoint)
    manager = request.HTTPPasswordMgrWithDefaultRealm()
    manager.add_password(None, endpoint, username, password)
    auth = request.HTTPBasicAuthHandler(manager)

    jar = CookieJar()
    processor = request.HTTPCookieProcessor(jar)
    opener = request.build_opener(auth, processor)
    request.install_opener(opener)

setup_earthdata_login_auth('urs.earthdata.nasa.gov')

COPIED
COPY

Example 6 - Authenticating in python

The username and password can also be set directly instead of using a .netrc file.

There is significant boiler-plate code involved in connecting to Harmony that can be avoided by using the harmony-py library. The equivalent code using harmony-py (when using a .netrc file) can be as simple as

from harmony import Client

harmony_client = Client() # defaults to Harmony production endpoint

COPIED
COPY

Example 7 - Using harmony-py to create a client with .netrc EDL authentication

harmony-py provides many other conveniences when using Harmony services. For these reasons harmony-py is the suggested way to access Harmony in code. For complete details see the documentation.

§ Using the Service APIs

This section provides an introduction to the Harmony service APIs for launching services for collections using either OGC Coverages or WMS. For more details on the OGC Coverages API see the API Documentation.

Each API requires a CMR collection concept ID or short name, and transformations can be performed using one of the following endpoints ({collectionId} and {variable} are placeholders):

https://harmony.earthdata.nasa.gov/{collectionId}/ogc-api-coverages/1.0.0/{variable}/coverage/rangeset

COPIED
COPY

Example 8 - OGC Coverages endpoint

https://harmony.earthdata.nasa.gov/{collectionId}/wms

COPIED
COPY

Example 9 - WMS endpoint

§ OGC Coverages Request Parameters

The primary Harmony services REST API conforms to the OGC Coverages API version 1.0.0. As such it accepts parameters in the URL path as well as query parameters.

§ URL Path Parameters

parameter	description
collection	(required) This is the NASA EOSDIS collection or data product. There are two options for inputting a collection of interest: 1. Provide a concept ID, which is an ID provided in the Common Metadata Repository (CMR) metadata 2. Use the data product short name, e.g. SENTINEL-1_INTERFEROGRAMS. Must be URL encoded.
variable	(required) Names or concept ids of the UMM-Var variables to be retrieved, or "all" to retrieve all variables. Multiple variables may be retrieved by separating them with a comma. The psuedo-variable "parameter_vars" may be used, in which case the variables are specified using the variable query parameter or variable form data parameter. This is useful if you need to subset using many variables.

---

Table 1 - Harmony OGC Coverages API URL path (required) parameters

§ Query Parameters

parameter	description
subset	get a subset of the coverage by slicing or trimming along one axis. Harmony supports the axes "lat" and "lon" for spatial subsetting, and "time" for temporal, regardless of the names of those axes in the data files. Harmony also supports arbitrary dimension names for subsetting on numeric ranges for that dimension.
outputCrs	reproject the output coverage to the given CRS. Recognizes CRS types that can be
interpolation	specify the interpolation method used during reprojection and scaling
scaleExtent	scale the resulting coverage along one axis to a given extent
scaleSize	scale the resulting coverage along one axis to a given size
concatenate	requests results to be concatenated into a single result
granuleId	the CMR Granule ID for the granule which should be retrieved
granuleName	passed to the CMR search as the readable_granule_name parameter. Supports * and ? wildcards for multiple and single character matches. Wildcards can be used any place in the name, but leading wildcards are discouraged as they require a lot of resources for the underlying search
grid	the name of the output grid to use for regridding requests. The name must match the UMM
point	only collections that have a geometry that contains a spatial point are selected. The spatial point is provided as two numbers: * Longitude, coordinate axis 1 * Latitude, coordinate axis 2 The coordinate reference system of the values is WGS84 longitude/latitude.
width	number of columns to return in the output coverage
height	number of rows to return in the output coverage
forceAsync	if "true", override the default API behavior and always treat the request as asynchronous
format	the mime-type of the output format to return
maxResults	limits the number of input files processed in the request
skipPreview	if "true", override the default API behavior and never auto-pause jobs
ignoreErrors	if "true", continue processing a request to completion even if some items fail. If "false" immediately fail the request. Defaults to true
destinationUrl	destination url specified by the client; currently only s3 link urls are supported (e.g. s3://my-bucket-name/mypath) and will result in the job being run asynchronously
variable	the variable(s) to be used for variable subsetting. Multiple variables can be specified as a comma-separated list. This parameter is only used if the url variable path element is "parameter_vars"

---

Table 2 - Harmony OGC Coverages API query parameters

For POST requests the body should be multipart/form-data and may also contain

• shape: perform a shapefile subsetting request on a supported collection by passing the path to a GeoJSON file (*.json or .geojson), an ESRI Shapefile (.zip or .shz), or a kml file (.kml) as the "shape" parameter

A sample OGC Coverages request is as follows

curl -Lnbj https://harmony.earthdata.nasa.gov/C1234208438-POCLOUD/ogc-api-coverages/1.0.0/collections/bathymetry/coverage/rangeset?maxResults=1

COPIED
COPY

Example 10 - Curl command for an OGC Coverages request

§ WMS Requests

Harmony provides an implementation of the OGC Web Map Service (WMS) API version 1.3.0. Harmony only supports the GetCapabilities and GetMap requests.

The API uses both URL path and query parameters.

§ URL Path Parameters

parameter	required	description
collection	Y	this parameter is the same as the collection parameter described in the OGC Coverages API above.

---

Table 3 - Harmony WMS API URL path (required) parameters

§ Common Query Parameters

parameter	required	description
service	Y	the service for the request. Must be equal to 'WMS'
version	Y	the WMS version to use. Must be equal to '1.3.0'
request	Y	the action being requested. Valid values are GetCapabilities and GetMap

---

Table 4 - Required query parameters for both GetCapabilities and GetMap

§ Query Parameters for GetMap - Standard WMS

parameter	required	description
layers	Y	comma-separated list of layer names to display on map
bbox	Y	the bounding box for the map as comma separated values in WSEN order
crs	Y	Spatial Reference System for map output. Value is in form EPSG:nnn
format	Y	output format mime-type
styles	Y	Styles in which layers are to be rendered. Value is a comma-separated list of style names, or empty if default styling is required. Style names may be empty in the list, to use default layer styling.
width	Y	width in pixels of the output
height	Y	height in pixels of the output
bgcolor	N	Background color for the map image. Value is in the form RRGGBB. Default is FFFFFF (white).
exceptions	N	Format in which to report exceptions. Default value is application/vnd.ogc.se_xml
transparent	N	whether the output background should be transparent (true or false). default is false

---

Table 5 - Standard WMS query parameters for GetMap

§ Additional Harmony parameters for WMS requests

parameter	required	description
dpi	N	the dots-per-inch (DPI) resolution for image output
map_resolution	N	the DPI resolution for image output
granuleId	N	the CMR Granule ID for the granule of interest
granuleName	N	passed to the CMR search as the readable_granule_name parameter. Supports * and ? wildcards for multiple and single character matches. Wildcards can be used any place in the name, but leading wildcards are discouraged as they require a lot of resources for the underlying search

---

Table 6 - Additional (non-OGC) query parameters for Harmony WMS queries

GetCapabilities requests return an XML document, while GetMap requests return an image.

§ Available Services

Harmony requests are declarative rather than imperative, so a request specifies the particular data of interest, time range of interest, spatial bounds of interest, desired output format, etc. Harmony then matches this declaration against available services and invokes the matching services on behalf of the user. All of which is to say the user does not request specific services directly. Despite this, it can be useful for a user to know what services are available, what their capabilities are, and which services can work together.

§ Service Versions

Harmony services run in containers in pods in a Kubernetes cluster. It is not possible for users to interact directly with these pods, but it may be useful to know some of the details about the running versions. The specific docker image and tag for each service can be retrieved from the versions route.

§ Service Capabilities

The following tables provide an overview of the deployed services with a description of each and what capabilities they provide.

§ User Owned Buckets for Harmony Output

Users may store Harmony output directly in their own S3 buckets by specifying the bucket/path in their requests with the destinationUrl parameter. For example

https://harmony.earthdata.nasa.gov/C1234208438-POCLOUD/ogc-api-coverages/1.0.0/collections/bathymetry/coverage/rangeset?concatenate=true&subset=lon(-160%3A160)&subset=lat(-80%3A80)&skipPreview=true&maxResults=1&destinationUrl=3%3A%2F%2Fmy-staging-bucket

COPIED
COPY

Example 11 - Request to store output in user owned S3 bucket

would place the output in s3://my-example-bucket. Note that the value of destinationUrl must be a full S3 path and must be URL encoded.

Four things are required to enable Harmony to write to your bucket.

1. The bucket must be in the same AWS region as Harmony, i.e., us-west-2.
2. The bucket must have ACLs disabled. This is the default for S3 buckets.
3. Harmony must have permission to write to the bucket.
4. Harmony must have permission to check the bucket's location.

Numbers three and four on the list can be accomplished by setting an appropriate bucket policy. You can obtain a bucket policy for your bucket using the policy generator at https://harmony.earthdata.nasa.gov/staging-bucket-policy and passing in the bucketPath query parameter. For example

https://harmony.earthdata.nasa.gov/staging-bucket-policy?bucketPath=my-example-bucket

The bucketPath parameter can be one of the following

1. A bucket name, e.g., my-example-bucket
2. A bucket name + path, e.g., my-example-bucket/my/path
3. A full S3 url with our without a path, e.g., s3://my-example-bucket/my/path

The third option is compatible with the destinationUrl parameter for requests.

A sample policy generated by the endpoint is shown below:

{
  'Version': '2012-10-17',
  'Statement': [
    {
      'Sid': 'write permission',
      'Effect': 'Allow',
      'Principal': {
        'AWS': 'arn:aws:iam::123456789012:root',
      },
      'Action': 's3:PutObject',
      'Resource': 'arn:aws:s3:::my-bucket/*',
    },
    {
      'Sid': 'get bucket location permission',
      'Effect': 'Allow',
      'Principal': {
        'AWS': 'arn:aws:iam::123456789012:root',
      },
      'Action': 's3:GetBucketLocation',
      'Resource': 'arn:aws:s3:::my-bucket',
    },
  ]
}

COPIED
COPY

Example 12 - Sample bucket policy to enable writing Harmony output

§ Summary of Available Endpoints

All of the public endpoints for Harmony users other than the OGC Coverages and WMS APIs are listed in the following table. The Coverages and WMS APIs are described in the Using the Service APIs section.

route	description
/	The Harmony landing page
/capabilities	Get harmony capabilities for the provided collection
/cloud-access	Generates AWS credentials for accessing processed data in S3
/docs	These documentation pages
/docs/api	The Swagger documentation for the OGC Coverages API
/jobs	The jobs API for getting job status, pausing/continuing/canceling jobs
/stac	The API for retrieving STAC catalog and catalog items for processed data
/staging-bucket-policy	The policy generator for external (user) bucket storage
/versions	Returns JSON indicating the image and tag each deployed service is running
/service-image-tag	The API for managing service image tags/versions
/workflow-ui	The Workflow UI for monitoring and interacting with running jobs

---

Table 7 - Harmony routes other than OGC Coverages and WMS

§ Get Harmony capabilities for the provided collection

Returns information related to what Harmony operations are supported for a given collection in JSON format. The collection can be identified by either collection concept id or short name.

§ Query Parameters

Either collectionId or shortName must be provided.

parameter	description
collectionId	Concept id of the collection to retrieve capabilities for
shortName	Short name of the collection to retrieve capabilities for
version	(optional) The version of the capabilities result format, currently only 1 or 2 is supported.

---

Table 8 - Harmony capabilities endpoint parameters

§ Response

The returned JSON response will have the configured capabilities in Harmony for the given collection in terms of supported features (e.g. variable subsetting, boundingbox subsetting, concatenation, reprojection, etc.), the output formats, the list of Harmony services that are applicable for the collection, the list of variables that are associated with the collection and the version of the capabilites result format. See below for the root level fields in the capabilites response and their descriptions:

field	description
conceptId	Concept id of the collection
shortName	Short name of the collection
variableSubset	(boolean) True if variable subsetting is supported by any of the Harmony services for the collection.
bboxSubset	(boolean) True if bounding box subsetting is supported by any of the Harmony services for the collection.
shapeSubset	(boolean) True if shape file subsetting is supported by any of the Harmony services for the collection.
concatenate	(boolean) True if concatenation is supported by any of the Harmony services for the collection.
reproject	(boolean) True if reprojection is supported by any of the Harmony services for the collection.
outputFormats	A list of supported output formats for the collection.
services	A list of JSON objects describing the supported Harmony services for the collection.
variables	A list of JSON objects describing the associated variables of the collection.
capabilitiesVersion	The version of the capabilities result format.

---

Table 9 - Harmony capabilities endpoint response fields

§ Getting Harmony capabilities for a given collection by collection concept id

https://harmony.earthdata.nasa.gov/capabilities?collectionId=<collection-concept-id>

COPIED
COPY

Example 13 - Getting Harmony capabilities for a given collection by collection concept id

§ Getting Harmony capabilities for a given collection by collection short name

https://harmony.earthdata.nasa.gov/capabilities?shortName=<collection-short-name>

COPIED
COPY

Example 14 - Getting Harmony capabilities for a given collection by collection short name

§ Monitoring Jobs with the Jobs API and the Workflow-UI

Jobs can be monitored using the jobs API as well as with the Workflow-UI web application.

There are two set of jobs API endpoints with the same sub paths and parameters: one with path jobs to view and control the current user's own jobs; the other with admin/jobs path to view and control all users' jobs if the current user has admin permission to do so. For simplicity, we will only list the ones for regular user below.

§ Getting the list of jobs for a user

https://harmony.earthdata.nasa.gov/jobs

COPIED
COPY

Example 15 - Getting the user's list of jobs using the jobs API

Returns the list of Harmony jobs submitted by the user. By default, 10 jobs are returned in the response. User can use the paging query parameters to page through the whole result set or/and change the number of jobs that will be returned in each page.

§ Query Parameters

parameter	description
page	Current page number
limit	Number of jobs in each page

---

Table 10 - Harmony jobs endpoint parameters

§ Response

The returned JSON response will list the total number of jobs that blong to the user, details of the jobs on the current page and links to traverse to the previous, next, first or last pages of the result set:

field	description
count	Total number of jobs
jobs	A list of JSON objects with fields describe the job in details. For details, see job status response.
links	A list of links to traverse to the previous, next, first or last pages of the result set.

---

Table 11 - Harmony jobs response fields

§ Getting job status

Get details for a given job.

https://harmony.earthdata.nasa.gov/jobs/<job-id>

COPIED
COPY

Example 16 - Getting job status

§ Response

The returned JSON response list the details of the given job:

field	description
username	Username that owns the job
status	Status of the job
message	Processing message of the job
progress	Percentage of the job processing progress. 100 for a job that has been processed completely.
createdAt	Timestamp when the job was submitted to Harmony
updatedAt	Timestamp when the job was last updated in Harmony
dataExpiration	Timestamp when the result data of the job will be cleaned up from Harmony
links	A list of JSON objects with links to STAC catalog and result data of the job
request	The original request url of the job
numInputGranules	number of input granules in the job
jobID	ID of the job in Harmony

---

Table 12 - Harmony job response fields

§ Pausing a job

User can pause a job with the following API call. The returned response is the same as the job status response with the job status as paused.

https://harmony.earthdata.nasa.gov/jobs/<job-id>/pause

COPIED
COPY

Example 17 - Pausing a running job

§ Resuming a paused job

User can resume a paused job with the following API call. The returned response is the same as the job status response with the job status as running.

https://harmony.earthdata.nasa.gov/jobs/<job-id>/resume

COPIED
COPY

Example 18 - Resuming a paused job

§ Canceling a job

User can cancel a job with the following API call. The returned response is the same as the job status response with the job status as canceled.

https://harmony.earthdata.nasa.gov/jobs/<job-id>/cancel

COPIED
COPY

Example 19 - Canceling a running job

§ Skipping preview

Jobs involving many granules will by default pause automatically after the first few granules are processed. This allows the user to examine the output to make sure things look right before waiting for the entire job to complete. If things looks good, the user can resume the paused job, if not they can cancel the paused job.

If a user wishes to skip this step they can pass the skipPreview flag mentioned in the Services API section, or they can tell an already running job to skip the preview using the following:

https://harmony.earthdata.nasa.gov/jobs/<job-id>/skip-preview

COPIED
COPY

Example 20 - Skipping the preview on a many-granule job

§ Retrieving STAC catalog and catalog items for processed data

Harmony uses STAC catalog to provide input and output, and communicate between services. The following endpoints provides access to the output STAC catalog and catalog items of a finished job.

§ Getting the STAC catalog of a single job

https://harmony.earthdata.nasa.gov/stac/<job-id>

COPIED
COPY

Example 21 - Getting the STAC catalog of a single job

§ Getting the STAC catalog item within a job's STAC catalog

The <item-index> is the index of the STAC catalog item within the job's output STAC catalog. It starts at 0.

https://harmony.earthdata.nasa.gov/stac/<job-id>/<item-index>

COPIED
COPY

Example 22 - Getting the STAC catalog item within a job's STAC catalog

§ Getting in region S3 AWS access keys using the cloud-access API

Harmony supports analysis in place without forcing a user to download the outputs from their requests. The results for each harmony request are stored an AWS S3 bucket in the us-west-2 region. In order to access these results natively in S3 the user can get temporary AWS access credentials using the harmony cloud-access endpoints. Note that data can only be accessed from within the us-west-2 region.

§ Get AWS S3 access credentials as JSON

https://harmony.earthdata.nasa.gov/cloud-access

COPIED
COPY

Example 23 - Getting AWS S3 access credentials as JSON using the cloud-access API

The returned JSON response returns fields required to set the AWS S3 access credentials:

field	description
AccessKeyId	The AWS access key ID.
SecretAccessKey	The AWS secret access key.
SessionToken	The AWS session token associated with the access keys.
Expiration	The date and time when the access credentials expire.

---

Table 13 - Harmony cloud-access response fields

§ Get AWS S3 access credentials as a shell script to use in a terminal

https://harmony.earthdata.nasa.gov/cloud-access.sh

COPIED
COPY

Example 24 - Getting a shell script that can be sourced to set AWS S3 credentials

§ Getting the versions of images used in each of the Harmony backend service chains

Using the versions endpoint, a user can get a list of all of the docker images and versions used in harmony backend service chains. Service providers use this endpoint to verify the versions of their services deployed to an environment.

§ Get backend service version information

https://harmony.earthdata.nasa.gov/versions

COPIED
COPY

Example 25 - Getting backend service versions using the versions API

The returned JSON response is an array of service version information with the following fields:

field	description
name	The name of the backend service chain as defined in services.yml.
images	An array of the images used in the service chain.

Each image has the following fields:

field	description
image	The name of the docker image.
tag	The docker image tag.
lastUpdated	(optional) The time the image was last updated if available.
imageDigest	(optional) The image digest if available.

---

Table 14 - Harmony versions response fields