Direct DAAC S3 Bucket Access (BETA)

Authors: Alex Mandel (Development Seed), Brian Freitag (NASA MSFC), Jamison French (Development Seed)

Date: September 27, 2023

Description: In this tutorial, we demonstrate how to assume the MAAP data reader role to access specific DAAC buckets.

This tutorial demonstrates an experimental feature to allow access to DAACs without using EarthDataLogin.

This method currently works for a select number of DAACs and their EarthDataCloud datasets which are stored in AWS S3: - NSIDC - ORNL - GES DISC - LPDAAC - PODAAC

Run This Notebook

To access and run this tutorial within MAAP’s Algorithm Development Environment (ADE), please refer to the “Getting started with the MAAP” section of our documentation.

Disclaimer: this tutorial must be run within MAAP’s ADE to assume the necessary permissions. This tutorial was tested using the vanilla workspace image. If you encounter issues with the installs, ensure you have the latest version of pip installed.

Additional Resources

Importing Packages

If the packages below are not installed already, uncomment the following cell.

[ ]:

%pip install h5netcdf fsspec s3fs xarray rioxarray --quiet

[1]:

import boto3
import fsspec
import xarray
import rioxarray
import matplotlib.pyplot as plt
import rasterio
from rasterio.session import AWSSession

Access The Data

We’ll create a couple helper functions to setup the assumed role session and view the data.

[2]:

def assume_role_credentials(ssm_parameter_name):
    # Create a session using your current credentials
    session = boto3.Session()

    # Retrieve the SSM parameter
    ssm = session.client('ssm', "us-west-2")
    parameter = ssm.get_parameter(
        Name=ssm_parameter_name,
        WithDecryption=True
    )
    parameter_value = parameter['Parameter']['Value']

    # Assume the DAAC access role
    sts = session.client('sts')
    assumed_role_object = sts.assume_role(
        RoleArn=parameter_value,
        RoleSessionName='TutorialSession'
    )

    # From the response that contains the assumed role, get the temporary
    # credentials that can be used to make subsequent API calls
    credentials = assumed_role_object['Credentials']

    return credentials

# We can pass assumed role credentials into fsspec
def fsspec_access(credentials):
    return fsspec.filesystem(
        "s3",
        key=credentials['AccessKeyId'],
        secret=credentials['SecretAccessKey'],
        token=credentials['SessionToken']
    )

# We can also pass assumed role credentials into rasterio AWSSession
def rasterio_access(credentials):
    aws_session = AWSSession(
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken']
    )

    return rasterio.Env(aws_session)

Initialize the assumed role sessions

[3]:

s3_fsspec = fsspec_access(assume_role_credentials("/iam/maap-data-reader"))
s3_rasterio = rasterio_access(assume_role_credentials("/iam/maap-data-reader"))

LP DAAC Access

We can also use rasterio to directly inspect our TIF objects.

[9]:

lp_object = "s3://lp-prod-protected/HLSL30.020/HLS.L30.T56JMN.2023225T234225.v2.0/HLS.L30.T56JMN.2023225T234225.v2.0.B11.tif"

with s3_rasterio:
    with rasterio.open(lp_object) as src:
        print(f'Width: {src.width}')
        print(f'Height: {src.height}')
        print(f'Bounds: {src.bounds}')
        print(f'CRS: {src.crs}')
        print(f'Count: {src.count}')
        print(f'Data type: {src.dtypes}')

Width: 3660
Height: 3660
Bounds: BoundingBox(left=399960.0, bottom=-3309780.0, right=509760.0, top=-3199980.0)
CRS: EPSG:32656
Count: 1
Data type: ('int16',)

[ ]:

Direct DAAC S3 Bucket Access (BETA)

Run This Notebook

Additional Resources

Importing Packages

Access The Data

NSIDC DAAC Access

ORNL DAAC Access

PO DAAC Access

GES DISC Access

LP DAAC Access