Sharing Data Products to the MAAP Data Store

This example demonstrates how to share files to the MAAP data store that have been generated on or uploaded to a local Algorithm Development Environment (ADE).

We start by importing the MAAP package, importing the os module, and creating a new MAAP class.

[1]:
# import the MAAP package
from maap.maap import MAAP

# import the os module
import os

# create MAAP class
maap = MAAP()

For this example, we use the Cloud Optimized GeoTIFF (COG) files located within a directory named “shared-buckets/alexdevseed/cog-tests” that we wish to share. We set a variable to the data directory (in this case '/projects/shared-buckets/alexdevseed/cog-tests') and then use it to create a list of file names. Note that the argument within f.endswith() will change depending on the type of files we wish to share. In the code block below, we display the file list to verify that the directories within the list are correct.

[2]:
# set data directory
path = '/projects/shared-buckets/alexdevseed/cog-tests'

# create a list of files within our data directory which end with '.tif'
files = [f"{path}/{f}" for f in os.listdir(path) if f.endswith('.tif')]

# display the list of files
files
[2]:
['/projects/shared-buckets/alexdevseed/cog-tests/Landsat8_275_comp_cog_2015-2020_dps.tif',
 '/projects/shared-buckets/alexdevseed/cog-tests/boreal_agb_20211015_0249_cog-ovr3.tif',
 '/projects/shared-buckets/alexdevseed/cog-tests/boreal_agb_20211015_0249_cog-ovr4.tif',
 '/projects/shared-buckets/alexdevseed/cog-tests/boreal_agb_20211015_0249_cog-ovr6.tif',
 '/projects/shared-buckets/alexdevseed/cog-tests/boreal_agb_20211015_0249_cog-ovr8.tif',
 '/projects/shared-buckets/alexdevseed/cog-tests/boreal_agb_20211015_0249_cog-s3o8.tif',
 '/projects/shared-buckets/alexdevseed/cog-tests/boreal_agb_20211015_0249_cog.tif']

Then we call the function maap.uploadFiles(), passing the list of file names from the ADE workspace as the argument. This returns a universally unique identifier (uuid) corresponding to the unique directory where the files were uploaded within an Amazon Web Services (AWS) Simple Storage Service (S3) bucket.

[3]:
# returns uuid corresponding to the unique directory where
# the files were uploaded within a MAAP AWS S3 bucket
maap.uploadFiles(files)
[3]:
'Upload file subdirectory: 318d3060-cbb3-4d66-bc06-c6c05b71f095 (keep a record of this if you want to share these files with other users)'

This uuid is used to generate metadata for these files after the user completes the MAAP User Shared Data Questionnaire. The metadata provides descriptive information for data identification and discovery. This questionnaire must be filled out in order for the shared data to appear in the data store. The MAAP User Shared Data Questionnaire may be accessed by clicking “Open User Metadata Form” on the Commands tab of the JuyterLab sidebar.

Open User Metadata Form

The questionnaire contains fields to fill out concerning information about the collection and the data, related URLs, keywords, and temporal and spatial extents.

MAAP User Shared Data Questionnaire

When completing the fields within the questionnaire, use the uuid corresponding to the directory where the files were uploaded for the “Upload Directory” field.

Upload Directory

Go through the sections of the questionnaire, filling out all of the necessary information. Once the questionnaire has been completed, a message should appear showing that the questionnaire response has been received.

Questionnaire response received!

At this point, the data team will be alerted, and a team member will review the generated metadata. After successful review, the team member will publish the metadata. This will trigger the data associated with the data processing job IDs or upload directory ID to be ingested and published to the MAAP data store and MAAP Common Metadata Repository (CMR). Note that while data can be in any format for ingest, the auto-generation of temporal and spatial metadata is only supported for HDF and GeoTIFF formats.