Sharing Data Products to the MAAP Data Store

This example demonstrates how to share files to the MAAP data store that have been generated on or uploaded to a local Algorithm Development Environment (ADE).

We import the MAAP package, import the os module, and create a new MAAP class.

[1]:
# import the MAAP package
from maap.maap import MAAP

# import the os module
import os

# create MAAP class
maap = MAAP()

For this example, we have a Cloud Optimized GeoTIFF (COG) file within a directory named “data” that we wish to share. We set a data directory (in this case “data”) and then use it to create a list of files. Note that the argument within f.endswith() will change depending on the type of files we wish to share. We can display the list of the files to verify the directories within the list are correct.

[4]:
# set data directory
path = 'data'

if not os.path.exists(path):
    os.makedirs(path)

# create a list of files within our data directory which end with '.tif'
files = [f"{path}/{f}" for f in os.listdir(path) if f.endswith('.tif')]

# display the list of files
files
[4]:
[]

We then call maap.uploadFiles() with a list of one or more file names from the ADE workspace (in this case, files or ['data/ndvi_cog_example.tif']). This returns a universally unique identifier (uuid) corresponding to the unique directory where the files were uploaded in a MAAP Amazon Web Services (AWS) Simple Storage Service (S3) bucket.

[5]:
# returns uuid corresponding to the unique directory where the files were uploaded in a MAAP AWS S3 bucket
maap.uploadFiles(files)
[5]:
'Upload file subdirectory: f4a5045e-38a0-4cb7-b8d5-e15b6de35a3a (keep a record of this if you want to share these files with other users)'

Metadata provides descriptive information for data identification and discovery. This uuid is used to generate metadata for these files after the user completes a brief questionnaire. The questionnaire must be filled out in order for the shared data to appear in the data store. Access the questionnaire by clicking “Share your own data” on the Commands tab of the JupyterLab sidebar. Please refer to the following video for a demonstration of sharing data to the MAAP data store, including the next steps to take after running code similar to the above example.

MAAP User-Shared Data WF.mov

Once the questionnaire has been completed, the data team will be alerted, and a team member will review the generated metadata. After successful review, the team member will publish the metadata. This will trigger the data associated with the data processing job IDs or upload directory ID to be ingested and published to the MAAP data store and MAAP Common Metadata Repository (CMR). Note that while data can be in any format for ingest, the auto-generation of temporal and spatial metadata is only supported for HDF and GeoTIFF formats.