To access and run this tutorial within MAAP’s Algorithm Development Environment (ADE), please refer to the “Getting started with the MAAP” section of our documentation.
Disclaimer: this tutorial must be run within MAAP’s ADE to assume the necessary permissions. This tutorial was tested using the Pangeo workspace image. If you encounter issues with the installs, ensure you have the latest version of pip installed.
We’ll create a couple helper functions to setup the assumed role session and view the data.
[2]:
defassume_role_credentials(ssm_parameter_name):# Create a session using your current credentialssession=boto3.Session()# Retrieve the SSM parameterssm=session.client('ssm',"us-west-2")parameter=ssm.get_parameter(Name=ssm_parameter_name,WithDecryption=True)parameter_value=parameter['Parameter']['Value']# Assume the DAAC access rolests=session.client('sts')assumed_role_object=sts.assume_role(RoleArn=parameter_value,RoleSessionName='TutorialSession')# From the response that contains the assumed role, get the temporary# credentials that can be used to make subsequent API callscredentials=assumed_role_object['Credentials']returncredentials# We can pass assumed role credentials into fsspecdeffsspec_access(credentials,requester_pays=False):returnfsspec.filesystem("s3",key=credentials['AccessKeyId'],secret=credentials['SecretAccessKey'],token=credentials['SessionToken'],requester_pays=requester_pays)# We can also pass assumed role credentials into rasterio AWSSessiondefrasterio_access(credentials,requester_pays=False):aws_session=AWSSession(aws_access_key_id=credentials['AccessKeyId'],aws_secret_access_key=credentials['SecretAccessKey'],aws_session_token=credentials['SessionToken'],requester_pays=requester_pays)returnrasterio.Env(aws_session)
Accessing GES DISC, LP DAAC and NSIDC Requester Pays Buckets
All DAACs support Temporary S3 Credentials through Earthdata Login (EDL) authentication.
Some NASA DAACs, such as GES DISC, LP DAAC and NSIDC also store protected data in S3 buckets that use the RequesterPays model. To access these datasets, you need temporary AWS credentials and must explicitly declare that you agree to pay for data access. Below is an example of how to authenticate using awsstsassume-role and access a file using gdalinfo.
nsidc_object="s3://nsidc-cumulus-prod-protected/ATLAS/ATL08/006/2023/06/21/ATL08_20230621235543_00272011_006_02.h5"fsspec_caching={"cache_type":"blockcache",# block cache stores blocks of fixed size and uses eviction using a LRU strategy."block_size":8*1024*1024,# size in bytes per block, adjust depends on the file size but the recommended size is in the MB}ds=xarray.open_dataset(s3_fsspec_requesterpays.open(nsidc_object,"rb",**fsspec_caching),group="gt1l/land_segments",engine="h5netcdf",decode_times=True)ds
[6]:
<xarray.Dataset> Size: 5MB
Dimensions: (delta_time: 21468, ds_geosegments: 5, ds_surf_type: 5)
Coordinates:
* delta_time (delta_time) datetime64[ns] 172kB 2023-06-21T23:55:51....
latitude (delta_time) float32 86kB ...
longitude (delta_time) float32 86kB ...
Dimensions without coordinates: ds_geosegments, ds_surf_type
Data variables: (12/41)
asr (delta_time) float32 86kB ...
atlas_pa (delta_time) float32 86kB ...
beam_azimuth (delta_time) float32 86kB ...
beam_coelev (delta_time) float32 86kB ...
brightness_flag (delta_time) float32 86kB ...
cloud_flag_atm (delta_time) float32 86kB ...
... ...
snr (delta_time) float32 86kB ...
solar_azimuth (delta_time) float32 86kB ...
solar_elevation (delta_time) float32 86kB ...
surf_type (delta_time, ds_surf_type) int8 107kB ...
terrain_flg (delta_time) float64 172kB ...
urban_flag (delta_time) float64 172kB ...
Attributes:
Description: Contains data categorized as land at 100 meter intervals.
data_rate: Data are stored as aggregates of 100 meters.
xarray.Dataset
delta_time: 21468
ds_geosegments: 5
ds_surf_type: 5
delta_time
(delta_time)
datetime64[ns]
2023-06-21T23:55:51.147658304 .....
contentType :
referenceInformation
description :
Mean time for the segment in number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.
Latitude of the center-most signal photon within each segment.
long_name :
latitude
source :
Land ATBD section 2.4
standard_name :
latitude
units :
degrees
valid_max :
90.0
valid_min :
-90.0
[21468 values with dtype=float32]
longitude
(delta_time)
float32
...
contentType :
modelResult
description :
Longitude of the center-most signal photon within each segment.
long_name :
longitude
source :
Land ATBD section 2.4
standard_name :
longitude
units :
degrees
valid_max :
180.0
valid_min :
-180.0
[21468 values with dtype=float32]
asr
(delta_time)
float32
...
contentType :
auxiliaryInformation
description :
Apparent surface reflectance
long_name :
apparent surface reflectance
source :
ATL09
units :
1
[21468 values with dtype=float32]
atlas_pa
(delta_time)
float32
...
contentType :
referenceInformation
description :
Off nadir pointing angle (in radians) of the satellite to increase spatial sampling in the non-polar regions. ATLAS_PA =radians(90degs-beam_coelev).
long_name :
atlas pointing angle
source :
ATL03
units :
radians
[21468 values with dtype=float32]
beam_azimuth
(delta_time)
float32
...
contentType :
referenceInformation
description :
Azimuth(in radians) of the unit pointing vector for the reference photon in the local ENU frame in radians. The angle is measured from north and positive towards East.
long_name :
beam azimuth
source :
ATL03
units :
radians
[21468 values with dtype=float32]
beam_coelev
(delta_time)
float32
...
contentType :
referenceInformation
description :
Co-elevation (CE) is direction from vertical of the laser beam as seen by an observer located at the laser ground spot.
long_name :
beam co-elevation
source :
ATL03
units :
radians
[21468 values with dtype=float32]
brightness_flag
(delta_time)
float32
...
contentType :
qualityInformation
description :
Flag indicating that the ground surface is bright (e.g. snow-covered or other bright surfaces)
flag_meanings :
not_bright_surface bright_surface
flag_values :
[0 1]
long_name :
brightness flag
source :
Land ATBD section 2.4.21
units :
1
[21468 values with dtype=float32]
cloud_flag_atm
(delta_time)
float32
...
contentType :
qualityInformation
description :
Cloud confidence flag from ATL09 that indicates the number of cloud or aerosol layers identified in each 25Hz atmospheric profile. If the flag is greater than 0, aerosols or clouds could be present. Valid range is 0 - 10.
long_name :
cloud flag atm
source :
ATL09
units :
1
valid_max :
10
valid_min :
0
[21468 values with dtype=float32]
cloud_fold_flag
(delta_time)
float32
...
contentType :
modelResult
description :
Flag that indicates this profile likely contains cloud signal folded down from above 15 km to the last 2-3 km of the profile. See ATL09 ATBD Table 3.9 for detailed flag value meanings.
Time of the first photon contained within the data segment, in seconds since the ATLAS SDP GPS Epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.
long_name :
delta time begin
source :
Derived (gps_seconds-gps_sec_offset)
[21468 values with dtype=datetime64[ns]]
delta_time_end
(delta_time)
datetime64[ns]
...
contentType :
referenceInformation
description :
Time of the last photon contained within the data segment, in seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.
long_name :
delta time end
source :
Derived (gps_seconds-gps_sec_offset)
[21468 values with dtype=datetime64[ns]]
dem_flag
(delta_time)
float32
...
contentType :
referenceInformation
description :
Indicates source of the DEM height. Values: 0=None, 1=Arctic, 2=Global, 3=MSS, 4=Antarctic.
flag_meanings :
none arctic global mss antarctic
flag_values :
[0 1 2 3 4]
long_name :
dem source flag
source :
Atmosphere ATBD
units :
1
valid_max :
4
valid_min :
0
[21468 values with dtype=float32]
dem_h
(delta_time)
float32
...
contentType :
referenceInformation
description :
Best available DEM (in priority of Arctic/Antarctic/Global/MSS) value at the geolocation point. Height is in meters above the WGS84 Ellipsoid.
long_name :
dem height
source :
Arctic, Antarctic, Global, MSS DEM
units :
meters
[21468 values with dtype=float32]
dem_removal_flag
(delta_time)
int8
...
contentType :
qualityInformation
description :
Flag indicating more than dem_removal_percent_limit (default 20.0) removed from land segment due to failing DEM-QA tests
flag_meanings :
below_threshold above_threshold
flag_values :
[0 1]
long_name :
dem removal flag
source :
ATBD section 2.4.11
units :
1
[21468 values with dtype=int8]
h_dif_ref
(delta_time)
float32
...
contentType :
modelResult
description :
Difference between h_te_median and ref_DEM
long_name :
h dif from reference
source :
Land ATBD section 2.4
units :
meters
[21468 values with dtype=float32]
last_seg_extend
(delta_time)
float32
...
contentType :
auxiliaryInformation
description :
The distance (km) that the last ATL08 processing segment in a file is either extended or overlapped with the previous ATL08 processing segment.
long_name :
last segment extended
source :
Land ATBD 13March2019, Section 2.4.20
standard_name :
last_seg_extend
units :
kilometers
[21468 values with dtype=float32]
latitude_20m
(delta_time, ds_geosegments)
float32
...
contentType :
modelResult
description :
Center latitude of 20m geosegments within each 100m land segment.
long_name :
latitude of 20m segments
source :
ATBD January 2021, Section TBD
units :
degree
valid_max :
90.0
valid_min :
-90.0
[107340 values with dtype=float32]
layer_flag
(delta_time)
int8
...
contentType :
modelResult
description :
This flag is a combination of multiple flags (cloud_flag_atm, cloud_flag_asr, and bsnow_con) and takes daytime/nighttime into consideration. A value of 1 means clouds or blowing snow are likely present. A value of 0 indicates the likely absence of clouds or blowing snow.
flag_meanings :
likely_clear likely_cloudy
flag_values :
[0 1]
long_name :
consolidated cloud flag
source :
ATL09
units :
1
[21468 values with dtype=int8]
longitude_20m
(delta_time, ds_geosegments)
float32
...
contentType :
modelResult
description :
Center longitude of 20m geosegments within each 100m land segment.
long_name :
longitude of 20m segments
source :
ATBD January 2021, section TBD
units :
degree
valid_max :
180.0
valid_min :
-180.0
[107340 values with dtype=float32]
msw_flag
(delta_time)
float32
...
contentType :
qualityInformation
description :
Multiple Scattering warning flag. The multiple scattering warning flag (ATL09 parameter msw_flag) has values from -1 to 5 where zero means no multiple scattering and 5 the greatest. If no layers were detected, then msw_flag = 0. If blowing snow is detected and its estimated optical depth is greater than or equal to 0.5, then msw_flag = 5. If the blowing snow optical depth is less than 0.5, then msw_flag = 4. If no blowing snow is detected but there are cloud or aerosol layers detected, the msw_flag assumes values of 1 to 3 based on the height of the bottom of the lowest layer: < 1 km, msw_flag = 3; 1-3 km, msw_flag = 2; > 3km, msw_flag = 1. A value of -1 indicates that the signal to noise of the data was too low to reliably ascertain the presence of cloud or blowing snow. We expect values of -1 to occur only during daylight.
Flag indicating the data were acquired in night conditions: 0=day, 1=night. Flag is derived from solar elevation at the geolocated segment. IF solar elevation is above threshold it is day, if not then it is night. Threshold is set in atlas_l3a_const_mod.
flag_meanings :
day night
flag_values :
[0 1]
long_name :
night flag
source :
Land ATBD section 2.4.8
units :
1
[21468 values with dtype=float64]
ph_ndx_beg
(delta_time)
int64
...
contentType :
auxiliaryInformation
description :
Index (1-based) within the photon-rate data (/land_segments/photons) of the first photon within this each land segment.
long_name :
photon index begin
source :
Derived
units :
1
[21468 values with dtype=int64]
ph_removal_flag
(delta_time)
int8
...
contentType :
auxiliaryInformation
description :
Flag indicating more than ph_removal_percent_limit (default 50.0) removed from land segment due to failing QA tests
flag_meanings :
below_threshold above_threshold
flag_values :
[0 1]
long_name :
ph removal flag
source :
ATBD section 4.13
units :
1
[21468 values with dtype=int8]
psf_flag
(delta_time)
int8
...
contentType :
referenceInformation
description :
Flag is set to 1 if the point spread function (computed as sigma_atlas_land) has exceeded the threshold (1 m)
flag_meanings :
below_threshold above_threshold
flag_values :
[0 1]
long_name :
point spread function flag
source :
Land/Veg ATBD
units :
1
valid_max :
1
valid_min :
0
[21468 values with dtype=int8]
rgt
(delta_time)
int16
...
contentType :
referenceInformation
description :
The reference ground track (RGT) is the track on the earth at which a specified unit vector within the observatory is pointed. Under nominal operating conditions, there will be no data collected along the RGT, as the RGT is spanned by GT3 and GT4. During slews or off-pointing, it is possible that ground tracks may intersect the RGT. The ICESat-2 mission has 1387 RGTs.
long_name :
reference ground track
source :
Operations
units :
1
valid_max :
1387
valid_min :
1
[21468 values with dtype=int16]
sat_flag
(delta_time)
float32
...
contentType :
auxiliaryInformation
description :
Flag derived from full_sat_fract on the ATL03 data product, averaged over 5 geosegments in 100m land segment
Daily snow/ice cover from ATL09 at the 25 Hz rate(275m) indicating likely presence of snow and ice within each segment. 0=ice free water; 1=snow free land; 2=snow; 3=ice.
flag_meanings :
ice_free_water snow_free_land snow ice
flag_values :
[0 1 2 3]
long_name :
segment snowcover
source :
ATBD section 4.2.16
units :
1
valid_max :
3
valid_min :
0
[21468 values with dtype=float32]
segment_watermask
(delta_time)
float64
...
contentType :
auxiliaryInformation
description :
Water mask(i.e. flag) indicating inland water as referenced from the Global Raster Water Mask(ANC33) at 250 m spatial resolution.
flag_meanings :
no_water water
flag_values :
[0 1]
long_name :
segment watermask
source :
ATBD section 2.4.15
units :
1
valid_max :
1
valid_min :
0
[21468 values with dtype=float64]
sigma_across
(delta_time)
float32
...
contentType :
referenceInformation
description :
Total cross-track uncertainty due to PPD and POD knowledge. Read from ATL03 product gtx/geolocation/sigma_across. Sigma_atlas_y is reported on ATL08 as the uncertainty of the center-most reference photon of the 100m ATL08 segment.
long_name :
sigma atlas y
source :
ATL03
units :
1
[21468 values with dtype=float32]
sigma_along
(delta_time)
float32
...
contentType :
referenceInformation
description :
Total along-track uncertainty due to PPD and POD knowledge. Read from ATL03 product gtx/geolocation/sigma_along. Sigma_atlas_x is reported on ATL08 as the uncertainty of the center-most reference photon of the 100m ATL08 segment.
long_name :
sigma atlas x
source :
ATL03
units :
1
[21468 values with dtype=float32]
sigma_atlas_land
(delta_time)
float32
...
contentType :
referenceInformation
description :
Total vertical geolocation error due to ranging and local surface slope. The parameter is computed for ATL08 as described in equation 1.2.
long_name :
sigma atlas land
source :
Land ATBD section 2.5.13
units :
1
[21468 values with dtype=float32]
sigma_h
(delta_time)
float32
...
contentType :
referenceInformation
description :
Estimated uncertainty for the reference photon bounce point ellipsoid height: 1- sigma (m) provided at the geolocation segment rate on ATL03. Sigma_h is reported on ATL08 as the uncertainty of the center-most reference photon of the 100m ATL08 segment.
long_name :
height uncertainty
source :
ATL03
units :
1
[21468 values with dtype=float32]
sigma_topo
(delta_time)
float32
...
contentType :
referenceInformation
description :
Total uncertainty that include sigma_h plus geolocation uncertainty due to local slope (equation 1.3). The local slope is multiplied by the geolocation uncertainty factor. This will be used to determine the total vertical geolocation error due to ranging and local slope.
long_name :
sigma atlas topo
source :
Land ATBD section 2.5.12
units :
1
[21468 values with dtype=float32]
snr
(delta_time)
float32
...
contentType :
modelResult
description :
The signal to noise ratio of geolocated photons as determined by the ratio of the superset of ATL03 signal and DRAGANN found signal photons used for processing the ATL08 segments to the background photons (i.e. noise) within the same ATL08 segments.
long_name :
signal to noise ratio
source :
ATBD section 2.5.14
units :
1
[21468 values with dtype=float32]
solar_azimuth
(delta_time)
float32
...
contentType :
referenceInformation
description :
The direction, eastwards from north, of the sun vector as seen by an observer at the laser ground spot.
long_name :
solar azimuth
source :
ATL03g ATBD
units :
degrees_east
[21468 values with dtype=float32]
solar_elevation
(delta_time)
float32
...
contentType :
referenceInformation
description :
Solar Angle above or below the plane tangent to the ellipsoid surface at the laser spot. Positive values mean the sun is above the horizon, while negative values mean it is below the horizon. The effect of atmospheric refraction is not included. This is a low precision value, with approximately TBD degree accuracy.
long_name :
solar elevation
source :
ATL03g ATBD
units :
degrees
[21468 values with dtype=float32]
surf_type
(delta_time, ds_surf_type)
int8
...
contentType :
referenceInformation
description :
Flags describing which surface types this interval is associated with. 0=not type, 1=is type. Order of array is land, ocean, sea ice, land ice, inland water.
flag_meanings :
not_type is_type
flag_values :
[0 1]
long_name :
surface type
source :
ATL03 ATBD, Section 4
units :
1
valid_max :
1
valid_min :
0
[107340 values with dtype=int8]
terrain_flg
(delta_time)
float64
...
contentType :
auxiliaryInformation
description :
Terrain flag quality check to indicate a deviation above a threshold from the reference DEM height reported on the product.
flag_meanings :
below_threshold above_threshold
flag_values :
[0 1]
long_name :
terrain flag
source :
Land ATBD section 2.4.8
units :
1
[21468 values with dtype=float64]
urban_flag
(delta_time)
float64
...
contentType :
qualityInformation
description :
The urban flag indicates that a segment is likely located over an urban area.