{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Adding Cloud-Optimized GeoTIFFs to the MAAP Biomass Earthdata Dashboard\n", "\n", "Author(s): Aimee Barciauskas (Development Seed)\n", "\n", "Date: Oct 14, 2021\n", "\n", "Description: The following notebook steps through how to add a dataset to the MAAP Dashboard.\n", "\n", "Note, there are 2 scenarios:\n", "\n", "1. Adding a single Cloud-Optimized GeoTIFF (COG), and\n", "2. Adding many distinct COGs as a \"mosaic\" with mosaicJSON.\n", "\n", "High-level, the steps are:\n", "\n", "1. Inspect your Cloud-Optimized GeoTIFF(s) (COGs) to understand the best rescale and colormap name parameters. Optionally create a mosaic.\n", "2. Define a colormap. Colormaps provide mappings of data values to RGB values.\n", "3. Create a PR to the datasets repo to add or update your dataset.\n", "\n", "The MAAP dashboard has 3 environments:\n", "\n", "1. Developer-in-test (DIT): https://biomass.dit.maap-project.org\n", "2. Staging: https://biomass.staging.maap-project.org\n", "3. Production: https://biomass.maap-project.org\n", "\n", "These instructions will guide you towards adding your dataset to `biomass.dit.maap-project.org`. The MAAP Dashboard team will \"promote\" changes to staging and production periodically (release schedule forthcoming)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Run This Notebook\n", "\n", "To access and run this tutorial within MAAP's Algorithm Development Environment (ADE), please refer to the \"Getting started with the MAAP\" section of our documentation.\n", "\n", "Disclaimer: it is highly recommended to run a tutorial within MAAP's ADE, which already includes packages specific to MAAP, such as maap-py. Running the tutorial outside of the MAAP ADE may lead to errors." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Additional Resources\n", "- [Cloud Native Geoguide](https://guide.cloudnativegeo.org/)\n", "- [Examples of Working with COGs](https://guide.cloudnativegeo.org/cloud-optimized-geotiffs/cogs-examples.html)\n", "- [Rio Tiler Colors](https://cogeotiff.github.io/rio-tiler/colormap/)\n", "- [Matplotlib Colors](https://matplotlib.org/stable/tutorials/colors/colormaps.html) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Importing and Installing Packages\n", "\n", "To be able to run this notebook you'll need the following requirements:\n", "- rasterio\n", "- rio-cogeo\n", "- requests\n", "- cogeo-mosaic\n", "- folium" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the packages below are not installed already, uncomment the following cell:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [] }, "outputs": [], "source": [ "# %pip install rasterio\n", "# %pip install rio-cogeo\n", "# %pip install requests\n", "# %pip install cogeo-mosaic\n", "# %pip install folium" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "import glob\n", "import json\n", "import os\n", "import matplotlib\n", "\n", "import requests\n", "from pprint import pprint\n", "from cogeo_mosaic.mosaic import MosaicJSON\n", "from cogeo_mosaic.backends import MosaicBackend\n", "from folium import Map, TileLayer, WmsTileLayer\n", "\n", "titiler_endpoint = \"https://titiler.maap-project.org\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1: Inspect Cloud-Optimized GeoTIFF(s)\n", "\n", "In this step, we ensure that our data is valid, accessible, and looks as expected. We've included a helper function to translate MAAP ADE local paths to their respective S3 urls. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Accessing Files" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [], "source": [ "project_dir = \"/projects/shared-buckets//\"\n", "\n", "# e.g.\n", "project_dir = \"/projects/shared-buckets/alexdevseed/landsat8/viz/\"" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['Landsat8_30542_comp_cog_2015-2020_dps.tif',\n", " 'Landsat8_30543_comp_cog_2015-2020_dps.tif',\n", " 'Landsat8_30822_comp_cog_2015-2020_dps.tif',\n", " 'Landsat8_30823_comp_cog_2015-2020_dps.tif']\n" ] } ], "source": [ "# Search for files to include, use recursive if nested folders (common in DPS output)\n", "files = glob.glob(os.path.join(project_dir, \"Landsat8*.tif\"), recursive=False)\n", "files = [os.path.basename(f) for f in files]\n", "pprint(files[:10])\n", "\n", "# Use the first product\n", "_tif = files[0]\n", "\n", "\n", "# Helper function\n", "def local_to_s3(url):\n", " \"\"\"A Function to convert local paths to s3 urls\"\"\"\n", " return url.replace(\"/projects/shared-buckets\", \"s3://maap-ops-workspace/shared\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/projects/shared-buckets/alexdevseed/landsat8/viz/Landsat8_30542_comp_cog_2015-2020_dps.tif is a valid cloud optimized GeoTIFF\n" ] } ], "source": [ "%%bash -s \"$project_dir\" \"$_tif\"\n", "rio cogeo validate $1/$2" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'band_descriptions': [['b1', 'Blue'],\n", " ['b2', 'Green'],\n", " ['b3', 'Red'],\n", " ['b4', 'NIR'],\n", " ['b5', 'SWIR'],\n", " ['b6', 'NDVI'],\n", " ['b7', 'SAVI'],\n", " ['b8', 'MSAVI'],\n", " ['b9', 'NDMI'],\n", " ['b10', 'EVI'],\n", " ['b11', 'NBR'],\n", " ['b12', 'NBR2'],\n", " ['b13', 'TCB'],\n", " ['b14', 'TCG'],\n", " ['b15', 'TCW'],\n", " ['b16', 'ValidMask'],\n", " ['b17', 'Xgeo'],\n", " ['b18', 'Ygeo']],\n", " 'band_metadata': [['b1', {}],\n", " ['b2', {}],\n", " ['b3', {}],\n", " ['b4', {}],\n", " ['b5', {}],\n", " ['b6', {}],\n", " ['b7', {}],\n", " ['b8', {}],\n", " ['b9', {}],\n", " ['b10', {}],\n", " ['b11', {}],\n", " ['b12', {}],\n", " ['b13', {}],\n", " ['b14', {}],\n", " ['b15', {}],\n", " ['b16', {}],\n", " ['b17', {}],\n", " ['b18', {}]],\n", " 'bounds': [-117.10749852280769,\n", " 50.78795362739066,\n", " -116.50936927974429,\n", " 51.16389512140189],\n", " 'colorinterp': ['gray',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined',\n", " 'undefined'],\n", " 'count': 18,\n", " 'driver': 'GTiff',\n", " 'dtype': 'float32',\n", " 'height': 1000,\n", " 'maxzoom': 12,\n", " 'minzoom': 9,\n", " 'nodata_type': 'None',\n", " 'overviews': [2, 4],\n", " 'width': 1000}\n" ] } ], "source": [ "# Getting COG information\n", "cog_info = requests.get(\n", " f\"{titiler_endpoint}/cog/info\",\n", " params={\n", " \"url\": f\"{local_to_s3(project_dir)}{_tif}\",\n", " },\n", ").json()\n", "\n", "bounds = cog_info[\"bounds\"]\n", "pprint(cog_info)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'count': 1000000.0,\n", " 'histogram': [[128480.0,\n", " 807237.0,\n", " 53374.0,\n", " 7274.0,\n", " 2056.0,\n", " 743.0,\n", " 370.0,\n", " 299.0,\n", " 129.0,\n", " 38.0],\n", " [0.0,\n", " 4814.5,\n", " 9629.0,\n", " 14443.5,\n", " 19258.0,\n", " 24072.5,\n", " 28887.0,\n", " 33701.5,\n", " 38516.0,\n", " 43330.5,\n", " 48145.0]],\n", " 'majority': 0.0,\n", " 'masked_pixels': 0.0,\n", " 'max': 48145.0,\n", " 'mean': 7467.137024,\n", " 'median': 7989.0,\n", " 'min': 0.0,\n", " 'minority': 11663.0,\n", " 'percentile_2': 0.0,\n", " 'percentile_98': 12191.0,\n", " 'std': 2803.86812414882,\n", " 'sum': 7467137024.0,\n", " 'unique': 20318.0,\n", " 'valid_percent': 100.0,\n", " 'valid_pixels': 1000000.0}\n" ] } ], "source": [ "# Getting band information\n", "cog_stats = requests.get(\n", " f\"{titiler_endpoint}/cog/statistics\",\n", " params={\n", " \"url\": f\"{local_to_s3(project_dir)}{_tif}\",\n", " },\n", ").json()\n", "pprint(cog_stats[\"b1\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Parameters for the TiTiler\n", "\n", "These parameters will be pased to `titiler_endpoint` for visualization.\n", "\n", "Note the values below: We're setting the `rescale` equal to the selected band's `min`,`max` values and selecting the `gist_earth_r` colormap. You should modify the `colormap_name` as makes sense for your dataset. This notebook includes a section on what colormaps are available and how to configure different types of colormaps and legends." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "band = \"b1\"\n", "bidx = 1\n", "rescale = f\"{cog_stats[band]['min']},{cog_stats[band]['max']}\"\n", "\n", "params = {\n", " \"tile_format\": \"png\",\n", " \"tile_scale\": \"1\",\n", " \"TileMatrixSetId\": \"WebMercatorQuad\",\n", " \"url\": f\"{local_to_s3(project_dir)}{_tif}\",\n", " \"bidx\": 1, # Select which band to use\n", " \"resampling\": \"nearest\",\n", " \"rescale\": rescale,\n", " \"return_mask\": \"true\",\n", " \"colormap_name\": \"gist_earth_r\",\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Scenario 1: Adding a Single COG\n", "\n", "#### Upload File\n", "\n", "Only use the following steps if you only have one COG to share to the dashboard. If you want to create a mosaic from multiple COGs, skip Scenario 1 and go to Scenario 2.\n", "\n", "If you haven't already, upload the file to S3 and make note of the location. In this tutorial, we're using a landsat8/visualization TIF already in S3 for the `url` parameter value." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Test COG with TiTiler and Folium" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "response = requests.get(f\"{titiler_endpoint}/cog/tilejson.json\", params=params).json()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m = Map(\n", " tiles=\"OpenStreetMap\",\n", " location=((bounds[1] + bounds[3]) / 2, (bounds[0] + bounds[2]) / 2),\n", " zoom_start=cog_info[\"minzoom\"],\n", ")\n", "\n", "tiles = TileLayer(tiles=response[\"tiles\"][0], opacity=1, attr=\"USGS\")\n", "\n", "tiles.add_to(m)\n", "m" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Scenario 2: Adding Data from Multiple COGs by Creating a Mosaic\n", "\n", "Many datasets are comprised of many tiles distributed spatially over the globe. In order to visualize them all together, we can use [mosaicJSON](https://github.com/developmentseed/mosaicjson-spec) to create a mosaic for the dynamic tiler API. The dynamic tiler API knows how to read this mosaicJSON and select which tiles to render based on the current zoom, x and y coordinates across spatially distinct COGs." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "tiles = [f\"{local_to_s3(project_dir)}{file}\" for file in files]" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "MosaicJSON(mosaicjson='0.0.3', name=None, description=None, version='1.0.0', attribution=None, minzoom=1, maxzoom=16, quadkey_zoom=1, bounds=(-117.19773367251135, 50.19386902261471, -116.26013039328576, 51.16389512140189), center=(-116.72893203289856, 50.67888207200831, 1), tiles={'0': ['s3://maap-ops-workspace/shared/alexdevseed/landsat8/viz/Landsat8_30542_comp_cog_2015-2020_dps.tif', 's3://maap-ops-workspace/shared/alexdevseed/landsat8/viz/Landsat8_30543_comp_cog_2015-2020_dps.tif', 's3://maap-ops-workspace/shared/alexdevseed/landsat8/viz/Landsat8_30822_comp_cog_2015-2020_dps.tif', 's3://maap-ops-workspace/shared/alexdevseed/landsat8/viz/Landsat8_30823_comp_cog_2015-2020_dps.tif']}, tilematrixset=None, asset_type=None, asset_prefix=None, data_type=None, colormap=None, layers=None)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mosaicdata = MosaicJSON.from_urls(tiles, minzoom=1, maxzoom=16)\n", "mosaicdata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Using MosaicJSON with TiTiler\n", "\n", "There are 2 options for using mosaicJSON with titiler:\n", "\n", "1. (Preferred) Post mosaicJSON to titiler `mosaics` endpoint and use the `mosaicjson/mosaics` endpoint for dynamic tiling.\n", "2. Upload mosaicJSON to S3 and pass the S3 url to the titiler `mosaicjson/tiles` endpoint." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Post MosaicJSON to TiTiler" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'id': '1cecf064-1f2c-4adf-b7b3-7121fdc8f97d',\n", " 'links': [{'href': 'https://titiler.maap-project.org/mosaics/1cecf064-1f2c-4adf-b7b3-7121fdc8f97d',\n", " 'rel': 'self',\n", " 'title': 'Self',\n", " 'type': 'application/json'},\n", " {'href': 'https://titiler.maap-project.org/mosaics/1cecf064-1f2c-4adf-b7b3-7121fdc8f97d/mosaicjson',\n", " 'rel': 'mosaicjson',\n", " 'title': 'MosaicJSON',\n", " 'type': 'application/json'},\n", " {'href': 'https://titiler.maap-project.org/mosaics/1cecf064-1f2c-4adf-b7b3-7121fdc8f97d/tilejson.json',\n", " 'rel': 'tilejson',\n", " 'title': 'TileJSON',\n", " 'type': 'application/json'},\n", " {'href': 'https://titiler.maap-project.org/mosaics/1cecf064-1f2c-4adf-b7b3-7121fdc8f97d/tiles/{z}/{x}/{y}',\n", " 'rel': 'tiles',\n", " 'title': 'Tiles',\n", " 'type': 'application/json'},\n", " {'href': 'https://titiler.maap-project.org/mosaics/1cecf064-1f2c-4adf-b7b3-7121fdc8f97d/WMTSCapabilities.xml',\n", " 'rel': 'wmts',\n", " 'title': 'WMTS',\n", " 'type': 'application/json'}]}\n" ] } ], "source": [ "mosaic_links = requests.post(\n", " url=f\"{titiler_endpoint}/mosaics\",\n", " headers={\n", " \"Content-Type\": \"application/vnd.titiler.mosaicjson+json\",\n", " },\n", " json=mosaicdata.model_dump(exclude_none=True),\n", ").json()\n", "\n", "pprint(mosaic_links)\n", "\n", "mosaic_id = mosaic_links[\"id\"]" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'href': 'https://titiler.maap-project.org/mosaics/1cecf064-1f2c-4adf-b7b3-7121fdc8f97d/tilejson.json',\n", " 'rel': 'tilejson',\n", " 'type': 'application/json',\n", " 'title': 'TileJSON'}]" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tilejson_endpoint = list(\n", " filter(lambda x: x.get(\"rel\") == \"tilejson\", dict(mosaic_links)[\"links\"])\n", ")\n", "tilejson_endpoint" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Test Mosaic with TiTiler and Folium" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "params = {\n", " \"tile_format\": \"png\",\n", " \"bidx\": bidx,\n", " \"resampling\": \"nearest\",\n", " \"rescale\": rescale,\n", " \"return_mask\": \"true\",\n", " \"colormap_name\": \"viridis\",\n", " \"pixel_selection\": \"first\",\n", "}\n", "\n", "r_te = requests.get(tilejson_endpoint[0][\"href\"], params=params).json()\n", "\n", "tiles = TileLayer(tiles=f\"{r_te['tiles'][0]}\", opacity=1, attr=\"USGS\")\n", "\n", "tiles.add_to(m)\n", "m" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 2: Define a Color Map\n", "\n", "By default, the image will be displayed in greyscale if no `colormap_name` parameter is passed to the titiler API. Guidance below is provided to help determine what a valid colormap_name might be and how to create a legend for the dashboard.\n", "\n", "### Dashboard ColorRamps & Legends\n", "\n", "When using the dashboard, there 2 components for implementing a color scheme for your map. There is the map render and there is the legend.\n", "\n", "> Titiler used for Cloud Optimized Geotiff (COG) rendering accepts any color scheme from the python matplotlib library, and custom color formulas.\n", "\n", "* [Rio Tiler Colors](https://cogeotiff.github.io/rio-tiler/colormap/)\n", "* [Matplotlib Colors](https://matplotlib.org/stable/tutorials/colors/colormaps.html) \n", "\n", "Available `colormap_name` values for titiler: `above, accent, accent_r, afmhot, afmhot_r, autumn, autumn_r, binary, binary_r, blues, blues_r, bone, bone_r, brbg, brbg_r, brg, brg_r, bugn, bugn_r, bupu, bupu_r, bwr, bwr_r, cfastie, cividis, cividis_r, cmrmap, cmrmap_r, cool, cool_r, coolwarm, coolwarm_r, copper, copper_r, cubehelix, cubehelix_r, dark2, dark2_r, flag, flag_r, gist_earth, gist_earth_r, gist_gray, gist_gray_r, gist_heat, gist_heat_r, gist_ncar, gist_ncar_r, gist_rainbow, gist_rainbow_r, gist_stern, gist_stern_r, gist_yarg, gist_yarg_r, gnbu, gnbu_r, gnuplot, gnuplot2, gnuplot2_r, gnuplot_r, gray, gray_r, greens, greens_r, greys, greys_r, hot, hot_r, hsv, hsv_r, inferno, inferno_r, jet, jet_r, magma, magma_r, nipy_spectral, nipy_spectral_r, ocean, ocean_r, oranges, oranges_r, orrd, orrd_r, paired, paired_r, pastel1, pastel1_r, pastel2, pastel2_r, pink, pink_r, piyg, piyg_r, plasma, plasma_r, prgn, prgn_r, prism, prism_r, pubu, pubu_r, pubugn, pubugn_r, puor, puor_r, purd, purd_r, purples, purples_r, rainbow, rainbow_r, rdbu, rdbu_r, rdgy, rdgy_r, rdpu, rdpu_r, rdylbu, rdylbu_r, rdylgn, rdylgn_r, reds, reds_r, rplumbo, schwarzwald, seismic, seismic_r, set1, set1_r, set2, set2_r, set3, set3_r, spectral, spectral_r, spring, spring_r, summer, summer_r, tab10, tab10_r, tab20, tab20_r, tab20b, tab20b_r, tab20c, tab20c_r, terrain, terrain_r, twilight, twilight_r, twilight_shifted, twilight_shifted_r, viridis, viridis_r, winter, winter_r, wistia, wistia_r, ylgn, ylgn_r, ylgnbu, ylgnbu_r, ylorbr, ylorbr_r, ylorrd, ylorrd_r`\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example 1: Class Based Known Colors\n", "\n", "In this example, the raster represents classes of forest with 11 possible values. There are specific colors selected to correspond to each class. We combine the list of colors and the list of classes and format them for the legend parameter the dashboard needs.\n", "\n", "https://github.com/MAAP-Project/dashboard-datasets-maap/blob/main/datasets/taiga-forest-classification.json" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[\n", " {\n", " \"color\": \"#5255A3\",\n", " \"label\": \"Sparse & Uniform\"\n", " },\n", " {\n", " \"color\": \"#1796A3\",\n", " \"label\": \"Sparse & Diffuse-gradual\"\n", " },\n", " {\n", " \"color\": \"#FDBF6F\",\n", " \"label\": \"Sparse & Diffuse-rapid\"\n", " },\n", " {\n", " \"color\": \"#FF7F00\",\n", " \"label\": \"Sparse & Abrupt \"\n", " },\n", " {\n", " \"color\": \"#FFFFBF\",\n", " \"label\": \"Open & Uniform \"\n", " },\n", " {\n", " \"color\": \"#D9EF8B\",\n", " \"label\": \"Open & Diffuse-gradual\"\n", " },\n", " {\n", " \"color\": \"#91CF60\",\n", " \"label\": \"Open & Diffuse-rapid\"\n", " },\n", " {\n", " \"color\": \"#1A9850\",\n", " \"label\": \"Open & Abrupt\"\n", " },\n", " {\n", " \"color\": \"#C4C4C4\",\n", " \"label\": \"Intermediate & Closed\"\n", " },\n", " {\n", " \"color\": \"#FF0000\",\n", " \"label\": \"Non-forest edge (dry)\"\n", " },\n", " {\n", " \"color\": \"#0000FF\",\n", " \"label\": \"Non-forest edge (wet)\"\n", " }\n", "]\n" ] } ], "source": [ "colors = [\n", " \"#5255A3\",\n", " \"#1796A3\",\n", " \"#FDBF6F\",\n", " \"#FF7F00\",\n", " \"#FFFFBF\",\n", " \"#D9EF8B\",\n", " \"#91CF60\",\n", " \"#1A9850\",\n", " \"#C4C4C4\",\n", " \"#FF0000\",\n", " \"#0000FF\",\n", "]\n", "labels = [\n", " \"Sparse & Uniform\",\n", " \"Sparse & Diffuse-gradual\",\n", " \"Sparse & Diffuse-rapid\",\n", " \"Sparse & Abrupt \",\n", " \"Open & Uniform \",\n", " \"Open & Diffuse-gradual\",\n", " \"Open & Diffuse-rapid\",\n", " \"Open & Abrupt\",\n", " \"Intermediate & Closed\",\n", " \"Non-forest edge (dry)\",\n", " \"Non-forest edge (wet)\",\n", "]\n", "\n", "legend = [dict(color=colors[i], label=labels[i]) for i in range(0, len(colors))]\n", "print(json.dumps(legend, indent=2))\n", "\n", "# Copy and Paste the output below to your dashboard config." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example 2: Discrete ColorRamp\n", "\n", "In this example, the range of values is known, but the color scale has many non-sequential colors. Starting with the premade color list, we create a continuous color ramp that uses the known colors as stops points. Arbitrarly 12 breaks looked decent in the dashboard legend so we split it into 12 discrete colors. Then combine the list of values and colors into the correct json syntax.\n", "\n", "https://github.com/MAAP-Project/dashboard-datasets-maap/blob/main/datasets/ATL08.json" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0,\"#636363\"],\n", "[2,\"#c47e5d\"],\n", "[4,\"#fda467\"],\n", "[6,\"#fed886\"],\n", "[8,\"#fff1a7\"],\n", "[10,\"#f8fcb6\"],\n", "[12,\"#e0f294\"],\n", "[14,\"#b8e077\"],\n", "[16,\"#86ca5f\"],\n", "[18,\"#3aa754\"],\n", "[20,\"#118145\"],\n", "[22,\"#005a32\"]]\n" ] } ], "source": [ "forest_ht = matplotlib.colors.LinearSegmentedColormap.from_list(\n", " \"forest_ht\",\n", " [\n", " \"#636363\",\n", " \"#FC8D59\",\n", " \"#FEE08B\",\n", " \"#FFFFBF\",\n", " \"#D9EF8B\",\n", " \"#91CF60\",\n", " \"#1A9850\",\n", " \"#005A32\",\n", " ],\n", " 12,\n", ")\n", "cols = [matplotlib.colors.to_hex(forest_ht(i)) for i in range(forest_ht.N)]\n", "\n", "cats = range(0, 25, (25 // len(cols)))\n", "legend = [[cats[i], cols[i]] for i in range(0, len(cols))]\n", "text = json.dumps(legend, separators=(\",\", \": \"))\n", "\n", "print(text.replace(\"],[\", \"],\\n[\"))\n", "\n", "# Copy and Paste the output below to your dashboard config." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example 3: Continuous ColorRamp\n", "\n", "In this example, we are using a built in ColorRamp from matplotlib. So we just need to extract enough colors to fill the legend adequately, and convert the colors to hex codes.\n", "\n", "https://github.com/MAAP-Project/dashboard-datasets-maap/blob/main/datasets/topo.json" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['#fdfbfb', '#e3c3b5', '#c9a87a', '#bab060', '#9db059', '#76a652', '#45994a', '#3a8c66', '#2e7c7f', '#1f567b', '#0f2577', '#000000']\n" ] } ], "source": [ "cmap_name = \"gist_earth_r\"\n", "cmap = matplotlib.cm.get_cmap(cmap_name, 12)\n", "cols = [matplotlib.colors.to_hex(cmap(i)) for i in range(cmap.N)]\n", "print(cols)\n", "\n", "# Copy and Paste the output below to your dashboard config." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3: Create and Submit Pull Request to Add Dashboard Dataset" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# This example is for a continuous color ramps\n", "dataset_type = \"raster\"\n", "dataset_id = \"paraguay-estimated-biomass\"\n", "dataset_name = \"Estimated Biomass in Paraguay\"\n", "\n", "stops = cols\n", "legend_type = \"gradient-adjustable\"\n", "info = \"Estimated biomass within 6km grids.\"\n", "\n", "sample_bidx = 1\n", "sample_band_min = 0\n", "sample_band_max = 4000\n", "parameters = (\n", " f\"colormap_name={cmap_name}&rescale={sample_band_min},{sample_band_max}&bidx={bidx}\"\n", ")" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# Single COG\n", "tiles_link = f\"{titiler_endpoint}/cog/tiles/{{z}}/{{x}}/{{y}}.png?url=s3://example-bucket/path/to/object/example.tif&{parameters}\"\n", "\n", "# Mosaic\n", "mosaic_link = (\n", " f\"{titiler_endpoint}/mosaic/{mosaic_id}/tiles/{{z}}/{{x}}/{{y}}?{parameters}\"\n", ")" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"id\": \"paraguay-estimated-biomass\",\n", " \"name\": \"Estimated Biomass in Paraguay\",\n", " \"type\": \"raster\",\n", " \"swatch\": {\n", " \"color\": \"#6976d7\",\n", " \"name\": \"Moody Blue\"\n", " },\n", " \"source\": {\n", " \"type\": \"raster\",\n", " \"tiles\": [\n", " \"https://titiler.maap-project.org/cog/tiles/{z}/{x}/{y}.png?url=s3://example-bucket/path/to/object/example.tif&colormap_name=gist_earth_r&rescale=0,4000&bidx=1\"\n", " ]\n", " },\n", " \"legend\": {\n", " \"type\": \"gradient-adjustable\",\n", " \"min\": 0,\n", " \"max\": 4000,\n", " \"stops\": [\n", " \"#fdfbfb\",\n", " \"#e3c3b5\",\n", " \"#c9a87a\",\n", " \"#bab060\",\n", " \"#9db059\",\n", " \"#76a652\",\n", " \"#45994a\",\n", " \"#3a8c66\",\n", " \"#2e7c7f\",\n", " \"#1f567b\",\n", " \"#0f2577\",\n", " \"#000000\"\n", " ]\n", " },\n", " \"info\": \"Estimated biomass within 6km grids.\"\n", "}\n" ] } ], "source": [ "dataset_dict = {\n", " \"id\": dataset_id,\n", " \"name\": dataset_name,\n", " \"type\": dataset_type,\n", " \"swatch\": {\"color\": \"#6976d7\", \"name\": \"Moody Blue\"},\n", " \"source\": {\"type\": dataset_type, \"tiles\": [tiles_link]},\n", " \"legend\": {\n", " \"type\": legend_type,\n", " \"min\": sample_band_min,\n", " \"max\": sample_band_max,\n", " \"stops\": stops,\n", " },\n", " \"info\": info,\n", "}\n", "print(json.dumps(dataset_dict, indent=4))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Clone the Datasets Repository\n", "\n", "```bash\n", "git clone git@github.com:MAAP-Project/biomass-dashboard-datasets.git\n", "cd biomass-dashboard-datasets\n", "git checkout -b feature/dataset-name\n", "# select and copy json above\n", "echo >> datasets/paraguay-estimated-biomass.json\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Add JSON to Product or Country Pilot\n", "\n", "In `country_pilots/paraguay/country_pilot.json`:\n", "```json\n", "{\n", " \"id\": \"paraguay\",\n", " \"label\": \"Paraguay\",\n", " //...\n", " \"datasets\": [\n", " {\n", " \"id\": \"paraguay-forest-mask\"\n", " },\n", " {\n", " \"id\": \"paraguay-tree-cover\"\n", " },\n", " {\n", " \"id\": \"paraguay-estimated-biomass\"\n", " }\n", " ]\n", "}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Add Content to summary.html\n", "\n", "There should be a `summary.html` file corresponding to the product or country pilot you are working on, for example: `country_pilots/paraguay/summary.html`. Add or modify content in that file as appropriate." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Add Dataset(s) to config.yml\n", "\n", "In `config.yml`:\n", "\n", "```yaml\n", "DATASETS:\n", "- paraguay-estimated-biomass.json\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Pull Request\n", "\n", "Once you have added the dataset json file and summary content, submit a PR to https://github.com/MAAP-Project/biomass-dashboard-datasets. A member of the data team will review the PR and when it is merged your content will appear in biomass.dit.maap-project.org." ] } ], "metadata": { "interpreter": { "hash": "06a5d0145d2c0de679875a5546b4cbbf4cf8aec2ac17ee1072f90657257481e7" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 4 }