{ "cells": [ { "cell_type": "markdown", "id": "58641795-4f3e-40af-9374-b882f2be6eb1", "metadata": {}, "source": [ "# Job Management with maap-py" ] }, { "cell_type": "markdown", "id": "55668263-b357-4eff-a44b-3a54e9068f97", "metadata": {}, "source": [ "## Set up maap.py\n", "\n", "1. Open a Jupyter Notebook then click the `MAAP` button from the notebook toolbar. This will paste the code snippet below into your notebook.\n", "\n", "![submit_job_4.png](../_static/faq_images/submit_job_4.png)\n", "\n", "```\n", "from maap.maap import MAAP\n", "maap = MAAP()\n", "```\n" ] }, { "cell_type": "markdown", "id": "cd3c4962", "metadata": {}, "source": [ "## Passing Credentials for Other Services into Jobs (Secrets Management)\n", "\n", "You may have an Algorithm that downloads or uploads data to other systems (e.g. Google Earth Engine, etc.). In order for your Algorithm to access those other systems during a DPS run, your login credentials or login token must be passed into the DPS Job. This is done via \"secrets\", which keeps your credentials encrypted during transmission.\n", "\n", "You may store and retrieve secrets using maap-py, which allows you to use those secrets during a Job that you run.\n", "\n", "Typically you will use a Jupyter notebook to **manage your secrets** using the methods below. An example notebook for Secrets Managment can be [viewed and downloaded here](./Secrets_Manager.ipynb).\n", "\n", "Inside your Algorithm code, you will use the maap-py secrets.get_secret(\"SECRET_NAME\") method shown below to have the Algorithm retrieve and **use the value of your secret**. This may be a hard-coded secret name in your Algorithm, or the secret name may be passed in at runtime—this would be important if different scientists have named their secret differently.\n", "\n", "### Add a Secret\n", "\n", "```\n", "maap.secrets.add_secret(\"\", \"\")\n", "\n", "ex.\n", "response = maap.secrets.add_secret(\"MY_TOKEN\", \"98aj48j(774hh9*H\")\n", "print(response)\n", ">>> {'secret_name': 'MY_TOKEN'}\n", "```\n", "\n", "### Get a List of Your Secrets\n", "\n", "```\n", "maap.secrets.get_secrets()\n", "\n", "ex.\n", "response = maap.secrets.get_secrets()\n", "print(response)\n", ">>> [{'secret_name': 'MY_TOKEN'}]\n", "```\n", "\n", "### Get the Value of a Specific Secret\n", "\n", "```\n", "maap.secrets.get_secret(\"\")\n", "\n", "ex.\n", "response = maap.secrets.get_secret(\"MY_TOKEN\")\n", "print(response)\n", ">>> 98aj48j(774hh9*H\n", "```\n", "\n", "### Delete a Secret\n", "\n", "```\n", "maap.secrets.delete_secret(\"\")\n", "\n", "ex.\n", "response = maap.secrets.delete_secret(\"MY_TOKEN\")\n", "print(response)\n", ">>> {'code': 200, 'message': 'Successfully deleted secret MY_TOKEN'}\n", "```" ] }, { "cell_type": "markdown", "id": "f6c3a51a", "metadata": {}, "source": [ "## Submit a Job\n", "\n", "Use the `submitJob` method and provide your algorithm inputs. The example below will run the `run-dps-test_ubuntu` algorithm. \n", "\n", ".. note::\n", "Experimental feature: The output data will be put into a folder named for your `algo_id` and the `identifier`. You can use the same `idenfitier` on several jobs (e.g. in a batch) to group related output data in one place. In the [View & Submit Jobs GUI](jobsui.ipynb#Submit-a-Job) this is the Job Tag field in the Submit form.\n", "\n", "```\n", "maap.submitJob(identifier=\"test-job\", \n", " algo_id=\"run-dps-test_ubuntu\",\n", " version=\"delay10\",\n", " queue=\"maap-dps-worker-8gb\",\n", " input_file=\"https://raw.githubusercontent.com/MAAP-Project/dps-unit-test/main/README.md\")\n", "```\n", "\n", "4. Run the notebook to submit the job. The cell output for a job that was submitted successfully will look similar to this:\n", "\n", "```\n", "{'status': 'success',\n", " 'http_status_code': 200,\n", " 'job_id': '86fbac52-24b0-4963-8b67-59d0fc09946a'}\n", "```" ] }, { "cell_type": "markdown", "id": "c81e7aed", "metadata": {}, "source": [ "## Monitor a Job\n", "\n", "1. Use the `getJobStatus` method and provide the job ID that was created upon job submission.\n", "\n", "```\n", "r = maap.getJobStatus(\"86fbac52-24b0-4963-8b67-59d0fc09946a\")\n", "r.text\n", "```\n", "\n", "2. Run the notebook to get the job status. The output should resemble the xml snippet below. In this example, the job status is `Succeeded`.\n", "\n", "```\n", "'86fbac52-24b0-4963-8b67-59d0fc09946aSucceeded'\n", "```\n", "\n", "### Job Status\n", "Job status may be different between the HySDS Figaro job-monitoring dashboard and the Jobs UI. Below is a mapping of status terms:\n", "```\n", "MAAP <- HySDS\n", "Accepted <- job-queued\n", "Running <- job-started\n", "Success <- job-completed\n", "Failed <- job-offline or job-failed\n", "job-revoked <- job-revoked (when a queued or running job is stopped before completion)\n", "\n", "HySDS state not valid/used in MAAP: job-deduped\n", "```\n" ] }, { "cell_type": "markdown", "id": "dd5ea78e", "metadata": {}, "source": [ "## Get Job Results\n", "\n", "1. Use the `getJobResult` method and provide the job ID that was created upon job submission.\n", "\n", "```\n", "r = maap.getJobResult(\"86fbac52-24b0-4963-8b67-59d0fc09946a\")\n", "r.text\n", "```\n", "\n", "2. Run the notebook to get the job result. The output should resemble the xml snippet below.\n", "\n", "```\n", "86fbac52-24b0-4963-8b67-59d0fc09946ahttp://maap-dit-workspace.s3-website-us-west-2.amazonaws.com/anonymous/dps_output/run-dps-test_ubuntu/delay10/2023/05/10/15/39/51/905070s3://s3-us-west-2.amazonaws.com:80/maap-dit-workspace/anonymous/dps_output/run-dps-test_ubuntu/delay10/2023/05/10/15/39/51/905070https://s3.console.aws.amazon.com/s3/buckets/maap-dit-workspace/anonymous/dps_output/run-dps-test_ubuntu/delay10/2023/05/10/15/39/51/905070/?region=us-east-1&tab=overview\n", "```" ] }, { "cell_type": "markdown", "id": "c92f2f06", "metadata": {}, "source": [ "## Cancel Job\n", "\n", "1. Use the `cancelJob` method and provide the job ID that was created upon job submission.\n", "\n", "```\n", "r = maap.cancelJob(\"fac2904d-b45d-4cf3-971f-45586a6bc78a\")\n", "print(r)\n", "```\n", "\n", "2. Run the notebook to get the job result. The output should resemble the xml snippet below.\n", "\n", "```\n", "fac2904d-b45d-4cf3-971f-45586a6bc78aAccepted\n", "```\n", "\n", ".. note::\n", "This method submits a request to cancel a job so it may take several minutes before the cancellation takes effect. Users may check the job's status to confim cancellation. Cancelled jobs will return a status of `Dismissed` if they were cancelled after starting. Queued jobs that are cancelled will be purged from the system." ] }, { "cell_type": "markdown", "id": "7100c385", "metadata": {}, "source": [ "## List Jobs\n", "\n", "1. Use the `listJobs` method to retrieve a list of jobs that the current user (determined by the `MAAP_PGT` token environment variable) has submitted.\n", "\n", "```\n", "r = maap.listJobs()\n", "print(r.text)\n", "```\n", "\n", "Here is a sample truncated output:\n", "\n", "```\n", "{\"code\": 200, \"jobs\": [{\"b308b3cd-8848-4154-b682-aaf3d39734ee\": {\"resource\": \"job\", \"payload_hash\": \"0f217c35a6bdacf46b164106af9ab473\", \"job_id\": \"job-dps_tutorial_mlucas_test__main-20240626T172521.763123Z\", \"status\": \"job-completed\", \"context\": {\"tag\": \"test-new\", \"_prov\": {\"wasGeneratedBy\": \"task_id:a4761c87-dc68-4511-9a3e-e628446f1e44\", \"wasDerivedFrom\": [\"url:https://photojournal.jpl.nasa.gov/tiff/PIA00127.tif\"]}, \"container_image_name\": \"container-dps_tutorial:main\", \"outsize\": 30, \"container_specification\": {\"version\": \"main\", \"digest\": \"sha256:e539186c9e76fb923da7b3bfb6a3c700403e5302a71c3e8ba8635b7f216ab7e3\", \"id\": \"container-dps_tutorial:main\", \"url\": \"s3://s3-us-west-2.amazonaws.com/maap-dit-registry/container-dps_tutorial:main.tar.gz\"}, \"container_mappings\": {\"$HOME/.aws\": \"/home/ops/.aws\", \"$HOME/.netrc\": \"/home/ops/.netrc\", \"/tmp\": [\"/tmp\", \"rw\"]}, \"_command\": \"/app/dps_wrapper.sh '/app/dps_tutorial/gdal_wrapper/run_gdal.sh' 'output.tif' '30'\", \"localize_urls\": [{\"url\": \"https://photojournal.jpl.nasa.gov/tiff/PIA00127.tif\"}], \"output_file\": \"output.tif\", \"_disk_usage\": \"1GB\", \"container_image_url\": \"s3://s3-us-west-2.amazonaws.com/maap-dit-registry/container-dps_tutorial:main.tar.gz\", \"container_image_id\": \"sha256:e539186c9e76fb923da7b3bfb6a3c700403e5302a71c3e8ba8635b7f216ab7e3\", \"job_specification\": {\"post\": [\"hysds.triage.triage\"], \"resource\": \"jobspec\", \"container\": \"container-dps_tutorial:main\", \"soft_time_limit\": 86400, \"time_limit\": 86400, \"recommended-queues\": [\"maap-dps-worker-8gb\"], \"job-version\": \"main\", \"command\": \"/app/dps_wrapper.sh '/app/dps_tutorial/gdal_wrapper/run_gdal.sh'\", \"params\": [{\"destination\": \"localize\", \"name\": \"input_file\", \"value\": \"https://photojournal.jpl.nasa.gov/tiff/PIA00127.tif\"}, {\"destination\": \"positional\", \"name\": \"output_file\", \"value\": \"output.tif\"}...\n", "```\n", "\n", "Users may provide optional query parameters to filter the list on desired fields. The following example will return a list of jobs submitted by the current user that all ran the `job-dps_tutorial_mlucas:main` algorithm:\n", "\n", "```\n", "r = maap.listJobs(algo_id=\"job-dps_tutorial_mlucas\", version=\"main\")\n", "```\n", "\n", "Users may also request just the list of jobs id's and their corresponding tags rather than full job details. This is controlled using the `get_job_details` option. By default `get_job_details` is set to `True`, but if set to `False` will return a compact list, as shown in this example:\n", "\n", "```\n", "r = maap.listJobs(algo_id=\"job-dps_tutorial_mlucas\", version=\"main\", get_job_details=False)\n", "print(r.text)\n", "```\n", "\n", "```\n", "{\"code\": 200, \"jobs\": [{\"id\": \"e6b6b27d-d409-4f6b-8aa2-505c2dc150fd\", \"tags\": [\"test\"]}, {\"id\": \"fd504b51-096a-42fe-8f1d-a157352dfad0\", \"tags\": [\"test\"]}, {\"id\": \"69dc095a-4895-43a2-951a-763f3a33b0bf\", \"tags\": [\"test\"]}, {\"id\": \"c7983efc-689f-4008-b87a-bc3450213152\", \"tags\": [\"test\"]}, {\"id\": \"41288295-6744-424e-ae26-c1dd9e26c288\", \"tags\": [\"test\", \"test\"]}], \"message\": \"success\"}\n", "```\n", "\n", "These are the available query parameters:\n", "\n", "| Parameter | Description | Type | Optional/Required? | Acceptable Values\n", "|-----------------|-------------|-----------------------------|-------------------|----------|\n", "| algo_id | algorithm id e.g. `job-dps_tutorial_mlucas` | string | optional | Valid string |\n", "| end_time | job end time | string e.g. `2024-01-01T00:00:00.000Z` | optional | Valid time string |\n", "| get_job_details | Flag that determines whether to return full job details or just a list of job id's with their corresponding tags. Default is `True`.| bool | optional | `False` or `True`|\n", "| offset | pagination offset | number | optional | Integer |\n", "| page_size | pagination page size | number | optional | Integer |\n", "| priority | job priority | number | optional | 0-9 |\n", "| queue | resource queue | string | optional | Valid string |\n", "| start_time | job start time | string e.g. `2024-01-01T00:00:00.000Z` | optional | Valid string |\n", "| status | job status | string | optional | `job-queued`, `job-started`, `job-completed`, `job-failed`, `job-revoked`, `job-offline` |\n", "| tag | user-specified tag | string | optional | Valid string |\n", "| version | algorithm version i.e. github branch | string | optional | Valid string |\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" }, "vscode": { "interpreter": { "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e" } } }, "nbformat": 4, "nbformat_minor": 5 }