Job Management with maap-py

Set up maap.py

  1. Open a Jupyter Notebook then click the MAAP button from the notebook toolbar. This will paste the code snippet below into your notebook.

submit_job_4.png

  1. Provide the MAAP host. For normal operations, this would be api.maap-project.org.

from maap.maap import MAAP
maap = MAAP(maap_host='api.maap-project.org')

Submit a Job

Use the submitJob method and provide your algorithm inputs. The example below will run the run-dps-test_ubuntu algorithm.

Note

Experimental feature: The output data will be put into a folder named for your algo_id and the identifier. You can use the same idenfitier on several jobs (e.g. in a batch) to group related output data in one place. In the View & Submit Jobs GUI this is the Job Tag field in the Submit form.

maap.submitJob(identifier="test-job",
               algo_id="run-dps-test_ubuntu",
               version="delay10",
               username="anonymous",
               queue="maap-dps-worker-8gb",
               input_file="https://raw.githubusercontent.com/MAAP-Project/dps-unit-test/main/README.md")

  1. Run the notebook to submit the job. The cell output for a job that was submitted successfully will look similar to this:

{'status': 'success',
 'http_status_code': 200,
 'job_id': '86fbac52-24b0-4963-8b67-59d0fc09946a'}

Monitor a Job

  1. Use the getJobStatus method and provide the job ID that was created upon job submission.

r = maap.getJobStatus("86fbac52-24b0-4963-8b67-59d0fc09946a")
r.text

  1. Run the notebook to get the job status. The output should resemble the xml snippet below. In this example, the job status is Succeeded.

'<wps:StatusInfo xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:schemaLocation="http://schemas.opengis.net/wps/2.0/wps.xsd" xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><wps:JobID>86fbac52-24b0-4963-8b67-59d0fc09946a</wps:JobID><wps:Status>Succeeded</wps:Status></wps:StatusInfo>'

Job Status

Job status may be different between the HySDS Figaro job-monitoring dashboard and the Jobs UI. Below is a mapping of status terms:

MAAP <- HySDS
Accepted <- job-queued
Running <- job-started
Success <- job-completed
Failed <- job-offline or job-failed
job-revoked <- job-revoked (when a queued or running job is stopped before completion)

HySDS state not valid/used in MAAP: job-deduped

Get Job Results

  1. Use the getJobResult method and provide the job ID that was created upon job submission.

r = maap.getJobResult("86fbac52-24b0-4963-8b67-59d0fc09946a")
r.text

  1. Run the notebook to get the job result. The output should resemble the xml snippet below.

<wps:Result xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:schemaLocation="http://schemas.opengis.net/wps/2.0/wps.xsd" xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><wps:JobID>86fbac52-24b0-4963-8b67-59d0fc09946a</wps:JobID><wps:Output id="output-2023-05-10T15:39:51.905070"><wps:Data>http://maap-dit-workspace.s3-website-us-west-2.amazonaws.com/anonymous/dps_output/run-dps-test_ubuntu/delay10/2023/05/10/15/39/51/905070</wps:Data><wps:Data>s3://s3-us-west-2.amazonaws.com:80/maap-dit-workspace/anonymous/dps_output/run-dps-test_ubuntu/delay10/2023/05/10/15/39/51/905070</wps:Data><wps:Data>https://s3.console.aws.amazon.com/s3/buckets/maap-dit-workspace/anonymous/dps_output/run-dps-test_ubuntu/delay10/2023/05/10/15/39/51/905070/?region=us-east-1&amp;tab=overview</wps:Data></wps:Output></wps:Result>