Writing and Managing Code with MAAP
Writing and editing code in the MAAP is done in a Jupyter workspace.
Code is version-controlled using git, which may be GitHub or MAAP GitLab. Jupyter also provides a GUI widget to help with code push/pull as a sidebar tool. Git is intended to help with collaborative code development and version-control.
To assist connections to the MAAP system from a Jupyter notebook, a helper library called
maap.py provides Python-native calls to the underlying RESTful MAAP API. Often a separate Jupyter notebook is used to run and monitor jobs with API calls. When working with Jupyter notebooks, a manual save must be done to create a checkpoint. Checkpoints are an emergency backup of the notebook and are different than the automatic saving of the notebook. Jupyter notebook checkpoints bloat git
unnecessarily, so you can add the following to your
.gitignore file to prevent this:
For more information on
.gitignore files, see here.
Helpful Templates while developing Algorithms in MAAP
This algorithm repository example is a good starting point for a new algorithm, as it contains the various accessory files that facilitate running the algorithm at scale
Which templates will help you? Let the development or documentation team know!
For example: conda.yml with some default packages, run_script.sh
Working with code repositories like GitHub and GitLab
Connecting to Github using a Personal Access Token
Clone a Repository with GitHub
Here is an example repository you can use for this getting started guide: https://github.com/MAAP-Project/dps-unit-test
Copy the Github clone link from https://github.com/MAAP-Project/dps-unit-test
Open the built-in Jupyter Github UI to the left of the file browser. Choose “Clone a Repository” and paste in the .git link you copied from the Github repository. You can also access this menu through the Git tab at the top of the Jupyter window.
You should see a new folder created with the repo you cloned. If you browse to that folder and open up the Jupyter Github UI again, it will show you some info about that repo.
If you want to make changes to the code and have your own copy of it to register, clone the code into a public repository in Github or in MAAP Gitlab.
To open the IPython Notebook, go to a section directory and double-click on appropriate “.ipynb” file. For more information about the using Git in Jupyterlab, see https://github.com/jupyterlab/jupyterlab-git .
The MAAP GitLab Code repository
After creating your MAAP account, you can create a code repository by navigating to the MAAP GitLab account at https://repo.maap-project.org. This GitLab account is connected to your ADE workspaces automatically when signing into the ADE.
You can then follow the same steps above to clone a repository from the MAAP GitLab.
Customizing your workspace environment
Your Jupyter workspace has a set of pre-installed libraries, depending on which Stack you selected. If you need libraries that are not pre-installed, we suggest using an environment manager;
conda is pre-installed to help with this.
Using maap.py to access MAAP functionality from Python notebooks
The MAAP platform offers a variety of functionality to run and monitor large-scale processing jobs. Access to the functionality is gained via the underlying RESTful MAAP API. In a Python notebook, you will typically use this API via a helper library called
maap.py, which will make using MAAP platform features easy, using Python syntax. For example, registering algorithms, running batches of jobs, monitoring jobs, or accessing data.