Resource Queues

Resource Queues on the MAAP Data Processing Service are machines on the cloud that will be processing DPS Jobs. These are EC2 machines on AWS and are managed by auto scaling groups.

When a user submits a job, a message is sent to the AWS Auto Scaling Group which provisions the appropriate machine instance for processing, which takes about two minutes. The name of the queue provides an indication of what resources the machine will have.

For example:

  • maap-dps-worker-8gb will spin a machine that has at max 8gb of RAM, like t3.large

  • maap-dps-worker-16vcpu-32gb will spin a machine that has 16 cores and 32gb RAM, like c5.4xlarge.

Queue Access and Organizations

Only job queues assigned to your organization(s) will be allowed for use when registering an Algorithm or submitting Jobs. When submitting Jobs via the Jobs UI, only your org-assigned queues are visible in the queue selection menu. Any publicly-accessible queues, such as the sandbox queue, are accessible to all users of the MAAP platform.

Public Queues

Public queues are queues that are available to all MAAP users irrespective of their association with any organization (i.e guest accounts). Users who belong to an organization and have access to private queues are recommended to use the private queues. Currently available public queues on the MAAP platform are:

Queue

Memory

Time limit

Guest access

AWS Instance Type

maap-dps-sandbox

8gb

10 min

yes

t3.large or t3a.large

maap-dps-worker-8gb

8gb

unlimited

no

t3.large or t3a.large

maap-dps-worker-16gb

16gb

unlimited

no

t3.xlarge or t3a.xlarge

maap-dps-worker-32gb

32gb

unlimited

no

r5.xlarge or r5a.xlarge

maap-dps-sandbox is a queue introduced in v4.1.0 to allow users to quickly test their algorithm in DPS. This has a max job runtime of 10mins and max RAM of 8gb.

Note: The platform team will start throttling users ability to submit a batch of jobs on public queues to about 10 jobs per hour.

Private Organization Queues

Users and user-groups doing science on the MAAP are should reach out to the platform team to enable private organizational queues. Features of private queues are:

  • 24 hr time limit on individual jobs (can be increased upon request).

  • No job throttling.

  • Custom resources can be requested on the queue, like higher CPU, GPU, etc.

  • Dedicated queue, i.e not shared with other users on the MAAP reducing the amount of time spent waiting in the queued state.