sw.models_mixins package

Submodules

sw.models_mixins.container module

sw.models_mixins.container

Container

class sw.models_mixins.container.Container(image)

Bases: object

Container as Singularity

Attributes : - image : image file name

Container is always of ‘singularity’ type, whereas Software available types are : ‘docker’, ‘singularity’.

Important : When recording a Software, as a ‘docker’ container, into ws database, a singularity image file has to be built from the docker container and installed as expected by ws (cf ‘get_singularity_image_file_name_from’).

Note : The image file full path depends on the used resource (request /run /muse/run…)

classmethod create(sw, rsrc)

Creates a Container for sw software on rsrc resource

The container is or comes from sw container, depending on sw container type (docker, singularity).

execute(wk, rsrc, user_cmd, cmd_option)

Executes container command under workspace run_path

The command is defined from user_cmd and cmd_option.

Files produced into workspace : stdout file, stderr file, cmd file.

Looks at the stderr file to detect error case.

Note : preparation has to be called before (folder and files copy…)

classmethod get_error_msg(stderr_file_path)

classmethod get_singularity_image(sw, rsrc)

Returns the name of the singularity image file about sw software

The singularity image file is or comes from sw container, depending on sw container type : - If sw container type is ‘singularity’, then the singularity image

file is the sw container one.

• If sw container type is ‘docker’, then the singularity image file is the one built from the sw container docker image.

classmethod get_txt_stdout(stdout_file_path)

classmethod make_cmd(run_path, image_path, user_cmd, cmd_option)

Makes and returns command for singularity image_path

Note : cmd_option is given as singularity syntax

make_local_cmd(run_path, rsrc, user_cmd, cmd_option)

Makes and returns local command for singularity container

make_remote_cmd(run_path, rsrc, user_cmd, cmd_option)

Makes and returns remote command for singularity container

classmethod make_ws_script_file(run_path, user_cmd)

Builds script file to be run by container (in order to run user_cmd)

Returns the ws_cmd command to call the built script file

Note : mount $PWD:/WS to run ws_cmd

sw.models_mixins.resource module

sw.models_mixins.resource

Calcul resources

Methods in remote cases suppose some mounts between the ws and remote machines

See install/*/install.txt, conf/config.py

class sw.models_mixins.resource.ClusterResource

Bases: sw.models_mixins.resource.RemoteResource

Cluster remote calcul resource

classmethod get_job_id_file_name()

get_job_state(job_id)

Returns cr and state related to job_id

Outputs :

• statejob status.

  ValuesRUNNING, RESIZING, SUSPENDED, COMPLETED,

  CANCELLED, FAILED, TIMEOUT, PREEMPTED, NODE_FAIL.

  RUNNING : job has received its allocation. RESIZING : ??? SUSPENDED : job has been cancelled after resources allocation. COMPLETED : all processes have finished execution on all nodes. CANCELLED : job has been interrupted. FAILED : job has finished with returned code different from 0. TIMEOUT : the delay for the job has been finished. PREEMPTED : job has been preempted (its priority increased by admin). NODE_FAIL : job has finished because of an error on 1 or more node(s).

• cr : Values IN_PROGRESS, ENDED_OK, ENDED_NOK.

state and cr come from “sacct” remote command.

Note about sacct command : If more information is available on the job state than will fit into the current field width (for example, the uid that CANCELLED a job), then the state will be followed by a “+” (it is possible to increase the size of the displayed state).

try_get_job_id(local_run_path)

Tries once to read job_id value, returns it if found and else None

job_id value is read into JOB_ID file

class sw.models_mixins.resource.MuseResource

Bases: sw.models_mixins.resource.ClusterResource

cluster_launch(wk, sw_cmd, cluster_config=None)

Remote cluster launches sw_cmd command

Launches sw_cmd command (sbatch) under workspace run_path

Files produced into workspace : stdout file, stderr file, cmd file.

Looks at output files to detect error case.

Note : preparation has to be called before (folder and files copy…)

Inputs :

• wk, where run_path : run folder (local) where resulting files

• sw_cmd (remote adapted)

• cluster_config

Outputs :

• retcode : returned code

• sbatch_msg_stdout : stdout message

• sbatch_msg_stderr : stderr message

classmethod create_cluster_script_file(script_file_path, script_cmd, cluster_config)

Creates the script file allowing to run script_cmd on muse cluster

A default configuration, overwritten by the input cluster_config one.

Inputs : - cluster_config[‘sbatch_list’] list - cluster_config[‘module_list’] list

Some SLURM parameters : (see https://meso-lr.umontpellier.fr/documentation-utilisateurs)

• job-name : job name

• account : project name

• nodes : number of nodes

• ntasksnumber of tasks

  ntasks = ntasks-per-node * nodes

• ntasks-per-node : number of tasks per node

• ntasks-per-corenumber of tasks per core

  generally ntasks-per-core = 1

• partition : group of machines where job running

• memmemory reserved per node. MegaBytes.

  Facultatif : if not defined, the whole node memory allocated

• timetime limit for job running

  (after, the job is automatically stopped) HH:MM:SS format (HH : hours, MM : minutes, SS : seconds.

• mail-user : email address to send notifications

classmethod get_identity()

classmethod get_local_requests_home()

classmethod get_local_simg_home()

classmethod get_remote_host()

classmethod get_remote_requests_home()

classmethod get_remote_simg_home()

class sw.models_mixins.resource.RemoteResource

Bases: sw.models_mixins.resource.Resource

Remote calcul resource

classmethod get_remote_path(local_path, local_prefix, remote_prefix)

Returns path where local prefix has been replaced by remote one

get_remote_requests_path(path)

Returns requests path where Local root replaced by Remote root

get_remote_simg_path(path)

Returns simg path where Local root replaced by Remote root

class sw.models_mixins.resource.Resource

Bases: object

Calcul resource

Available calcul resource identities : - ‘wsm’ ws machine (local resource) - ‘muse’ Muse cluster

Used to specialize some methods according to the resource

classmethod create(identity='wsm')

Creates a resource as defined by identity

class sw.models_mixins.resource.WsmResource

Bases: sw.models_mixins.resource.Resource

classmethod get_identity()

classmethod get_local_requests_home()

classmethod get_local_simg_home()

sw.models_mixins.workspace module

sw.models_mixins.workspace

Workspace (local paths depending on calcul resource)

class sw.models_mixins.workspace.RunWorkspace(run_path)

Bases: object

RunWorkspace has only run_path with its existing folder

classmethod define_report_file_path(path)

classmethod define_stderr_file_path(path)

classmethod define_stdout_file_path(path)

get_cmd_file_path()

get_report_file_path()

get_run_path()

classmethod get_run_workspace(run_path)

get_stderr_file_path()

get_stdout_file_path()

make_report_file(report)

make_run_zip(report=None)

class sw.models_mixins.workspace.Workspace

Bases: sw.models_mixins.workspace.RunWorkspace, object

classmethod create(rsrc)

Creates and initializes a workspace about rsrc resource

home_path is defined and created

home_path subdirectories are declared here. They will later be used or not (depending on use cases). They will be defined and created if/when used.

define_data_path()

define_run_path()

get_data_path()

get_home_path()

get_remote_stderr_file_path(remote_run_path)

get_remote_stdout_file_path(remote_run_path)

make_run_path()

Creates run folder under home_path and returns its value

Module contents

sw.models_mixins

This package is part of ws web services

copyright

Copyright (C) 2020-2024 INRAE http://www.inrae.fr

license

GPLv3, see LICENSE file for more details

authors

see AUTHORS file

For models of sw application