Installation procedure in development case

Installation of ws in development case

*******************************************************************************

                         Install ws on ws machine

    Case of development on a PC under ubuntu with django development server

         Versions : ubuntu 18.04 LTS, ubuntu 20.04 LTS

*******************************************************************************

"Rename" ws path as /opt/ws, if ws is installed somewhere else
(ln -s 'ws path' /opt/ws).

Some required basic tools to be installed if not already installed
==================================================================

- python 3                         : apt-get install python3
- pip3                             : apt-get install python3-pip
- python3 virtual environment tool : apt-get install python3-venv

- misc : apt-get install curl git man vim zip

- sqlite3 :
  apt-get install sqlite3 


Containers tools
================

- Singularity : see /opt/ws/ws/install/install_singularity.txt

- Docker

Miscellaneous libraries
=======================

libcurl installation
--------------------
Those libraries are needed by pycurl.

  sudo apt-get install libcurl4-openssl-dev
  sudo apt-get install libssl-dev

locale
------
cf /bin/bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)

  (to see :
   Into /etc/locale.gen, decomment line : fr_FR.UTF-8 UTF-8
   sudo locale-gen)

Get ws source code
==================

  - get ws source code :
    git clone https://forgemia.inra.fr/nathalie.rousse/ws.git

  - copy (or link) ws as /opt/ws

Folders out from /opt/ws to create if not already existing
==========================================================
(For more description, see /opt/ws/ws/install/install.txt)
For folders owners and rights, see lower

  mkdir /var/www/html/softwares

  mkdir /opt/ws_rsrc
      
  - for ws machine :
    mkdir /opt/ws_rsrc/wsm
    mkdir /opt/ws_rsrc/wsm/simg
    mkdir /opt/ws_rsrc/wsm/requests

  - for muse cluster : will be done at install_muse.txt

Remote cluster
==============

Muse cluster : see install_muse.txt
------------

wsenv virtual environment
=========================

- wsenv venv creation :
  python3 -m venv /opt/ws/fab/install/wsenv

- wsenv venv activation :
  source /opt/ws/fab/install/wsenv/bin/activate

( just to control the activation of wsenv venv : which python )

- to solve "invalid command 'bdist_wheel'" problem :
  pip3  install --upgrade pip wheel setuptools

- installation (into wsenv) from requirement.txt :
  pip3 install -r /opt/ws/ws/install/requirement.txt

ws static files generation (creation, updating)
===============================================

Static files (css, js, images)  :

  - Activate wsenv virtual environment (if not yet done) :
    source /opt/ws/fab/install/wsenv/bin/activate

  - generate :
    cd /opt/ws/ws/projects/ws
    python manage.py collectstatic

ws database creation (and updating, managing)
=============================================

Create database
---------------

Database /opt/ws/db/ws.sqlite3 :

  - Activate wsenv virtual environment (if not yet done) :
    source /opt/ws/fab/install/wsenv/bin/activate

  cd /opt/ws/ws/projects/ws

  - create/update database :

   (see INSTALLED_APPS into settings.py for applications list) :
    python manage.py makemigrations sw
    python manage.py makemigrations conf
    python manage.py makemigrations load
    python manage.py makemigrations acs

    python manage.py migrate

Feed database
-------------

- you can add into database as many softwares as you want
  (then you can build a setup of it, in order to reinstall it if required :
  see /opt/ws_deliv/DELIV/db/README.txt, /opt/ws/ws/install/admin_db.txt).

- to create admin user into database :
  python manage.py createsuperuser
                                   ... => admin user (wsadm)

- to feed database from ws_deliv db setup (.json) :
  see /opt/ws_deliv/DELIV/DB.txt, see
  "To restore the database from setup" into /opt/ws/ws/install/admin_db.txt

- to feed database from ws db backup (.json) : see
  "To restore the database from backup" into /opt/ws/ws/install/admin_db.txt

Admin help notes
----------------

  See /opt/ws/ws/install/admin_db.txt (save, restore...)

ws main documentation
=====================

- Main documentation generation :

  - Activate wsenv virtual environment (if not yet done) :
    source /opt/ws/fab/install/wsenv/bin/activate

  - Prerequisite (some files to generate and import) :

    cd /opt/ws/doc/misc
    generate POST_run.pdf from POST_run.odg
    generate POST_muse_run.pdf from POST_muse_run.odg
    mv POST_run.pdf /opt/ws/doc/source/api/files/POST_run.pdf
    mv POST_muse_run.pdf /opt/ws/doc/source/api/files/POST_muse_run.pdf

    (for more see /opt/ws/doc/misc/README.md)

  - Generate :
    cd /opt/ws/doc
    make clean
    make cleanref
    make ref
    make html

    cd /opt/ws/doc ; make clean ; make cleanref ; make ref ; make html

  The resulting documentation is produced under /opt/ws/fab/doc

- Access to main documentation from /var/www/html :

  - Content :
    cd /var/www/html
    sudo ln -s /opt/ws/fab/doc/ ws

  - Entry point :
    sudo cp /opt/ws/doc/server/index.html /var/www/html/index.html

Run
===

  - Activate wsenv virtual environment (if not yet done) :
    source /opt/ws/fab/install/wsenv/bin/activate

  cd /opt/ws/ws/projects/ws
  (+ See static files updating and database updating)
  (+ Remote cluster mounts --> use the following 'grep' command)
  python manage.py runserver

  grep sshfs /opt/ws/ws/install/dev/install_muse.txt | grep reconnect
  cd /opt/ws/ws/projects/ws ; python manage.py runserver

Notes
=====

- ws online at :
  - ws web site     : http://127.0.0.1
  - ws web services : http://127.0.0.1:8000/api/...

Softwares installation
======================
Install the softwares you want to be able to be used by ws web services, as
required by ws.

  - See /opt/ws/softwares/install/install.txt : into database,
    containers (install_softwares.txt), and (maybe)
    documentation (install_softwares_documentation.txt)

*******************************************************************************

install_singularity.txt :

*******************************************************************************
  
                          Install Singularity 3.5.3

*******************************************************************************

Available to install the Singularity 3.5.3 version :
  - in development case (PC under ubuntu) or
  - in production case (ws VM under debian).

Follow https://sylabs.io/guides/3.5/user-guide/quick_start.html

Note
====

  The Singularity 3.5.3 version is the one used by ws (by default) on the
  Muse machine.

  To know available Singularity versions on the Muse machine :

    @muse :

      module available
      => singularity/2.2.1 singularity/2.4 singularity/2.6
         singularity/3.3 singularity/3.5

  To identify a Singularity version on the Muse machine :

    @muse : 

      module load singularity/3.5 ; singularity --version
      => singularity version 3.5.3

Install
=======

- Install system dependencies :

  sudo apt-get update
  sudo apt-get install build-essential libssl-dev uuid-dev libgpgme11-dev
  sudo apt-get install squashfs-tools libseccomp-dev
  sudo apt-get install wget pkg-config git cryptsetup

- Install Go (version archive suitable to Debian 10,
  chosen on the Go Downloads page https://golang.org/dl) :

  wget https://golang.org/dl/go1.15.2.linux-amd64.tar.gz
  sudo tar -C /usr/local -xzvf go1.15.2.linux-amd64.tar.gz
  rm go1.15.2.linux-amd64.tar.gz
  echo 'export PATH=/usr/local/go/bin:$PATH' >> ~/.bashrc && source ~/.bashrc

- Download Singularity from its release, compile and install it :

  export VERSION=3.5.3
  wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz
  tar -xzf singularity-${VERSION}.tar.gz
  cd singularity
  ./mconfig
  make -C builddir
  sudo make -C builddir install

- Control :
  singularity --version
  => singularity version 3.5.3

*******************************************************************************

install_muse.txt :

*******************************************************************************
  
                       Install communication between

              ws machine (PC) and remote muse machine (cluster)

*******************************************************************************

Case
====

- muse_user='roussen' user on @muse=@muse-login.hpc-lr.univ-montp2.fr (muse
                                                                     machine).
- pc_user='nrousse' user on @pc=@baudrimont PC (ws machine).
- We want to be able to remote use muse_user@muse cluster from pc_user@pc
  (roussen@muse-login.hpc-lr.univ-montp2.fr from nrousse@baudrimont).

Groups
======

  @muse (roussen@muse-login.hpc-lr.univ-montp2.fr) :
  - Identify muse_user group :
    id roussen
    => Response : uid=1654(roussen) gid=1112(record) groupes=1112(record)
       groupname=record, gidvalue=1112

  @pc (nrousse@baudrimont) :

  - Verify that groupname GID (record, gidvalue=1112) does not exist on @pc
    yet (to see all the UID/GID : cat /etc/passwd ; cat /etc/group; )

  - create groupname group (with same GID value as at @muse),
    add pc_user into :
    sudo addgroup --gid 1112 record
    sudo usermod -aG record nrousse

  - Verification :

    id nrousse
    => Response OK :
       uid=1321(nrousse) gid=500(miat) groupes=500(miat), ... ,1112(record)

Access authorization (by ssh key) from pc_user@pc to muse_user@muse
===================================================================

  @pc as pc_user (nrousse@baudrimont) :

    - Generate a ssh key for pc_user if not existing yet :
      ssh-keygen -o -t rsa -b 4096

    - Get pc_user@pc public key value :
      cat /home/nrousse/.ssh/id_rsa.pub

  @muse as muse_user (roussen@muse-login.hpc-lr.univ-montp2.fr) :

    - Register the ssh public key of pc_user@pc (nrousse@baudrimont) :

      - If /home/roussen/.ssh does not exist, generate ssh key to create it :
        ssh-keygen -o -t rsa -b 4096
      - Add into /home/roussen/.ssh/authorized_keys file the pc_user@pc public
        key (cf nrousse@baudrimont:/home/nrousse/.ssh/id_rsa.pub)

  @pc as pc_user (nrousse@baudrimont) :

    - Initialize the access (1st connexion) from pc_user@pc to muse_user@muse :

      ssh roussen@muse-login.hpc-lr.univ-montp2.fr
      ... The authenticity of host ... continue connecting (yes/no)? yes

Mount (requests, simg) between pc_user@pc and muse_user@muse
============================================================

Configuration description
=========================

  On muse_user@muse :  ws/simg                 scratch/ws/requests
  On pc_user@pc :     /opt/ws_rsrc/muse/simg  /opt/ws_rsrc/muse/requests

  Mounts to be done between :
    - muse_user@muse:ws/simg     and pc_user@pc:/opt/ws_rsrc/muse/simg
    - muse_user@muse:scratch/ws/requests
                                 and pc_user@pc:/opt/ws_rsrc/muse/requests

Install
=======

  @muse as muse_user (roussen@muse-login.hpc-lr.univ-montp2.fr)
  (if not already existing) :

    - Create folders if not already existing :

      mkdir /home/roussen/ws
      mkdir /home/roussen/ws/simg

      mkdir /home/roussen/scratch/ws
      mkdir /home/roussen/scratch/ws/requests

      (Note : /home/roussen/scratch is link to /lustre/roussen)

      (See install_softwares.txt to fill simg folder with *.simg files)

  @pc as pc_user (nrousse@baudrimont) :

    - Create folders if not already existing :

      sudo mkdir /opt/ws_rsrc/muse
      sudo mkdir /opt/ws_rsrc/muse/simg
      sudo mkdir /opt/ws_rsrc/muse/requests

      sudo chown -R nrousse:record /opt/ws_rsrc

    - Other preparations :

      sudo apt-get install sshfs
      Into /etc/fuse.conf file, uncomment : user_allow_other

    - Mount :
      sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,cache=no,allow_other roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests /opt/ws_rsrc/muse/requests
      sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,cache=no,allow_other roussen@muse-login.hpc-lr.univ-montp2.fr:ws/simg /opt/ws_rsrc/muse/simg

Mount monitoring :

  @pc as pc_user (nrousse@baudrimont) :

    - To verify that mount is active :
      mount

      => Mount not active if response missing lines :
      roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests on /opt/ws_rsrc/muse/requests type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1321,group_id=500,allow_other)
      roussen@muse-login.hpc-lr.univ-montp2.fr:ws/simg on /opt/ws_rsrc/muse/simg type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1321,group_id=500,allow_other)

    - Mount again if mount is not active anymore

    - memo :
      with -o nonempty : sshfs -o cache=no,allow_other -o nonempty roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests /opt/ws_rsrc/muse/requests
      unmount : fusermount -u  /opt/ws_rsrc/muse/requests

*******************************************************************************

See : Softwares installation