Installation procedure in production case

Installation of ws in production case

*******************************************************************************

                         Install ws on ws machine

      Case of production on a VM under debian with apache http server

            Versions : debian 10 (buster), apache2.4

*******************************************************************************

"Rename" ws path as /opt/ws, if ws is installed somewhere else
(ln -s 'ws path' /opt/ws).

Creation of the Virtual Machine for ws
======================================

The VM 147.100.179.250 (user:nrousse) has been created in debian 10 (Buster)
with apache2 + libapache2-mod-wsgi-py3

Misc
====

- Connexion :

  ssh nrousse@147.100.179.250

- Upgrade :

  sudo apt-get update
  sudo apt-get upgrade
  sudo apt-get dist-upgrade
  sudo apt-get autoremove

- Ident : (at first install)

  > hostname -I              147.100.179.250 
  > cat /etc/issue           Debian GNU/Linux 10
  > cat /etc/debian_version  10.5
  > uname -a                 Linux wsrecord 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux

Some required basic tools to be installed if not already installed
==================================================================

- sudo installation
  apt-get install sudo

- python 3 installation
  apt-get install python3

- python3 virtual environment tool
  apt-get install python3-venv

- pip3 installation
  apt-get install python3-pip

- misc
  apt-get install curl git man vim zip

- sqlite3 :
  apt-get install sqlite3 

- Apache HTTP Server

  (to verify apache service running : sudo systemctl status apache2)

  - Apache2.4 installation
    apt-get install apache2

  - mod_wsgi installation
    apt-get install libapache2-mod-wsgi-py3

Containers tools
================

- Singularity : see /opt/ws/ws/install/install_singularity.txt

- Docker not necessary (see /opt/ws/softwares/install/install_softwares.txt)

Miscellaneous libraries
=======================

libcurl installation
--------------------
Those libraries are needed by pycurl.

  sudo apt-get install libcurl4-openssl-dev
  sudo apt-get install libssl-dev

locale
------
cf /bin/bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8)

  (to see :
   Into /etc/locale.gen, decomment line : fr_FR.UTF-8 UTF-8
   sudo locale-gen)

Get ws source code
==================

  - get ws source code :
    git clone https://forgemia.inra.fr/nathalie.rousse/ws.git

  - copy (or link) ws as /opt/ws

Folders out from /opt/ws to create if not already existing 
==========================================================
(For more description, see ../install.txt)
For folders owners and rights, see lower

  mkdir /var/www/html/softwares

  mkdir /opt/ws_rsrc

  - for ws machine :
    mkdir /opt/ws_rsrc/wsm
    mkdir /opt/ws_rsrc/wsm/simg  
    mkdir /opt/ws_rsrc/wsm/requests

   (See /opt/ws/softwares/install/install_softwares.txt to fill simg folder
    with *.simg files)

  - for muse cluster : will be done at install_muse.txt

Remote cluster
==============

Muse cluster : see install_muse.txt
------------

wsenv virtual environment
=========================

- wsenv venv creation :
  python3 -m venv /opt/ws/fab/install/wsenv

- wsenv venv activation :
  source /opt/ws/fab/install/wsenv/bin/activate

( just to control the activation of wsenv venv : which python )

- to solve "invalid command 'bdist_wheel'" problem :
  pip3  install --upgrade pip wheel setuptools

- installation (into wsenv) from requirement.txt :
  pip3 install -r /opt/ws/ws/install/requirement.txt

ws software configuration
=========================

- /opt/ws/ws/apps/conf/config.py file modifications :

      IN_PRODUCTION = True

- /opt/ws/ws/projects/ws/ws/settings.py file modifications :

      DEBUG = False

    - add '147.100.179.250' value (@vm) into ALLOWED_HOSTS list (if not yet) :

      ALLOWED_HOSTS = [ ..., 
                        '147.100.179.250',      # VM ws (production)
                        ..., ]

- URLs : If some IP addresses values directly appeared in source code
  (.py, .rst) as 127.0.0.1:8000 (development case), they should be modified
  as : 147.100.179.250 (@vm) here for production case.

    If needed, put the appropriate IP address :

    - ws documentation (.rst) :
      sed -i 's/127.0.0.1:8000/147.100.179.250/g' /opt/ws/doc/source/api/*.rst

    - ws code (*.py...) : 

      cd /opt/ws/softwares/doc
      sed -i 's/127.0.0.1:8000/147.100.179.250/g' default_doc/build_software_default_doc.py

      cd /opt/ws/softwares/install
      sed -i 's/127.0.0.1:8000/147.100.179.250/g' build_script_building_singularity_from_docker.py



ws static files generation (creation, updating)
===============================================

Static files (css, js, images)  :

  - Activate wsenv virtual environment (if not yet done) :
    source /opt/ws/fab/install/wsenv/bin/activate

  - generate :
    cd /opt/ws/ws/projects/ws
    python manage.py collectstatic

ws database creation (and updating, managing)
=============================================

Create database
---------------

Database /opt/ws/db/ws.sqlite3 :

  - Activate wsenv virtual environment (if not yet done) :
    source /opt/ws/fab/install/wsenv/bin/activate

  cd /opt/ws/ws/projects/ws

  - create/update database :

   (see INSTALLED_APPS into settings.py for applications list) :
    python manage.py makemigrations sw
    python manage.py makemigrations conf
    python manage.py makemigrations load
    python manage.py makemigrations acs

    python manage.py migrate

Feed database
-------------

- you can add into database as many softwares as you want
  (then you can build a setup of it, in order to reinstall it if required :
  see /opt/ws_deliv/DELIV/db/README.txt, /opt/ws/ws/install/admin_db.txt).

- to create admin user into database :
  python manage.py createsuperuser
                                   ... => admin user (wsadm)

- to feed database from ws_deliv db setup (.json) :
  see /opt/ws_deliv/DELIV/DB.txt, see
  "To restore the database from setup" into /opt/ws/ws/install/admin_db.txt

- to feed database from ws db backup (.json) : see
  "To restore the database from backup" into /opt/ws/ws/install/admin_db.txt

Admin help notes
----------------

  See /opt/ws/ws/install/admin_db.txt (save, restore...)

ws main documentation
=====================

- Main documentation generation :

  - Activate wsenv virtual environment (if not yet done) :
    source /opt/ws/fab/install/wsenv/bin/activate

  - Prerequisite (some files to generate and import) :

    cd /opt/ws/doc/misc
    generate POST_run.pdf from POST_run.odg 
    generate POST_muse_run.pdf from POST_muse_run.odg
    mv POST_run.pdf /opt/ws/doc/source/api/files/POST_run.pdf
    mv POST_muse_run.pdf /opt/ws/doc/source/api/files/POST_muse_run.pdf

    (for more see /opt/ws/doc/misc/README.md)

  - Generate :
    cd /opt/ws/doc
    make clean
    make cleanref
    make ref
    make html

    cd /opt/ws/doc ; make clean ; make cleanref ; make ref ; make html

  The resulting documentation is produced under /opt/ws/fab/doc

- Access to main documentation from /var/www/html :

  - Content :
    cd /var/www/html
    sudo ln -s /opt/ws/fab/doc/ ws

  - Entry point :
    sudo cp /opt/ws/doc/server/index.html /var/www/html/index.html

Apache configuration
====================

/etc/apache2/ports.conf file modifications
------------------------------------------
Modify/verify the apache2 ports.conf file so that the Apache2 server listens to
the port 80 dedicated to the ws.conf site
(see /opt/ws/ws/projects/ws/ws/ws.conf file)
and to the online documentation :

      Listen 80
      ...
      <IfModule ssl_module>
          Listen 443
      </IfModule>

/etc/apache2/apache2.conf file modifications
--------------------------------------------
Modify into the apache2.conf file the Timeout value (number of seconds
before receives and sends time out, initial value 300)

      Timeout 600

ws.conf file modifications
--------------------------
Give/control into the ws.conf file the appropriate values (ServerName,
ws paths), then install it in the appropriate location :

  - /opt/ws/ws/projects/ws/ws/ws.conf file modifications :

        ServerName : if needed, replace ws.toulouse.inra.fr by the
                     appropriate IP address

        erecord paths : /opt/ws/...

  - install ws.conf file :
    sudo cp /opt/ws/ws/projects/ws/ws/ws.conf /etc/apache2/sites-available/ws.conf

/etc/apache2/sites-available/000-default.conf verification
----------------------------------------------------------
Verify that the apache2 000-default.conf file corresponds with the ws main
documentation installation.

  - /etc/apache2/sites-available/000-default.conf verification :

      <VirtualHost *:80>

          DocumentRoot /var/www/html

Run
===

  - Apache2 user as owner :

    - ws code :
      sudo chown -R www-data:www-data /opt/ws

    - /var/www (ws online documentation and softwares online documentation) :
      sudo chown -R www-data:www-data /var/www

    - machines resources :
      sudo chown -R www-data:record /opt/ws_rsrc    __todo__ a remplacer par :
      sudo chown -R www-data:record /opt/ws_rsrc/wsm

  - Remote cluster mounts --> use the following 'grep' command :
    grep sshfs /opt/ws/ws/install/prod/install_muse.txt | grep reconnect

  - Activate ws.conf site :
    sudo a2ensite ws.conf

  - Activate 000-default.conf site : is supposed to be already done.

  - Apache run :
    sudo systemctl reload apache2

Notes
=====

- ws online at :
  - ws web site     : http://147.100.179.250
  - ws web services : http://147.100.179.250/api/...

- Regularly Verify that mount is active :
  see "Mount monitoring" into install_muse.txt

- apache log files : /var/log/apache2/error.log
                     /var/log/apache2/access.log

Softwares installation
======================
Install the softwares you want to be able to be used by ws web services, as
required by ws.

  - See /opt/ws/softwares/install/install.txt :
    containers (install_softwares.txt), and (maybe)
    documentation (install_softwares_documentation.txt)

  Softwares documentation
  -----------------------
  URLs : If some IP addresses values directly appeared in software
  documentation as 127.0.0.1:8000 (development case), they should be modified
  as : 147.100.179.250 (@vm) here for production case.

    If needed, put the appropriate IP address : __todo__ to be tested

    - software documentation :
      cd /var/www/html/softwares
      sed -i 's/127.0.0.1:8000/147.100.179.250/g' *
      sed -i 's/127.0.0.1:8000/147.100.179.250/g' */*
      sed -i 's/127.0.0.1:8000/147.100.179.250/g' */*/*
      sed -i 's/127.0.0.1:8000/147.100.179.250/g' */*/*/*

  Be careful : respect Apache2 rights (see "Run" above).

*******************************************************************************

install_singularity.txt :

*******************************************************************************
  
                          Install Singularity 3.5.3

*******************************************************************************

Available to install the Singularity 3.5.3 version :
  - in development case (PC under ubuntu) or
  - in production case (ws VM under debian).

Follow https://sylabs.io/guides/3.5/user-guide/quick_start.html

Note
====

  The Singularity 3.5.3 version is the one used by ws (by default) on the
  Muse machine.

  To know available Singularity versions on the Muse machine :

    @muse :

      module available
      => singularity/2.2.1 singularity/2.4 singularity/2.6
         singularity/3.3 singularity/3.5

  To identify a Singularity version on the Muse machine :

    @muse : 

      module load singularity/3.5 ; singularity --version
      => singularity version 3.5.3

Install
=======

- Install system dependencies :

  sudo apt-get update
  sudo apt-get install build-essential libssl-dev uuid-dev libgpgme11-dev
  sudo apt-get install squashfs-tools libseccomp-dev
  sudo apt-get install wget pkg-config git cryptsetup

- Install Go (version archive suitable to Debian 10,
  chosen on the Go Downloads page https://golang.org/dl) :

  wget https://golang.org/dl/go1.15.2.linux-amd64.tar.gz
  sudo tar -C /usr/local -xzvf go1.15.2.linux-amd64.tar.gz
  rm go1.15.2.linux-amd64.tar.gz
  echo 'export PATH=/usr/local/go/bin:$PATH' >> ~/.bashrc && source ~/.bashrc

- Download Singularity from its release, compile and install it :

  export VERSION=3.5.3
  wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz
  tar -xzf singularity-${VERSION}.tar.gz
  cd singularity
  ./mconfig
  make -C builddir
  sudo make -C builddir install

- Control :
  singularity --version
  => singularity version 3.5.3

*******************************************************************************

install_muse.txt :

*******************************************************************************

                       Install communication between

              ws machine (VM) and remote muse machine (cluster)

*******************************************************************************

Case
====

- muse_user='roussen' user on @muse=@muse-login.hpc-lr.univ-montp2.fr (muse
                                                                     machine).
On ws machine (@vm=@147.100.179.250), vm_user and apache_user :

- vm_user='nrousse' user on @vm=@147.100.179.250 VM (ws machine).
- We want to be able to remote use muse_user@muse cluster from vm_user@vm
  (roussen@muse-login.hpc-lr.univ-montp2.fr from nrousse@147.100.179.250).

- apache_user='www-data' user on @vm=@147.100.179.250 VM (ws machine).
- We want to be able to remote use muse_user@muse cluster from apache_user@vm
  (roussen@muse-login.hpc-lr.univ-montp2.fr from www-data@147.100.179.250).

Groups
======

  @muse (roussen@muse-login.hpc-lr.univ-montp2.fr) :
  - Identify muse_user group :
    id roussen
    => Response : uid=1654(roussen) gid=1112(record) groupes=1112(record)
       groupname=record, gidvalue=1112

  @vm (nrousse@147.100.179.250) :

  - Verify that groupname GID (record, gidvalue=1112) does not exist on @vm
    yet (to see all the UID/GID : cat /etc/passwd ; cat /etc/group; )

  - create groupname group (with same GID value as at @muse),
    add vm_user and apache_user into :
    sudo addgroup --gid 1112 record
    sudo usermod -aG record nrousse
    sudo usermod -aG record www-data

  - Verification :

    id www-data
    => Response OK :
       uid=33(www-data) gid=33(www-data) groupes=33(www-data),1112(record)

    id nrousse
    => Response OK :
       uid=1321(nrousse) gid=500(miat) groupes=500(miat), ... ,1112(record)

Access authorization (by ssh key) from apache_user@vm to muse_user@muse
=======================================================================

  @vm as vm_user (nrousse@147.100.179.250) :

    - Generate a ssh key for apache_user (www-data) under /opt/www-data.ssh
      if not existing yet :

      - Authorization (prerequisite) :
        sudo visudo -f /etc/sudoers.d/ws
        Then add into this /etc/sudoers.d/ws file the 2 lines :
          www-data ALL=(nrousse) NOPASSWD: /usr/bin/ssh-keygen
          www-data ALL=(nrousse) NOPASSWD: /usr/bin/ssh

      - www-data key folder (out of /var/www) :
        sudo mkdir /opt/www-data.ssh
        sudo chown -R www-data:www-data /opt/www-data.ssh
        sudo mkdir /var/www/.ssh
        sudo vim /var/www/.ssh/config
        Then add into this /var/www/.ssh/config file the 2 lines :
          Host *
          IdentityFile /opt/www-data.ssh/id_rsa
        sudo chown -R www-data:www-data /var/www/.ssh

      - ssh key generation :
        sudo -u www-data ssh-keygen -o -t rsa -b 4096
        ... Enter file in which to save the key : /opt/www-data.ssh/id_rsa
        ... no passphrase

  @muse as muse_user (roussen@muse-login.hpc-lr.univ-montp2.fr) :

    - Register the ssh public key of apache_user@vm
      (www-data@147.100.179.250) :

      - If /home/roussen/.ssh does not exist, generate ssh key to create it :
        ssh-keygen -o -t rsa -b 4096
      - Add into the /home/roussen/.ssh/authorized_keys file the apache_user@vm
        public key (cf www-data@147.100.179.250:/opt/www-data.ssh/id_rsa.pub)

  @vm as vm_user (nrousse@147.100.179.250) :

    - Initialize the access (1st connexion) from apache_user@vm to
      muse_user@muse :

      sudo -u www-data ssh roussen@muse-login.hpc-lr.univ-montp2.fr
      ... The authenticity of host ... continue connecting (yes/no)? yes

Access authorization (by ssh key) from vm_user@vm to muse_user@muse
===================================================================

  @vm as vm_user (nrousse@147.100.179.250) :

    - Generate a ssh key for vm_user if not existing yet :
      ssh-keygen -o -t rsa -b 4096

    - Get vm_user@vm public key value :
      cat /home/nrousse/.ssh/id_rsa.pub

  @muse as muse_user (roussen@muse-login.hpc-lr.univ-montp2.fr) :

    - Register the ssh public key of vm_user@vm (nrousse@147.100.179.250) :

      - If /home/roussen/.ssh does not exist, generate ssh key to create it :
        ssh-keygen -o -t rsa -b 4096
      - Add into /home/roussen/.ssh/authorized_keys file the vm_user@vm public
        key (cf nrousse@147.100.179.250:/home/nrousse/.ssh/id_rsa.pub)

  @vm as vm_user (nrousse@147.100.179.250) :

    - Initialize the access (1st connexion) from vm_user@vm to muse_user@muse :

      ssh roussen@muse-login.hpc-lr.univ-montp2.fr
      ... The authenticity of host ... continue connecting (yes/no)? yes

Mount (requests, simg) between vm_user@vm and muse_user@muse
============================================================

Configuration description
=========================

  On muse_user@muse :  ws/simg                 scratch/ws/requests
  On vm_user@vm :     /opt/ws_rsrc/muse/simg  /opt/ws_rsrc/muse/requests

  Mounts to be done between :
    - muse_user@muse:ws/simg     and vm_user@vm:/opt/ws_rsrc/muse/simg
    - muse_user@muse:scratch/ws/requests
                                 and vm_user@vm:/opt/ws_rsrc/muse/requests

Install
=======

  @muse as muse_user (roussen@muse-login.hpc-lr.univ-montp2.fr) 

    - Create folders if not already existing :

      mkdir /home/roussen/ws
      mkdir /home/roussen/ws/simg

      mkdir /home/roussen/scratch/ws
      mkdir /home/roussen/scratch/ws/requests

      (Note : /home/roussen/scratch is link to /lustre/roussen)

      (See /opt/ws/softwares/install/install_softwares.txt to fill simg folder
       with *.simg files)

  @vm as vm_user (nrousse@147.100.179.250) :

    - Create folders if not already existing :

      sudo mkdir /opt/ws_rsrc/muse
      sudo mkdir /opt/ws_rsrc/muse/simg
      sudo mkdir /opt/ws_rsrc/muse/requests

      sudo chmod 775 /opt/ws_rsrc/muse
      sudo chmod 775 /opt/ws_rsrc/muse/simg
      sudo chmod 775 /opt/ws_rsrc/muse/requests

    - required before mount below :
      sudo chown -R www-data:record /opt/ws_rsrc

    - Other preparations :

      sudo apt-get install sshfs
      Into /etc/fuse.conf file, uncomment : user_allow_other

    - Mount :
      sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,cache=no,allow_other roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests /opt/ws_rsrc/muse/requests
      sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,cache=no,allow_other roussen@muse-login.hpc-lr.univ-montp2.fr:ws/simg /opt/ws_rsrc/muse/simg

Mount monitoring :

  @vm as vm_user (nrousse@147.100.179.250) :

    - To verify that mount is active :
      mount

      => Mount not active if response missing lines :
      roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests on /opt/ws_rsrc/muse/requests type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1321,group_id=500,allow_other)
      roussen@muse-login.hpc-lr.univ-montp2.fr:ws/simg on /opt/ws_rsrc/muse/simg type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1321,group_id=500,allow_other)

    - Mount again if mount is not active anymore

    - memo :
      with -o nonempty : sshfs -o cache=no,allow_other -o nonempty roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests /opt/ws_rsrc/muse/requests
      unmount : fusermount -u  /opt/ws_rsrc/muse/requests

*******************************************************************************

See : Softwares installation