Installation procedure in production case¶
Installation of ws in production case¶
******************************************************************************* Install ws on ws machine Case of production on a VM under debian with apache http server Versions : debian 10 (buster), apache2.4 ******************************************************************************* "Rename" ws path as /opt/ws, if ws is installed somewhere else (ln -s 'ws path' /opt/ws). Creation of the Virtual Machine for ws ====================================== The VM 147.100.179.250 (user:nrousse) has been created in debian 10 (Buster) with apache2 + libapache2-mod-wsgi-py3 Misc ==== - Connexion : ssh nrousse@147.100.179.250 - Upgrade : sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade sudo apt-get autoremove - Ident : (at first install) > hostname -I 147.100.179.250 > cat /etc/issue Debian GNU/Linux 10 > cat /etc/debian_version 10.5 > uname -a Linux wsrecord 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux Some required basic tools to be installed if not already installed ================================================================== - sudo installation apt-get install sudo - python 3 installation apt-get install python3 - python3 virtual environment tool apt-get install python3-venv - pip3 installation apt-get install python3-pip - misc apt-get install curl git man vim zip - sqlite3 : apt-get install sqlite3 - Apache HTTP Server (to verify apache service running : sudo systemctl status apache2) - Apache2.4 installation apt-get install apache2 - mod_wsgi installation apt-get install libapache2-mod-wsgi-py3 Containers tools ================ - Singularity : see /opt/ws/ws/install/install_singularity.txt - Docker not necessary (see /opt/ws/softwares/install/install_softwares.txt) Miscellaneous libraries ======================= libcurl installation -------------------- Those libraries are needed by pycurl. sudo apt-get install libcurl4-openssl-dev sudo apt-get install libssl-dev locale ------ cf /bin/bash: warning: setlocale: LC_ALL: cannot change locale (fr_FR.UTF-8) (to see : Into /etc/locale.gen, decomment line : fr_FR.UTF-8 UTF-8 sudo locale-gen) Get ws source code ================== - get ws source code : git clone https://forgemia.inra.fr/nathalie.rousse/ws.git - copy (or link) ws as /opt/ws Folders out from /opt/ws to create if not already existing ========================================================== (For more description, see ../install.txt) For folders owners and rights, see lower mkdir /var/www/html/softwares mkdir /opt/ws_rsrc - for ws machine : mkdir /opt/ws_rsrc/wsm mkdir /opt/ws_rsrc/wsm/simg mkdir /opt/ws_rsrc/wsm/requests (See /opt/ws/softwares/install/install_softwares.txt to fill simg folder with *.simg files) - for muse cluster : will be done at install_muse.txt Remote cluster ============== Muse cluster : see install_muse.txt ------------ wsenv virtual environment ========================= - wsenv venv creation : python3 -m venv /opt/ws/fab/install/wsenv - wsenv venv activation : source /opt/ws/fab/install/wsenv/bin/activate ( just to control the activation of wsenv venv : which python ) - to solve "invalid command 'bdist_wheel'" problem : pip3 install --upgrade pip wheel setuptools - installation (into wsenv) from requirement.txt : pip3 install -r /opt/ws/ws/install/requirement.txt ws software configuration ========================= - /opt/ws/ws/apps/conf/config.py file modifications : IN_PRODUCTION = True - /opt/ws/ws/projects/ws/ws/settings.py file modifications : DEBUG = False - add '147.100.179.250' value (@vm) into ALLOWED_HOSTS list (if not yet) : ALLOWED_HOSTS = [ ..., '147.100.179.250', # VM ws (production) ..., ] - URLs : If some IP addresses values directly appeared in source code (.py, .rst) as 127.0.0.1:8000 (development case), they should be modified as : 147.100.179.250 (@vm) here for production case. If needed, put the appropriate IP address : - ws documentation (.rst) : sed -i 's/127.0.0.1:8000/147.100.179.250/g' /opt/ws/doc/source/api/*.rst - ws code (*.py...) : cd /opt/ws/softwares/doc sed -i 's/127.0.0.1:8000/147.100.179.250/g' default_doc/build_software_default_doc.py cd /opt/ws/softwares/install sed -i 's/127.0.0.1:8000/147.100.179.250/g' build_script_building_singularity_from_docker.py ws static files generation (creation, updating) =============================================== Static files (css, js, images) : - Activate wsenv virtual environment (if not yet done) : source /opt/ws/fab/install/wsenv/bin/activate - generate : cd /opt/ws/ws/projects/ws python manage.py collectstatic ws database creation (and updating, managing) ============================================= Create database --------------- Database /opt/ws/db/ws.sqlite3 : - Activate wsenv virtual environment (if not yet done) : source /opt/ws/fab/install/wsenv/bin/activate cd /opt/ws/ws/projects/ws - create/update database : (see INSTALLED_APPS into settings.py for applications list) : python manage.py makemigrations sw python manage.py makemigrations conf python manage.py makemigrations load python manage.py makemigrations acs python manage.py migrate Feed database ------------- - you can add into database as many softwares as you want (then you can build a setup of it, in order to reinstall it if required : see /opt/ws_deliv/DELIV/db/README.txt, /opt/ws/ws/install/admin_db.txt). - to create admin user into database : python manage.py createsuperuser ... => admin user (wsadm) - to feed database from ws_deliv db setup (.json) : see /opt/ws_deliv/DELIV/DB.txt, see "To restore the database from setup" into /opt/ws/ws/install/admin_db.txt - to feed database from ws db backup (.json) : see "To restore the database from backup" into /opt/ws/ws/install/admin_db.txt Admin help notes ---------------- See /opt/ws/ws/install/admin_db.txt (save, restore...) ws main documentation ===================== - Main documentation generation : - Activate wsenv virtual environment (if not yet done) : source /opt/ws/fab/install/wsenv/bin/activate - Prerequisite (some files to generate and import) : cd /opt/ws/doc/misc generate POST_run.pdf from POST_run.odg generate POST_muse_run.pdf from POST_muse_run.odg mv POST_run.pdf /opt/ws/doc/source/api/files/POST_run.pdf mv POST_muse_run.pdf /opt/ws/doc/source/api/files/POST_muse_run.pdf (for more see /opt/ws/doc/misc/README.md) - Generate : cd /opt/ws/doc make clean make cleanref make ref make html cd /opt/ws/doc ; make clean ; make cleanref ; make ref ; make html The resulting documentation is produced under /opt/ws/fab/doc - Access to main documentation from /var/www/html : - Content : cd /var/www/html sudo ln -s /opt/ws/fab/doc/ ws - Entry point : sudo cp /opt/ws/doc/server/index.html /var/www/html/index.html Apache configuration ==================== /etc/apache2/ports.conf file modifications ------------------------------------------ Modify/verify the apache2 ports.conf file so that the Apache2 server listens to the port 80 dedicated to the ws.conf site (see /opt/ws/ws/projects/ws/ws/ws.conf file) and to the online documentation : Listen 80 ... <IfModule ssl_module> Listen 443 </IfModule> /etc/apache2/apache2.conf file modifications -------------------------------------------- Modify into the apache2.conf file the Timeout value (number of seconds before receives and sends time out, initial value 300) Timeout 600 ws.conf file modifications -------------------------- Give/control into the ws.conf file the appropriate values (ServerName, ws paths), then install it in the appropriate location : - /opt/ws/ws/projects/ws/ws/ws.conf file modifications : ServerName : if needed, replace ws.toulouse.inra.fr by the appropriate IP address erecord paths : /opt/ws/... - install ws.conf file : sudo cp /opt/ws/ws/projects/ws/ws/ws.conf /etc/apache2/sites-available/ws.conf /etc/apache2/sites-available/000-default.conf verification ---------------------------------------------------------- Verify that the apache2 000-default.conf file corresponds with the ws main documentation installation. - /etc/apache2/sites-available/000-default.conf verification : <VirtualHost *:80> DocumentRoot /var/www/html Run === - Apache2 user as owner : - ws code : sudo chown -R www-data:www-data /opt/ws - /var/www (ws online documentation and softwares online documentation) : sudo chown -R www-data:www-data /var/www - machines resources : sudo chown -R www-data:record /opt/ws_rsrc __todo__ a remplacer par : sudo chown -R www-data:record /opt/ws_rsrc/wsm - Remote cluster mounts --> use the following 'grep' command : grep sshfs /opt/ws/ws/install/prod/install_muse.txt | grep reconnect - Activate ws.conf site : sudo a2ensite ws.conf - Activate 000-default.conf site : is supposed to be already done. - Apache run : sudo systemctl reload apache2 Notes ===== - ws online at : - ws web site : http://147.100.179.250 - ws web services : http://147.100.179.250/api/... - Regularly Verify that mount is active : see "Mount monitoring" into install_muse.txt - apache log files : /var/log/apache2/error.log /var/log/apache2/access.log Softwares installation ====================== Install the softwares you want to be able to be used by ws web services, as required by ws. - See /opt/ws/softwares/install/install.txt : containers (install_softwares.txt), and (maybe) documentation (install_softwares_documentation.txt) Softwares documentation ----------------------- URLs : If some IP addresses values directly appeared in software documentation as 127.0.0.1:8000 (development case), they should be modified as : 147.100.179.250 (@vm) here for production case. If needed, put the appropriate IP address : __todo__ to be tested - software documentation : cd /var/www/html/softwares sed -i 's/127.0.0.1:8000/147.100.179.250/g' * sed -i 's/127.0.0.1:8000/147.100.179.250/g' */* sed -i 's/127.0.0.1:8000/147.100.179.250/g' */*/* sed -i 's/127.0.0.1:8000/147.100.179.250/g' */*/*/* Be careful : respect Apache2 rights (see "Run" above). *******************************************************************************
install_singularity.txt :
******************************************************************************* Install Singularity 3.5.3 ******************************************************************************* Available to install the Singularity 3.5.3 version : - in development case (PC under ubuntu) or - in production case (ws VM under debian). Follow https://sylabs.io/guides/3.5/user-guide/quick_start.html Note ==== The Singularity 3.5.3 version is the one used by ws (by default) on the Muse machine. To know available Singularity versions on the Muse machine : @muse : module available => singularity/2.2.1 singularity/2.4 singularity/2.6 singularity/3.3 singularity/3.5 To identify a Singularity version on the Muse machine : @muse : module load singularity/3.5 ; singularity --version => singularity version 3.5.3 Install ======= - Install system dependencies : sudo apt-get update sudo apt-get install build-essential libssl-dev uuid-dev libgpgme11-dev sudo apt-get install squashfs-tools libseccomp-dev sudo apt-get install wget pkg-config git cryptsetup - Install Go (version archive suitable to Debian 10, chosen on the Go Downloads page https://golang.org/dl) : wget https://golang.org/dl/go1.15.2.linux-amd64.tar.gz sudo tar -C /usr/local -xzvf go1.15.2.linux-amd64.tar.gz rm go1.15.2.linux-amd64.tar.gz echo 'export PATH=/usr/local/go/bin:$PATH' >> ~/.bashrc && source ~/.bashrc - Download Singularity from its release, compile and install it : export VERSION=3.5.3 wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz tar -xzf singularity-${VERSION}.tar.gz cd singularity ./mconfig make -C builddir sudo make -C builddir install - Control : singularity --version => singularity version 3.5.3 *******************************************************************************
install_muse.txt :
******************************************************************************* Install communication between ws machine (VM) and remote muse machine (cluster) ******************************************************************************* Case ==== - muse_user='roussen' user on @muse=@muse-login.hpc-lr.univ-montp2.fr (muse machine). On ws machine (@vm=@147.100.179.250), vm_user and apache_user : - vm_user='nrousse' user on @vm=@147.100.179.250 VM (ws machine). - We want to be able to remote use muse_user@muse cluster from vm_user@vm (roussen@muse-login.hpc-lr.univ-montp2.fr from nrousse@147.100.179.250). - apache_user='www-data' user on @vm=@147.100.179.250 VM (ws machine). - We want to be able to remote use muse_user@muse cluster from apache_user@vm (roussen@muse-login.hpc-lr.univ-montp2.fr from www-data@147.100.179.250). Groups ====== @muse (roussen@muse-login.hpc-lr.univ-montp2.fr) : - Identify muse_user group : id roussen => Response : uid=1654(roussen) gid=1112(record) groupes=1112(record) groupname=record, gidvalue=1112 @vm (nrousse@147.100.179.250) : - Verify that groupname GID (record, gidvalue=1112) does not exist on @vm yet (to see all the UID/GID : cat /etc/passwd ; cat /etc/group; ) - create groupname group (with same GID value as at @muse), add vm_user and apache_user into : sudo addgroup --gid 1112 record sudo usermod -aG record nrousse sudo usermod -aG record www-data - Verification : id www-data => Response OK : uid=33(www-data) gid=33(www-data) groupes=33(www-data),1112(record) id nrousse => Response OK : uid=1321(nrousse) gid=500(miat) groupes=500(miat), ... ,1112(record) Access authorization (by ssh key) from apache_user@vm to muse_user@muse ======================================================================= @vm as vm_user (nrousse@147.100.179.250) : - Generate a ssh key for apache_user (www-data) under /opt/www-data.ssh if not existing yet : - Authorization (prerequisite) : sudo visudo -f /etc/sudoers.d/ws Then add into this /etc/sudoers.d/ws file the 2 lines : www-data ALL=(nrousse) NOPASSWD: /usr/bin/ssh-keygen www-data ALL=(nrousse) NOPASSWD: /usr/bin/ssh - www-data key folder (out of /var/www) : sudo mkdir /opt/www-data.ssh sudo chown -R www-data:www-data /opt/www-data.ssh sudo mkdir /var/www/.ssh sudo vim /var/www/.ssh/config Then add into this /var/www/.ssh/config file the 2 lines : Host * IdentityFile /opt/www-data.ssh/id_rsa sudo chown -R www-data:www-data /var/www/.ssh - ssh key generation : sudo -u www-data ssh-keygen -o -t rsa -b 4096 ... Enter file in which to save the key : /opt/www-data.ssh/id_rsa ... no passphrase @muse as muse_user (roussen@muse-login.hpc-lr.univ-montp2.fr) : - Register the ssh public key of apache_user@vm (www-data@147.100.179.250) : - If /home/roussen/.ssh does not exist, generate ssh key to create it : ssh-keygen -o -t rsa -b 4096 - Add into the /home/roussen/.ssh/authorized_keys file the apache_user@vm public key (cf www-data@147.100.179.250:/opt/www-data.ssh/id_rsa.pub) @vm as vm_user (nrousse@147.100.179.250) : - Initialize the access (1st connexion) from apache_user@vm to muse_user@muse : sudo -u www-data ssh roussen@muse-login.hpc-lr.univ-montp2.fr ... The authenticity of host ... continue connecting (yes/no)? yes Access authorization (by ssh key) from vm_user@vm to muse_user@muse =================================================================== @vm as vm_user (nrousse@147.100.179.250) : - Generate a ssh key for vm_user if not existing yet : ssh-keygen -o -t rsa -b 4096 - Get vm_user@vm public key value : cat /home/nrousse/.ssh/id_rsa.pub @muse as muse_user (roussen@muse-login.hpc-lr.univ-montp2.fr) : - Register the ssh public key of vm_user@vm (nrousse@147.100.179.250) : - If /home/roussen/.ssh does not exist, generate ssh key to create it : ssh-keygen -o -t rsa -b 4096 - Add into /home/roussen/.ssh/authorized_keys file the vm_user@vm public key (cf nrousse@147.100.179.250:/home/nrousse/.ssh/id_rsa.pub) @vm as vm_user (nrousse@147.100.179.250) : - Initialize the access (1st connexion) from vm_user@vm to muse_user@muse : ssh roussen@muse-login.hpc-lr.univ-montp2.fr ... The authenticity of host ... continue connecting (yes/no)? yes Mount (requests, simg) between vm_user@vm and muse_user@muse ============================================================ Configuration description ========================= On muse_user@muse : ws/simg scratch/ws/requests On vm_user@vm : /opt/ws_rsrc/muse/simg /opt/ws_rsrc/muse/requests Mounts to be done between : - muse_user@muse:ws/simg and vm_user@vm:/opt/ws_rsrc/muse/simg - muse_user@muse:scratch/ws/requests and vm_user@vm:/opt/ws_rsrc/muse/requests Install ======= @muse as muse_user (roussen@muse-login.hpc-lr.univ-montp2.fr) - Create folders if not already existing : mkdir /home/roussen/ws mkdir /home/roussen/ws/simg mkdir /home/roussen/scratch/ws mkdir /home/roussen/scratch/ws/requests (Note : /home/roussen/scratch is link to /lustre/roussen) (See /opt/ws/softwares/install/install_softwares.txt to fill simg folder with *.simg files) @vm as vm_user (nrousse@147.100.179.250) : - Create folders if not already existing : sudo mkdir /opt/ws_rsrc/muse sudo mkdir /opt/ws_rsrc/muse/simg sudo mkdir /opt/ws_rsrc/muse/requests sudo chmod 775 /opt/ws_rsrc/muse sudo chmod 775 /opt/ws_rsrc/muse/simg sudo chmod 775 /opt/ws_rsrc/muse/requests - required before mount below : sudo chown -R www-data:record /opt/ws_rsrc - Other preparations : sudo apt-get install sshfs Into /etc/fuse.conf file, uncomment : user_allow_other - Mount : sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,cache=no,allow_other roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests /opt/ws_rsrc/muse/requests sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,cache=no,allow_other roussen@muse-login.hpc-lr.univ-montp2.fr:ws/simg /opt/ws_rsrc/muse/simg Mount monitoring : @vm as vm_user (nrousse@147.100.179.250) : - To verify that mount is active : mount => Mount not active if response missing lines : roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests on /opt/ws_rsrc/muse/requests type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1321,group_id=500,allow_other) roussen@muse-login.hpc-lr.univ-montp2.fr:ws/simg on /opt/ws_rsrc/muse/simg type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1321,group_id=500,allow_other) - Mount again if mount is not active anymore - memo : with -o nonempty : sshfs -o cache=no,allow_other -o nonempty roussen@muse-login.hpc-lr.univ-montp2.fr:scratch/ws/requests /opt/ws_rsrc/muse/requests unmount : fusermount -u /opt/ws_rsrc/muse/requests *******************************************************************************
See : Softwares installation