List of MOLGENIS servers

These are production servers: For now:

• Apache runs on 80
• Molgenis runs on 8080 (either via Tomcat or standalone)

Server	Alias	Server Port	Connector Port	Application	App name	Contact	Backup?	GPFS
http://molgenis01.target.rug.nl	Static downloads for all apps	8080	NA	NA	NA	mswertz		n
http://molgenis02.target.rug.nl	http://www.xqtl.org/stable	8080	NA	xQTL demo	xqtl	jvelde		n
http://molgenis03.target.rug.nl	http://fwnstable.animaldb.org	8080	NA	AnimalDDB	animaldb	eroos	SQL dump copied to GPFS daily	n
http://molgenis04.target.rug.nl	http://www.wormqtl.org	8080	NA	WormQTL	wormqtl	jvelde		y
http://molgenis05.target.rug.nl	http://www.deb-central.org	8080	NA	COL7A1		rwagner		n
http://molgenis06.target.rug.nl	na	8080	NA	BBMRI Catalogue	bbmri_gcc.war	antonakd		n
http://molgenis07.target.rug.nl	LFN	8080	NA	Learning From Nature	xqtl	jvelde		n
http://molgenis08.target.rug.nl	IBD	8080	NA	Parel IBD	xqtl	jvelde		y
http://molgenis09.target.rug.nl	http://www.chd7.org	80	NA	CHD7		rwagner		n
http://molgenis10.target.rug.nl				TIFN + GIDS		roan		n
http://molgenis11.target.rug.nl		80	NA	designgg		jvelde		n
http://molgenis12.target.rug.nl		80	NA	c2cards		rwagner		n
http://molgenis14.target.rug.nl		80	NA	GoNL		rwagner + pneerincx		n

List of MOLGENIS test servers

These are test servers that we may destroy at any time

vm7.target.rug.nl	8305	8380	ngs_test		rwagner
vm7.target.rug.nl	8405	8480	autodeploy		eroos
vm7.target.rug.nl	7005	7080	c2cards		rwagner
vm7.target.rug.nl	-	9000 through 9020	xQTL apps and DesignGG		jvelde
application32.target.rug.nl	-	9000 through 9020	Reserved for xQTL / DesignGG / et al.		jvelde
vm7.target.rug.nl	6405	6480	CVDB		rwagner
vm7.target.rug.nl	7778	7777	Lifelines	lifelines	antonakd
vm7.target.rug.nl	9998	9999	compute	compute	george
vm7.target.rug.nl	7305	7380	humanpathwaydb		rwagner + morris
vm7.target.rug.nl	7405	7480	computegonl	computegonl	freerk
vm7.target.rug.nl	7605	7680	chado		rwagner
vm7.target.rug.nl	-	11000 through 12000	Used for selenium tests		jvelde
application32.target.rug.nl	-	11000 through 12000	Reserved for selenium tests		jvelde
vm7.target.rug.nl	4444	4888	Catalogue	catalogue.war	antonakd, chao
vm7.target.rug.nl	?	?	scrum_online	Standalone	eroos
vm7.target.rug.nl	?	?	animaldb	Standalone	eroos

FAQ for servers and VMs

General

• What is a virtual machine? - VM is a server that physically runs on top of another server.  http://en.wikipedia.org/wiki/Virtual_machine
• What is the advantage? - You can copy VMs easily to other hardware without need of reinstall (in particular useful if hardware brakes).
• Where do we use our VMs for? - for all our MOLGENIS deployments.
• How request one? - Ask Morris to add your new VM to the above list.
• Can VMs be automatically created and killed? - No, currently all server startup/shutdown goes via Ger or Wietze

Access

• What kind of account do I need for VM access, and where do I get it? You need the standard LDAP in gcc group.
• How do I get a /home/myname directory? - Get somebody having root to create it for you on a particular vm (Ger, Morris, Wietze, ...)
• How do I log in to a VM? - From UMCG ssh yourname@…. From home first go via vm7.target.rug.nl
• Do we use SSH keys for authentication? - No. For the moment everybody can setup their own keys if they like
• Is my account centralized across all VMs? - Yes. One login and password. Exceptions: vm7.target.rug.nl and gbic.target.rug.nl and millipede.service.rug.nl
• How do I install an additional application or library on a VM? - Binaries you can just put on your %PATH; otherwise as a 'root' to install RPMs
• Do I have root access on the VM's? - No. We aim to have VMs work without root access and have all software installed once centrally.
• Who have root access and when would they use it? - Ger, Morris, Wietze, ...
• Can I be a sudoer on a VM? Should I be? - QUESTION: can we install sudo on our servers??

Operating system

• Which OS do the VMs run? SUSE 11.x
• Is there anything I should pay special attention to?
• What are some basic commands and tools to get me started?

Storage and backups

• What is GPFS? - GPFS is our central storage. We have a small share of it at /target/gpfs2/gcc/.
• What is the relation to the VMs? - All VMs have access to the shared storage always using the same path /target/gpfs2/gcc/.
• When should I work on the local drive, and when on GPFS?
• How do I access the GPFS? Do I need additional permissions?
• Are database better located on the local drive, or on GPFS?
• What locations are suitable for certain data or user groups on GPFS?
• Can I request additional local harddrive space, or other upgrades like CPU or RAM?
• Are the VMs automatically backupped? If not can I request this? - BIG TODO
• Which folders on the VMs or GPFS are backupped? How can I verify or request this? - BIG TODO. Are whole VMs backupped?
• How do I restore a backup? - BIG TODO

Configuration

• How is my VM configured regarding to access and security, other than SSH/LDAP? - Everybody in 'gcc' group can login via ssh only.
• Where do I configure e.g. the firewall or proxy settings of a VM?
• Should I edit this if I can, and if so, what could go wrong?
• Which ports are opened by default, and what is their function? - Standard port 80 and port 8080 are open.
• How can I tweak memory usage of my applications? - TODO howto do this with ANT and Tomcat startup script
• How can I tweak memory usage of mysql? - TODO howto my.conf. In principle each VM should be preoptimized
• Can I monitor CPU, RAM, network or drive IO? - TODO

Hosting

• What processes should be run under which user, with regards to security? - TODO If with GPFS you ideally would like a user per app
• How secure should a demonstration system be, with regards to running tools, disk access, HTML exploits, etc? - TODO sandboxed (only local resources)
• How can I test this, and what is the worst case scenario? - Worst case: private data is shared or deleted; This should be prevented at all times.
• Should I prepare for this? - Yes: simplest solution is to limit access to local data only.
• Can I offer static files or HTML pages for download on a VM? How do I do this? - Put them in standard /srv/www/htdocs (served at port 80)
• Can I run other dynamic content scripts such as Perl / PHP via my VM? - No, please request if you need this
• How secure should a production system be? Should it be seperated from demonstration systems? - Yes
• What guarantees should or can I offer my customers? - TODO need recovery procedure; 1st line, nightly backup of the VM; 2nd line, complete reinstall

SOPs

• Install Java / ANT

Install and run standalone

#####################################
# SOP: Installation of standalone + mysql app 
#      on SUSE (any molgenis user)
#####################################

#change line below to match your app
app=animaldb

#become molgenis user
sudo su molgenis
cd /srv/molgenis

# if /srv/molgenis doesn't exist, ask 'root' user to run installation SOP

#checkout your app
mkdir $app
cd $app
svn co http://www.molgenis.org/svn/molgenis/trunk molgenis
svn co http://www.molgenis.org/svn/molgenis_apps/trunk molgenis_app

#create mysql db (if applicable)
mysql -u molgenis -pmolgenis -e "create database $app"; 

#generate app
cd molgenis_app
ant -f build_$app.xml clean-generate-compile-test

#run
kill -9 `lsof -i :8080 -t`
nohup ant -f build_$app.xml run &

Install on tomcat6 and mysql

#####################################
# SOP: Installation of tomcat6 + mysql app (any gcc member)
#####################################

#change line below to match your app
app=animaldb

#become molgenis user
sudo su molgenis
cd /srv/molgenis

# if /srv/molgenis doesn't exist, ask 'root' user to run installation SOP

#checkout
mkdir $app
cd $app
svn co http://www.molgenis.org/svn/molgenis/trunk molgenis
svn co http://www.molgenis.org/svn/molgenis_apps/trunk molgenis_apps
cd molgenis_apps

#create mysql
mysql -u molgenis -pmolgenis -e "create database $app"; 

#generate war
ant -f build_$app.xml clean-generate-compile-war

#delete previous war, if updating
rm /srv/tomcat6/webapps/$app.war

#copy new war (tomcat will automatically reload the app)
cp dist/war/$app.war /srv/tomcat6/webapps/$app.war

Create backup cron job

As root user edit /etc/crontab

00 04 * * * molgenis /usr/bin/mysqldump --user=molgenis --password=molgenis YOURDATABASENAME | gzip > /srv/molgenis/backups/YOURDATABASE_`date +%Y-%m-%d`.sql.gz

NB: this assumes the vm itself is backed up as well!!

Copy a file from sftp to molgenis VM

Upload your data to your home on GPFS2 using the sftp server:
See DataStorage
Then:

# login to Molgenis VM server
ssh username@molgenisXX.target.rug.nl

# Change to the molgenis user 
sudo su molgenis

# Change your umask the change the default permissions for new files and dirs you create.
umask 0077

#
# Use secure copy to fetch 'dir' from another server with GPFS mounted to the Molgenis VM:
#
# Option A: From the sftp server to the current directory (note: path is relative and you have a different home on the sftp server.)
scp -r username@sftp.gcc.rug.nl:home/username/dir ./ .
#
# Option B: From another server with GPFS mounted to the current directory (note: path is relative)
scp -r username@sftp.gcc.rug.nl:dir ./ .

Installation of SUSE VM

#####################################
# SOP: Installation of SUSE VM (root only)
#####################################

# stop and unmount gpfs
mmshutdown

# mysql (assume installed and running)
echo "grant all privileges on *.* to molgenis@localhost identified by 'molgenis'; flush privileges;"  > mysqlconfig
mysql -u root -p < mysqlconfig

# install particlar java and javac versions [interactive]
wget http://gbicdev.target.rug.nl/downloads/jdk-7u3-linux-x64.rpm
zypper in jdk-7u3-linux-x64.rpm

#update paths
update-alternatives --install "/usr/bin/java" "java" "/usr/java/latest/jre/bin/java" 1
update-alternatives --set java "/usr/java/latest/jre/bin/java"
update-alternatives --install "/usr/bin/javac" "javac" "/usr/java/latest/bin/javac" 1
update-alternatives --set javac "/usr/java/latest/bin/javac"
update-alternatives --install "/usr/bin/jre" "jre" "/usr/java/latest/jre" 1
update-alternatives --set jre "/usr/java/latest/jre"
echo "export JAVA_HOME=/usr/java/latest/" >> /srv/molgenis/.bashrc

# create molgenis home directory
mkdir /srv/molgenis
chown -R molgenis /srv/molgenis

# add su molgenis for all sudoers
echo "%gcc    ALL=(ALL) NOPASSWD:/bin/su molgenis" >> /etc/sudoers

#add molgenis to restart tomcat
echo "molgenis   ALL=(ALL) NOPASSWD:/etc/init.d/tomcat6 restart" >> /etc/sudoers


# assumes ant, tomcat, svn, libcurl, R installed (skipped for now)
# zypper install tomcat6

wget http://download.opensuse.org/repositories/devel:/tools:/scm:/svn/SLE_11/x86_64/subversion-1.7.4-75.1.x86_64.rpm
zypper in subversion-1.7.4-75.1.x86_64.rpm


# use 'zypper search java' to see what packages are there matching search term

# TODO: configure mysql large database

echo "HOME=/srv/molgenis/" >> /etc/environment

# allow molgenis user to add wars
usermod -G tomcat molgenis

Upload metadata

to log in first time with admin/admin go to the molgenis_apps folder and type:

ant -f build_APP.xml updatedbfillmeta
replace APP with your app name

NOTE

If Ant gives you Error: Could not find or load main class org.apache.tools.ant.launch.Launcher, input this:

export JAVA_HOME=/usr/java/latest/

And try again.

How to change firewall

As admin edit /etc/iptables-access.sh