xQTL workbench - Setup job submission
TODO! content here
Configure your local system to accept and run jobs
- Use Settings -> Install R packages
- BioConductor package, manual install??
- install.packages("bitops", lib="~/libs")
- install.packages("RCurl", lib="~/libs")
- install.packages("qtl", lib="~/libs")
- Curl install??
Configure your computer cluster to accept and run jobs
- You have a LINUX/UNIX cluster, then:
- Install OpenPBS (torque+maoi)
- Install R
- Your scheduler (master/head..) node must a SSH keyboard interactive login mechanism
- Try to log in to the cluster from your application server. This will accept the SSH key and ensure your firewall isn't blocking outgoing SSH.
Inside R: install.packages("RCurl",lib="~/libs")
might result in...
installing *source* package ‘RCurl’ ...
checking for curl-config... no
Cannot find curl-config
ERROR: configuration failed for package ‘RCurl’
Fix by... sudo apt-get install libcurl3-openssl-dev (SuSE: zypper in libcurl-devel)
inside, try again
*** installing help indices
** building package indices ...
* DONE (RCurl)
Install R packages plugin
- Press 'Install' for R/qtl (package 'qtl')
Going to install: qtl
EXECUTING: echo install.packages\(\"qtl\",repos=\"http://cran.xl-mirror.nl/\",lib=\"/Users/joeri/libs\"\)\;q\(\"no\"\) > /var/folders/Qs/QsYTu8N8HhGqehzsaQaA5U+++TQ/-Tmp-/tmpRfile_1315818859357251000.R
EXECUTING: R CMD BATCH /var/folders/Qs/QsYTu8N8HhGqehzsaQaA5U+++TQ/-Tmp-/tmpRfile_1315818859357251000.R
exit value = 127
Instalation of qtl failed
Means you first need to install R. After this is done, try again. Now it should read
Going to install: qtl
EXECUTING: echo install.packages\(\"qtl\",repos=\"http://cran.xl-mirror.nl/\",lib=\"/Users/joeri/libs\"\)\;q\(\"no\"\) > /var/folders/Qs/QsYTu8N8HhGqehzsaQaA5U+++TQ/-Tmp-/tmpRfile_1315820155628099000.R
EXECUTING: R CMD BATCH /var/folders/Qs/QsYTu8N8HhGqehzsaQaA5U+++TQ/-Tmp-/tmpRfile_1315820155628099000.R
Done installing qtl
After which the plugin checks whether the libraries are installed by looking at USER_HOME/libs. If a directory is not found, it is assumed that it is not installed.
checking dir: /Users/joeri/libs/bitops
checking dir: /Users/joeri/libs/qtl
checking dir: /Users/joeri/libs/RCurl
- Configure your local system to accept and run jobs
- Configure your computer cluster to accept and run jobs
Running on Tomcat
- NOTE: The Tomcat user is the 'special' user under which Tomcat is running and accessing the system. (ie. tomcat6)
- The Tomcat user must own the Tomcat home directory to install dependencies. (ie. /user/share/tomcat6)
- NOTE: If libraries are installed owned by root, they cannot be overwritten by Tomcat, meaning you could be stuck with an old version. If problems occur, delete the libraries and reinstall using the Dependency Manager plugin.
- The Tomcat user must own the Tomcat java working directory to be able to execute commands. (ie. /var/lib/tomcat6, use the tomcattmpdir helper servlet to find out)
- If 'RCurl' won't install under UNIX, try using the commandline as root and install the following package: 'libcurl3-openssl-dev'
Configuration for Windows
- Install RTools RTools
- Install Curl for Windows, use a generic binary. Put the install dir on your PATH. cURL
- Optional: if your cluster uses a module system to load programs, load this set:
module initadd torque module initadd maui module initadd R
- Then log out from the cluster, then log back in.
- Create a libs folder for the dependencies using mkdir libs
- Start R and install a few things:
install.packages("RCurl",lib="~/libs") install.packages("qtl",lib="~/libs") install.packages("snow",lib="~/libs")
- Install our custom 'ClusterJobs' library by performing a 'checkout' and R install. From your home folder, do:
wget -r -l2 http://www.xgap.org/svn/xgap_1_4_distro/handwritten/java/plugins/cluster/R/ClusterJobs/ cd www.xgap.org/svn/xgap_1_4_distro/handwritten/java/plugins/cluster/R/ R CMD INSTALL ClusterJobs --library=~/libs
- Make sure the firewall of your receiver (server that is running XGAP+Cluster+R/qtl) is not blocking port 8080 so the cluster can report back progress and save your results.
other old stuff
Adding new ComputationResources
It is possible to add new types of ComputationResource, places where jobs can be executed.
The Java interface ComputationResource is located at /xgap_1_4_distro/handwritten/java/plugins/cluster/interfaces/ComputationResource.java in your checked-out project and can be implemented.
In the folder /xgap_1_4_distro/handwritten/java/plugins/cluster/implementations/ you can find two implementations that we currently provide: LocalComputationResource.java and ClusterComputationResource.java.
They implement the following functions:
public boolean installDependencies()
Install the needed R-packages on this ComputationResource.
public boolean cleanupJob(int jobIs)
Remove job files and generated data from this ComputationResource.
public String executeCommand(Command command)
Execute a system command on this ComputationResource.
public List<String> executeCommands(List<Command> commands)
Execute multiple system commands. Not a wrapper for executeCommand, because it can behave differently.
At the moment, extending the system is not trivial because there are many ComputationResource specific parts in the core workings of ClusterPlugin.java. In further development, we will gradually make this more generic and easily extendable.