<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<font face="serif">Thanks for Nick for letting me hop on his account
and try some testing this evening. The new version of MPI has
some quirkiness that I have not encountered before, but after
doing some reading and testing things out, I think I have it. I
know it works on his account.<br>
<br>
There are SEVERAL things -- all small modifications to things we
have already done -- to get this all to work. I will try to
explain each one.<br>
<br>
FIRST -- PBS. Several of you noted today that you had not logged
into systems using the full path name -- like
csc204com21.cs.mercer.edu. You have just been using
csc204com21. While the shorthand works for logging in,
PBS/Torque requires the full path name to copy files. Just look
at your PBS e-mails and you will see a lot of "post-processing
errors". The problem is that the job on the node can't copy the
output back to zeus. From zeus login to zeus.cs.mercer.edu.
This will put the appropriate line in the ./.ssh/known_hosts file
and all should be good for copying files back to zeus. Make sure
that you do this for all the systems in lab 204!<br>
<br>
SECOND -- PATHS -- in all your PATH and LD_LIBRARY_PATH
environment variables move the CUDA stuff to the end and then move
the /usr/local/maui/bin just prior to the CUDA stuff in your
executable PATH. This way you will be assured of being able to
use showq on zeus, but will still have access to CUDA on the
clusters.<br>
<br>
THIRD -- I noted today that several of you could not run ther
"orted" command even though it was in your path. This only
happened when you ran mpirun. To get around this we need to tell
the MPI environment exactly where to find the MPI installation on
each system. Modify your mpirun command in your PBS job file to
look like this<br>
<br>
mpirun --prefix /usr/lib64/openmpi -np $n --hostfile
$PBS_NODEFILE --map-by ppr:1:node mpimmm<br>
<br>
FOURTH -- Cleanup Script -- modify the ortecleanup.pl script so it
uses the full path to orte-clean<br>
<br>
change the lines that have orte-clean so that they are now
/usr/lib64/openmpi/bin/orte-clean<br>
<br>
After making all these changes, try recompiling your code and
running the PBS job.<br>
</font>
<pre class="moz-signature" cols="72">--
Andrew J. Pounds, Ph.D. (<a class="moz-txt-link-abbreviated" href="mailto:pounds_aj@mercer.edu">pounds_aj@mercer.edu</a>)
Professor of Chemistry and Computer Science
Mercer University, Macon, GA 31207 (478) 301-5627
<a class="moz-txt-link-freetext" href="http://faculty.mercer.edu/pounds_aj">http://faculty.mercer.edu/pounds_aj</a>
</pre>
</body>
</html>