The Stanford NLP (Natural Language Processing) Group

A Brief Guide to Working with Multiple Machines in the NLP Group

Many students. Much code. A handful of machines. From such tragic seeds oft bloom the flowers of malcontent. Follow this guide that you may spend minutes bending the machines to thy will, rather than days bending to theirs.

Step 1: ssh

Everything depends on being able to ssh to any machine without your password.

ssh-keygen -d -q and hit enter thrice.
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys


 chmod 0600 !$

bash -c 'for i in `cat /u/nlp/machine-info/machines.txt`; do ssh $i echo $i; done' and type yes for each machine. y is insufficient. (Alternately, bug Miles for an/etc/ssh/ssh_known_hosts).

Step 2: William's machine info page

Learn to love: http://nlp.stanford.edu/local/machines.shtml. This displays the status of all NLP machines and is updated every 5 minutes.

You'll need to authenticate to access it. The username is nlp; the password is Chris's middle name backwards.

The machine info page also provides a rudimentary reservation system that is useful for critical work like upcoming paper deadlines. So it's good to get in the habit of checking this page before running any large jobs.

To attach an explanatory note to your jobs, put a line of text in ~/.javanlp-note. To claim a machine, put a line of the format machinename: reason in ~/.javanlp-claims. Both files need to be touched every 24 hours to keep them on the page. (In both cases, if you're a new user, make sure your userid is found in the list near the top of /u/nlp/machine-info/compile-sysinfo.rb.)

Step 3: William's `mstat` and `mps`

~wtm/bin/mstat displays a concise version of the machine info page.

~wtm/bin/mps polls all the machines and generates a ps-style listing of all your jobs. It takes a few seconds to run.

Step 4: screen

man screen for an introduction

Put the following in ~/.screenrc:

escape ^pp
defscrollback         999            # default: 100
defencoding utf8
defutf8         on
encoding utf8 utf8
hardstatus alwayslastline "%m/%d %c | %w  "
activity              "activity in %n (%t) [%w:%s]"

This makes ctrl-p the screen escape key rather than ctrl-a, allows utf8 characters to be displayed correctly, and shows a status line so you don't forget you're running screen already and try to run a nested one. (Painful!)

All you really need to know:

screen -S name to start a new screen with a name
screen -list to see all your screens
(within screen) ctrl-p d to detach the screen
screen -R name to return to the named screen

Step 5: William's `gos`

Run William's God of Screen: ~wtm/bin/gos for a menu of all running screens on all machines, and the ability to swap between them easily.

Step 6: switching between machines easily

I find it awfully useful to be able to quickly switch between machines while preserving the same working directory. Put the following in your ~/.bashrc.user:

export PS1="$FROM\h:\w\$ "

for i in `cat /u/nlp/machine-info/machines.txt`
do
  alias $i="history -a; ssh -t $i \"cd \`pwd\`;FROM=$FROM[`basename \`hostname\` .Stanford.EDU`] bash\""
done

Now you can simply type any machine name, e.g. "jar", and you'll be transported to jar while preserving your current directory and command history. Your prompt is updated to keep track of the machine stack.

Step 7: profit!

tbd