7.8. Running software remotely or in the background

Math IT Support → Remote Access to Math Accounts → Running software remotely or in the background

Introduction

If you're doing heavy-duty computation with Matlab/Maple/Mathematica or your own programs, you may have to wait a long time (perhaps days) for your computation to finish. Since you don't want to sit in front of a lab computer for days, or tie up your office computer, you can put tasks in the background, and/or put them on a remote machine. You can then log out and return later to  get your results,  Note that it is very easy to take up too much CPU time or memory with a background computational process. The "chivos" are dedicated to this purpose for grad students.

Commands

Before launching background/remote tasks, you must:

• Learn to use ssh to connect to remote systems.
• Learn to use top or ps, kill, and renice to monitor and manage processes you've started.
• Learn to use &, nice, nohup, and ulimit to launch processes in a desired way.
• For interactive (non-batch) programs: know about ssh -Y, and learn either vnc or screen.

You can type "man command" to get more information on any of these commands.  This article also gives a nice overview.

top

top displays a list of active processes, how much memory and CPU time each process is using and who owns it, etc. If a machine is sluggish, top will tell you why! From within the topdisplay, you can  kill or renice your running processes. Hit ? for some help, and read the manpage: man top. (If you spot someone else's runaway process eating too much CPU, you can email system or the other user . They might not know about it unless you inform them.)

ps

Makes a list of running processes along with additional information. A common invocation isps -aux, which lists a lot of extra information. Another useful invocation is ps -aux -forest, which draws lines to indicate which process spawned which process. (The commandpstree is similar.)

kill

Lets you kill one of your processes. You must know the numeric process ID. You can see the PID in the top display, or you can get a list using ps -aux. Instead of using kill as a separate command, you can access it inside top. Read man kill for more info. Generally you can first send a "kill 15 xxxxx" (where xxxxx is the process ID) and wait a few seconds.  If the process does not die, do a "kill 9". If that doesn't work, then that machine has become unstable and you can contact the computer support staff.

nice

This starts a task with a nice value, which means the task will do a better job sharing the CPU when you try to run other programs or when other users log in to the same computer. All heavy-duty tasks not running on chivos should be nice. Manpage: man nice.

renice

Forgot to start something using nice? Use this to go back and make it nicer. You can also do this inside top.  See: man renice.

&

Putting this after a command puts that command in the background, so that you can continue to give other commands. If you log out, your background task might get killed automatically (to avoid that, see nohup).  If the command  creates windows (e.g., Mathematica &), it will create them even from the background. Most people use & when starting programs like Mathematica so that they don't lose their prompt. (Try it both ways and the previous sentence will make more sense.) See: http://en.wikibooks.org/wiki/A_Quick_Introduction_to_Unix/Job_Control

nohup

To put something in the background so that it doesn't die when you logout, start it with nohup. (e.g. "nohup command &"  ) This works with text-only programs. In other words, the "nohupped" command won't work if it tries to create windows, or needs keyboard input.  The command must send all output to a file. You can use this with your own C/C++ programs, but you can't use it with most software packages. See: man nohup. screen does much more than  nohup.

ulimit

To prevent a program from using too much time or too much memory, you can set limits in a shell, and all programs started from that shell will have those limits imposed. Use ulimit -ato get a list of limits. To confirm you understand what is going on, set various values ridiculously low and try to run your program to make sure it gets stopped by the operating system. Then quit that shell, open a new shell, and set more reasonable limits. Go read man ulimit.

screen

This creates an environment in which you can start an interactive text-only program, such as GAP or Macaulay, or even Mathematica in its text-only non-notebook mode, but you can log out and it keeps going. On any linux machine, e.g. chivo, enter screen, a "screen" process starts with a new prompt. Then you can enter a command, e.g:  nice Singular, then type some Singular commands that will take days to complete, hit "control-A d" and log out. A day later you can ssh  to chivo, do screen -r to reconnect, and you'll be right back in the Singular session. It's very cool. Type man screen on a linux machine for more information.

ssh

From within a shell, you can use ssh user@remote to obtain a shell on a remote system. Then you can issue commands (such as many of the ones described on this page) on the remote system. If you use ssh -Y user@remote then there will also be interactive X forwarding, meaning that programs launched on the remote system can open windows on your local system (assuming you are logged in to a graphical X desktop). You might also want to read about using ssh to obtain remote shell access to your network account.

VNC

This is a graphical version of screen. You can create an entire virtual X desktop and connect to it from home, office, anywhere you can access a web browser. You'll have to read man vncserver and man vncviewer. You might want to read about using vnc to obtain remote desktop access to your network account.