psh allows you to spawn processes in Unix shell-style way. It is inspired by sh module but has a fully different implementation and API.
Unix shell is very convenient for spawning processes, connecting them into pipes, etc., but it has a very limited language which is often not suitable for writing complex programs. Python is a very flexible and reach language which is used in a wide variety of application domains, but its standard subprocess module is very limited. psh combines the power of Python language and an elegant shell-style way to execute processes.
Follow it on GitHub: https://github.com/KonishchevDmitry/psh.
Print output of echo -n "text":
from psh import sh
print sh.echo("-n", "text").execute().stdout()
Get a list of all available network interfaces (ifconfig | egrep -o "^[^[:space:]:]+"):
from psh import sh
interfaces = [ iface.rstrip("\n") for iface in sh.ifconfig() | sh.egrep("-o", "^[^[:space:]:]+") ]
Check free disk space on remote host myserver.com:
import re
from psh import sh
# ssh myserver.com 'df | egrep "^/dev/"'
with sh.ssh("myserver.com", sh.df() | sh.egrep("^/dev/"), _shell = True) as ssh:
for line in ssh:
match = re.search(r"^(/dev/[^\s]+)\s+(?:[^\s]+\s+){3}(\d+)%\s+(/.*)$", line.rstrip("\n"))
device, used, mount_point = match.groups()
if int(used) > 80:
print "{0} ({1}) ran out of disk space ({2}%)".format(device, mount_point, used)
Output:
/dev/sda1 (/) ran out of disk space (86%)
/dev/sda2 (/mnt/data) ran out of disk space (95%)
You can install the latest psh version using pip:
pip install psh
or using easy_install:
easy_install psh
or you can install the latest development version by issuing the following commands:
git clone https://github.com/KonishchevDmitry/psh.git && cd psh
python setup.py build
sudo python setup.py install
Python 2.6+ and 3.0+ are supported.
psh module has an object named sh which is a factory of Program objects. A Program object represents a program which can be executed. To obtain a Program object just write:
from psh import sh
echo = sh.echo
For programs that have dashes in their names, for example google-chrome, substitute the dash with an underscore:
from psh import sh
sh.google_chrome("http://google.com")
Note
For programs with more exotic characters in their names, like . or _ you may use sh.__call__() method:
from psh import sh
python = sh("python2.7")
script = sh("/path/to/script.sh")
To execute a program just call it as if it is a function and then call execute() method:
from psh import sh
sh.echo("text").execute()
sh("python2.7")("script.py").execute()
sh.echo("text") returns a Process instance which holds all arguments and state of the process which will be executed.
A process is not executed automatically by default when a Process object is created. This is done so to support Piping and Iterating over output. But if you want just simply run commands, you may use _defer = False option:
from psh import sh
sh.service("httpd", "start", _defer = False)
In this case service httpd start will be executed immediately and sh.service(...) call will return only when the process will be terminated. If you want to always run processes immediately, you may set _defer = False as default (see Setting default process options).
Commands support short-form (-a) and long-form (--arg) arguments as keyword arguments:
sh.useradd("ftp", m = True, system = True, shell = "/usr/sbin/nologin")
which is equal to:
sh.useradd("-m", "--system", "--shell", "/usr/sbin/nologin", "ftp")
where both resolve to:
useradd -m --system --shell /usr/sbin/nologin ftp
Shell-style piping is performed using Process object composition. Just pass one command as the input to another, and psh will create a pipe between the two:
process = sh.du() | sh.sort("-nr") | sh.head("-n", 3)
process.execute()
process.stdout()
In this case process.stdout() will return output of du | sort -nr | head -n 3.
Note
You can’t execute a pipe as in the following example because of Python’s evaluation order:
sh.du() | sh.sort("-nr") | sh.head("-n", 3).execute()
You may do this by storing a pipe in a variable:
process = sh.du() | sh.sort("-nr") | sh.head("-n", 3)
process.execute()
or just:
( sh.du() | sh.sort("-nr") | sh.head("-n", 3) ).execute()
psh can redirect the standard input, output and error streams:
# echo text > /dev/null 2>&1
sh.echo("text", _stdout = psh.DEVNULL, _stderr = psh.STDOUT)
# echo -n "text" | cat
sh.cat(_stdin = "text")
# cat < file
sh.cat(_stdin = psh.File("file"))
or even use Python’s generators as input:
# Output: "0\n1\n2\n3\n4\n"
sh.cat(_stdin = ( str(i) + "\n" for i in xrange(0, 5) ))
Normal processes exit with exit code 0. Process’ exit code can be obtained through status():
assert sh.true().execute().status() == 0
If a process terminates with a nonzero exit code, an exception is raised.
Some programs return nonzero exit codes even though they succeed. If you know which codes a program might returns and you don’t want to deal with doing no-op exception handling, you can use the _ok_statuses option:
sh.mount() | sh.egrep("^/dev/", _ok_statuses = [ 0, 1 ]) | sh.sort()
This means that the grep command will not generate an exception if the process exit with 0 or 1 exit code.
Note
Please notice that if you connect a few processes in a pipe, an exception will be raised even if a failed command is not the last command in the pipe. This gives you a great power of controlling process execution in a very easy way which is not available in the shell.
As you saw above, you can control process execution via options passed to the Process instance, such as _defer = False. But sometimes you may realize that the default option values is not very suitable for you and you override them almost in every command.
For example, you want all commands to be executed immediately saving their original input and output file descriptors. You can do this by overriding the default option values for the specific command:
from psh import Program, STDIN, STDOUT, STDERR
ssh = Program("ssh", "user@host", _stdin = STDIN, _stdout = STDOUT, _stderr = STDERR, _defer = False)
# Immediatly executes `ssh user@host df -h` preserving the original
# standart file descriptors.
ssh("df", "-h")
or you can override them for all commands you execute:
from psh import Sh, STDIN, STDOUT, STDERR
sh = Sh(_stdin = STDIN, _stdout = STDOUT, _stderr = STDERR, _defer = False)
# Immediatly executes `ssh user@host df -h` preserving the original
# standart file descriptors.
sh.ssh("user@host", "df", "-h")
You can use with statement on Process objects to guarantee that the process will be wait()‘ed when you leave the with context, which also frees all opened file descriptors and other resources (see Process reference for details).
Using with context with Process objects is the same as with all other Python’s objects:
from psh import sh
with sh.mount() as process:
process.execute(wait = False)
# do some task here
# process will be terminated here
You can iterate over process output as well you do for all Python’s file objects:
from psh import sh
with sh.cat("/var/log/messages") as cat:
for line in cat:
print line,
The process is automatically executed when iteration is initiated.
Note
You should always iterate over process output inside a with context (see Process reference for description why).
When you need to run a specific command on a remote host you have to run ssh and pass commands to it as arguments which breaks the all idea of creating and piping processes with psh. For this reason psh gives you a way to run processes on a remote host in the same way you use for the local host. The only thing you have to do is to run a remote shell process (ssh, pdsh, etc.) with _shell = True option and pass a Process object as an argument to it:
import re
from psh import sh
# ssh myserver.com 'df | egrep "^/dev/"'
with sh.ssh("myserver.com", sh.df() | sh.egrep("^/dev/"), _shell = True) as ssh:
for line in ssh:
match = re.search(r"^(/dev/[^\s+]+)\s+(?:[^\s]+\s+){3}(\d+)%\s+(/.*)$", line.rstrip("\n"))
device, used, mount_point = match.groups()
if int(used) > 80:
print "{0} ({1}) ran out of disk space ({2}%)".format(device, mount_point, used)
When _shell = True option is passed, all Process instances that you specified as arguments will be converted to a shell script, which is equal to the passed command, and ssh will execute it on the remote side.
For simple commands a generated script will be quite expectable. For example, sh.ssh("host", sh.echo("text", _stderr = psh.STDOUT), _shell = True) executes ssh host 'echo text 2>&1', but for piped commands the script will be more complex. For example, the sh.ssh("myserver.com", sh.df() | sh.egrep("^/dev/"), _shell = True) executes something like bash -c 'df | egrep '"'"'^/dev/'"'"'; statuses=(${PIPESTATUS[@]}); case ${statuses[0]} in 0);; *) exit 128;; esac; exit ${statuses[1]};' on myserver.com host. This complexity is required to detect errors in processes in the middle of the pipe.
Note
At this time _shell = True supports only basic I/O redirections such as >&2, < file, 2>&1, etc (see I/O redirection). Using other redirections causes an exception to be raised.
Note
Please note that there is a little difference in executing
sh.echo("data") | sh.grep("text") | sh.wc("-l")
and
ssh("host", sh.echo("data") | sh.grep("text") | sh.wc("-l"), _shell = True)
Both commands will raise ExecutionError, but for the first one ExecutionError.status() will return 1 from the failed grep command and for the second one ExecutionError.status() will return 128.
This is because there is no way to pass a pair “failed command, return status code” from within ssh without making the generated script ridiculously complex. So all error codes of all processes in the pipe except the last one is converted to 128.
Please read the Module reference which explains some important details, thread-safety guaranties and additional features.