Sunday, November 28, 2010

Xargs

Xargs -- a versatile UNIX command that allows you to build and execute command lines from standard input.  Think of it as a slightly more efficient way of processing lists of standard input that certain commands like find have trouble dealing with.  As noted in Wikipedia,


rm /path/*
or
rm `find /path -type f`
will fail with an error message of "Argument list too long" if there are too many files in /path. However this version (functionally equivalent to rm `find /path -type f`) will not fail:
find /path -type f -print0 | xargs -0 rm

I have recently used it to process a list of files, changing the permission of each file to be read-write for the user.

ls /home/user/*.txt | xargs chmod 600

I have also used it in conjunction with Cucumber.  In this particular scenario, I needed some way of producing a single text file that contained all of the features and scenarios, so that I could go over them with a team of developers, testers and business analysts.  At the moment, Cucumber does not support a way (or at least I have not found a method yet) of extracting all of the scenarios into one or more files.  There are some tools that are actively being developed that should alleviate this issue in the future (see Relish and Cucumber-In-The-Yard), but for now, I leveraged xargs.

find features/ -name *.feature -type f -print0 | xargs -0 cat > features_and_scenarios.txt

This finds all files that end with .feature and prints the results to standard output.  It then takes the output and pipes it into the xargs command, which runs it through the cat command and finally writes the output of the cat command to the features_and_scenarios.txt file.

For more information on the xargs command, have a look at the man page.

2 comments:

Anonymous said...

Using xargs without -0 can lead to surprising results because of the separator problem http://en.wikipedia.org/wiki/Xargs#The_separator_problem

Consider using GNU Parallel instead.

Watch the intro video: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Terminal Velocity said...

Thanks for the information. I was not aware of the separator problem. I will be installing GNU parallel.