The Command-Line Really Does Matter, Part 2

09:56 reading time

In Part 1 of this series, we explored how control characters can get in your way and what to do about them.

Now let’s have a look at loops and some interesting system limits. Loops are one of those features that give you awesome power.

Getting Loopy

Consider the situation where you have a large number of files in a directory, and you’d like to perform an operation on each one.

You have many options; here are a few:

  1. Use a for loop with a backtick operation
  2. Use a while loop that reads standard input for each file name
  3. Use the find command with -exec

The third variant is discussed later. For now, let’s just review the first two.

The first variant might look like:

for filename in `ls *`
  echo "A file called $filename"

This executes the list command first, loading its entire output into memory, and then evaluates each item (delimited by $IFS) as an argument to the for loop.

The second variant might resemble:

ls * | while read filename
  echo "A file called $filename"

This version effectively iterates over the output of the list command as it is executing; we don’t wait for it to finish before we start the loop.

We might therefore assume that the first variant may take longer to run or cause us to potentially bump into a system-level resource limit.

Is it true? We can simulate a large number of inputs to demonstrate differences in execution times and other repercussions.

Assuming we have GNU coreutils installed, we can use the seq command to produce a sequence of numbers as output.

(Aside: Modern bourne shell variants like ksh and bash include a built-in range feature (the {a..b} expression), but using this would not accurately reproduce the wait time on another process.)

~$ time (for n in `seq 500000`; do :; done)

real  0m2.096s
user  0m2.065s
sys 0m0.049s

This evaluates the seq command and then iterates over each item as an argument to the loop. The colon in the “do” block is a builtin command equivalent to true, acting as a no-op. Finally, we wrap this all up in a subprocess that we can measure with the time builtin. It takes about two seconds on my machine to complete.

How does the second variant stack up?

~$ time (seq 500000 | while read n; do :; done)

real  0m8.251s
user  0m5.540s
sys 0m4.620s

Curiously, reading from a pipe with the bourne again shell (“bash”) takes significantly longer. What about our memory footprint? On OS X, the -l flag to the time command (not the bash builtin) will show us all sorts of extra details, including memory usage.

~$ cat << END >
for n in \`seq 500000\`; do :; done
~$ (/usr/bin/time -l bash 2>&1 | grep resident
  32739328  maximum resident set size

This for-loop-with-backticks variant seems pretty memory-hungry, especially when compared to the while-loop-with-pipe version:

~$ cat << END >
seq 500000 | while read n; do :; done
~$ (/usr/bin/time -l bash 2>&1 | grep resident
    987136  maximum resident set size

… leaving us without a clear winner, as both strategies have their drawbacks. The second variant also increases script complexity if we want to use read within the body of loop, as we would need to use exec and possibly tee to help us alias and reroute streams.

An aside…

The motivation for this loop-strategy comparison arose from the varying command-line argument limits on different systems. For example, getconf ARG_MAX on OS X 10.9 yields 262144, while this very same limit on Debian 6 (AMD64) is considerably higher at 2097152.

Passing more than 262144 arguments to a command in OS X, regardless of the shell used, should cause an error:

~$ /bin/echo `seq 500000` | tr ' ' '\n' | tail -n 1
-bash: /bin/echo: Argument list too long

Note that the builtin echo for bash is not affected by this particular limit!

~$ echo `seq 500000` | tr ' ' '\n' | tail -n 1

Interestingly enough, the argument-length limit is even stricter than this particular system property in real life—at least on OS X.

That leads us to another fun limit.

The Shebang limit

Executable command-line scripts start with a “shebang line.” This informs the system what executable binary should be used to interpret the file.

An executable bash script, for example, could be created and executed like so:

~$ cat << END >
echo hello world from process ID \$\$
~$ chmod +x
~$ ./
hello world from process ID (some number)

The “shebang line”, in this case, is the line at the very top of the file, “#!/bin/bash.”

We could also run the script by just invoking it with bash, or, we could run it within the current shell process by using the source builtin (a.k.a. the .), i.e.:

~$ echo $$
~$ .
hello world from process ID 2845
~$ source
hello world from process ID 2845
~$ bash
hello world from process ID 63703
~$ ./
hello world from process ID 63704

The first two invocations of the script show the same process ID is being used; that is, the same shell process that we’re using interactively is also responsible for running the contents of the given “” file.

The third invocation illustrates running the “” in a separate process, but not directly as an executable. The first line is interpreted by the new bash process as a comment.

The fourth invocation shows us running “” as an “actual” executable, where our current shell process hands over control to the program listed at the top of the file; in this case, it’s bash.

We were “allowed” to run the script as an executable because we gave it executable permissions using the chmod command.

Now, depending on your operating system, you may have a limit to how long your shebang line can be.

Consider the path ////bin/bash. This evaluates to /bin/bash. Therefore, a shebang line with ////bin/bash will run the file using a new bash process:

echo hello world

Now consider a shebang line with 117 leading slashes. On OS X, we get a “hello world.” Not on CentOS 6!

~$ ./
-bash: ./ /////////////////////////////////////////////
/////////////////////////////////: bad interpreter: Permission denied

Our shebang line is 128 characters long, exceeding the limit of 127 characters as defined in the Linux kernel source. In the binfmts.h file, the BINPRM_BUF_SIZE is set to 128, which includes the C-string terminating null character.

Effectively, the desired interpreter, /bin/bash, is never read, because the shebang line is truncated prior to evaluation. Instead, Linux tries executing the / directory as the interpreter. While the root directory has the executable mode set, it’s not an executable file, and we receive a “bad interpreter” message. We could recompile the kernel with a larger limit, or find a way to shorten the length of our shebang line.

Our example of 117 leading slashes is clearly contrived, but now consider a heavily-nested deploy root that includes domain names, user names, branch names, etc. Each subtree contains run-time scripts nested somewhere deep down in there, each invoked as executables that contain the full path to their sandboxed interpreter as the shebang line. You might find yourself with an unanticipated real-world problem. (We did!)

Perhaps the interpreter gets chopped off entirely (as illustrated above), or a portion of the interpreter is chopped off, or arguments passed to the interpreter are excluded. This latter situation might be the most frustrating, as it won’t be obvious why the interpreter isn’t behaving in the desired way. (Consider #!/.../bin/bash -x where only the -x is chopped off. The diagnostic output you were expecting is missing, but the program otherwise executes as desired.)

More fun in Part 3!


Ian Melnick
Senior Software Engineer