Redirecting output

Bash provides a wide range of features for manipulating and redirecting output. Here we'll cover how to redirect output in Bash, including standard output (stdout), standard error (stderr), and how to pipe data between tools.

Input/Output streams

In command-line tools, stdin, stdout, and stderr are standard streams that are typically used to handle input, output, and error messages, respectively.

stdin is a standard input stream that is used to read data from the user or from another program: it is the input that a program receives. For example, when a program asks the user to enter data, the data is read from stdin.

stdout is a standard output stream that is used to display data to the user or to another program: it is the output that a program sends to the user or to another program. For example, when a program displays the result of a calculation, the result is sent to stdout.

stderr is a standard error stream that is used to display error messages or diagnostic messages to the user or to another program: it is the stream that a program uses to report errors, warnings, or other diagnostic information. For example, when a program encounters an error or a warning, the message is sent to stderr.

In command-line tools, these streams are usually represented by file descriptors: stdin is represented by file descriptor 0, stdout is represented by file descriptor 1, and stderr is represented by file descriptor 2. By default, the output of a command is sent to stdout, while error messages are sent to stderr.

Standard Output (stdout) redirection

Standard output is the default output channel for command-line programs, where the output is displayed on the terminal. However, sometimes it is desirable to redirect the output to a file instead. This can be done using the greater-than symbol (>) followed by the name of the file to which the output should be redirected.

For example, to redirect the output of a command to a file, use:

ls > filelist.txt

This will redirect the output of ls to filelist.txt instead of displaying it on the terminal.

This will replace the content of the target file!

It is also possible to append output to an existing file by using the double greater-than symbol (>>). For example:

ls >> filelist.txt

This will append the output of ls to the end of filelist.txt.

The output of a program can be suppressed and discarded entirely by redirecting it to the special file /dev/null. The null file accepts any amount of data, and immediately discards it.

ls > /dev/null

Standard Error (stderr) redirection

Standard error is the default channel for error messages and diagnostics, which are displayed on the terminal. However, it is also possible to redirect standard error to a file.

To redirect standard error, use the number 2 followed by the greater-than symbol (>). The number 2 represents the standard error stream.

For example, to redirect the standard error of a command to a file, use:

grep xyz somefile.txt 2> errors.txt

This will redirect any error messages generated by grep to errors.txt.

This will replace the contents of errors.txt, and just like redirecting stdout, you can use the >> operator to append, instead:

grep xyz somefile.txt 2>> errors.txt

Also just like how the stdout data can be suppressed by sending it to the /dev/null file, data on stderr can also be suppressed:

grep xyz somefile.txt 2> /dev/null

Combining outputs

The output of one stream can be combined with the output of another by using the >& option. This allows us to combine stdout and stderr for instance:

grep xyz somefile.txt 2>&1

The stdout stream has the id of 1 so we're sending stderr to stdout which will appear on the console.

Of course, the combined output can also be redirected:

grep xyz somefile.txt 2>&1 > foo.txt

This will log both the result of grep and any errors together.

Piping data between tools

One of the most powerful features of Bash is the ability to pipe data between tools. Piping allows the output of one command to be used as the input of another command, without the need for intermediate files.

To pipe output to another command, use the vertical bar (|) symbol. For example:

ls | grep test

It is possible to chain multiple commands together using pipes. For example:

ls | grep test | wc -l

This takes the output of ls, searches for test, and then uses the wc tool (word count) with the line option (-l) to count how many lines. It will print the number of files with test in the name!

These features provide a powerful and flexible way to manipulate and process data on the command line, and can help streamline many common tasks.

Incorporating other tools

Now that we can connect output from one command to another, we can use many built-in tools for data manipulation.

In the following examples, we use echo and cat filename as the source of data for demonstration purposes, but remember the beauty of piping between commands - the data can come from any other commands that output text, like ls, ps, grep, etc!

The "tr" command

The tr command in Linux is used to translate or delete characters from standard input and write to standard output. It can also be used to replace a set of characters with another set of characters.

The basic syntax of the tr command is as follows:

tr [options] set1 set2

where set1 is the set of characters to be translated, and set2 is the set of characters to translate set1 into. If set2 is shorter than set1, the extra characters in set1 are deleted from the output.

Some common options for the tr command include:

• -c: complement the set of characters in set1

• -d: delete characters in set1

• -s: squeeze repeated occurrences of characters in set1 into a single occurrence

Here are some examples of how you can use the tr command in a payload:

1. Translate all lowercase letters to uppercase:

echo "hello world" | tr '[:lower:]' '[:upper:]'

Output: HELLO WORLD

2. Delete all vowels from a string:

echo "hello world" | tr -d 'aeiou'

Output: hll wrld

3. Replace all spaces with tabs:

echo "hello world" | tr ' ' '\t'

Output: hello world

4. Complement the set of characters in a string (i.e., replace all non-alphanumeric characters with a space):

echo "hello world!" | tr -c '[:alnum:]' ' '

Output: hello world (Note the space at the end.)

The "sed" command

The sed command (short for "stream editor") is a powerful text-processing tool. It is used to manipulate, transform, and replace text in a file or a stream of text. It works by reading a file or stream line-by-line, making specified modifications, and then outputting the modified text.

The basic syntax of the sed command is as follows:

sed [options] 'command' filename

or chained with a pipe,

cat filename | sed [options] 'command'

Here are some examples of how you can use the sed command in a payload:

1. Replace a string in a file:

cat filename | sed -i 's/old_text/new_text/g' > newfile

This command will replace all occurrences of old_text with new_text in the filename file. The -i option specifies that the changes should be made in-place, meaning that the original file will be modified.

The output will be sent to newfile.

2. Delete a line from a file:

cat filename | sed -i '2d' > newfile

This command will delete the second line from the filename file. The d command is used to delete lines.

The output will be sent to newfile.

3. Insert a line before or after a matching pattern:

cat filename | sed -i '/pattern/i new_text' |
    sed -i '/pattern/a new_text' > newfile

These commands will insert new_text before or after the first occurrence of pattern in the filename file, respectively. The i command is used to insert text before a matching pattern, while the a command is used to insert text after a matching pattern.

The output will be sent to newfile.

4. Replace a string in a file only in lines matching a pattern:

cat filename | sed -i '/pattern/s/old_text/new_text/g' > newfile

This command will replace all occurrences of old_text with new_text only in the lines that match pattern in the filename file. The s command is used to substitute text, and the /pattern/ specifies that the substitution should only be made in lines that match the pattern.

The output will be sent to newfile.

These are just a few examples of how you can use the sed command in a bash script. There are many more options and commands available that you can use to manipulate text in a file or stream.

The "awk" tool

awk is another powerful text-processing tool. It is used to process and manipulate text files that are formatted in a specific way. It can extract and print specific fields, perform calculations, and search and replace text.

The basic syntax of the awk command is as follows:

awk 'pattern { action }' filename

or using pipes:

cat filename | awk 'pattern { action }'

Here are some examples of how you can use the awk command in a shell script:

1. Print specific fields from a file:

cat filename | awk '{ print $1, $3 }' > newfile

This command will print the first and third fields from the filename file, separated by a space. The fields are separated by whitespace by default.

The output will be sent to newfile.

2. Print lines matching a pattern:

cat filename | awk '/pattern/ { print }' > newfile

This command will print all lines from the filename file that match the pattern. The /pattern/ specifies the pattern to search for, and the { print } specifies the action to perform on matching lines.

The output will be sent to newfile.

3. Perform calculations:

cat filename | awk '{ total += $1 } END { print total }' > newfile

This command will add up all the numbers in the first field of the filename file and print the total at the end. The END keyword specifies that the action should be performed after all lines have been processed.

The output will be sent to newfile.

4. Search and replace text:

cat filename | awk '{ gsub("old_text", "new_text", $0); print }' > newfile

This command will search for all occurrences of old_text in each line of the filename file and replace them with new_text. The gsub function is used to globally substitute text, and $0 refers to the entire line.

The output will be sent to newfile.

These are just a few examples of how you can use the awk command in a shell script. There are many more functions and options available that you can use to manipulate text in a file or stream.

Summary

These tools provide a powerful and flexible way to manipulate and process data on the command line, and can help streamline many common tasks.

Not every command line tool uses stdout and stderr in the expected ways, but the vast majority of them do. Being able to manipulate and redirect the data streams from tools enables chaining for complex behavior and creating entirely new tools.