Knowing what these are, and how to wield them will make you a top 1% Bash scriptoor in the world. Trust me.

  • : - no-op command
  • ${} - variable expansion
  • $() - process substitution
  • $(()) - arithmetic expansion
  • (( )) - arithmetic thingy
  • <<<, << - herestring and heredoc
  • <, >, <> - stdin, stdout redirection
  • <() - process substitution

<<< - herestring

A way to feed a string or a var into a command’s stdin. Works only on commands that can accept input via stdin.

Instead of this:
echo "hello world" | sed 's/hello/bye/'

you can do this:
sed 's/hello/bye/' <<< "hello world"

both will output bye world.

but why prefer this over pipe?
pipe will create a subshell. Some times that’s not preferrable.

Like when you’ll be wanting to assign a variable in a while loop that’s after the pipe. But the assignment won’t have effect. Use herestring in that situation.

Example:

cat "$teams" | while IFS="|" read -r team_id team_name; do
  team_to_id["${team_name}"]="${team_id}"
done

This won’t work as expected. The variable assignment within the loop won’t persist because the while cmd runs in a subshell because that’s what pipe does.

So you should do this instead, using the herestring:

while IFS="|" read -r team_id team_name; do
  team_to_id["${team_name}"]="${team_id}"
done <<< "$teams"

How to remember this? Easy if you think of this as a special case of heredoc.
It’s a single-line heredoc. Hence why it’s called herestring.

<< - heredoc

The infamous heredoc! Well okay, but what does it do?

The same as the herestring above. It’s a way to feed a multi-line string into stdin for a command.

Simple usecase: Have a multiline string into a variable:

sql=$(cat <<SQL
select *
  from tablename
;
SQL
)

There are variations. The one I care about is the quoted one. The one above allows bash variable expansion. Inside, you can use things like ${}, $(). But if you don’t want that.. and want them just as a literals.. then quote the top delimiter:

sql=$(cat <<'SQL'
select *
  from tablename
;
SQL
)

Here’s a weirdo heredoc. This writes a multiline string into a file.
I think this would be useful in server automation scripts:

cat <<EOF > config.txt
host=localhost
port=5432
EOF

${} - variable expansion

The simple variable interpolation thingy! Except it’s called variable expansion, and can do a lot of things.

# replace first occurrence:
str="hello daddy hello"
echo ${str/hello/bye} # bye daddy hello

# replace all occurrence:
echo ${str//hello/bye} # bye daddy bye

# get length of a string:
echo ${#str} # 17

# use a default value if the var is unset
echo ${uname:-joe} # outputs joe if uname is unset, otherwise joe
echo ${uname:=joe} # outputs joe if uname is unset, and sets uname to joe

# case change
x=blah
echo ${x^^} # BLAH
echo ${x^} # Blah
echo ${x,,} # blah
echo ${x,} # bLAH

# extract substring
x="prasanna"
echo ${x:4:4} # anna

: - The no-op command

This command succeeds always. It produces nothing to stdout or stderr. It can take args, read from stdin and you can pipe to it.

You can use it for many things.

To fail the script with an error message when a variable is not set or passed to it:

: "${psql_username:? psql_username is not set}"

If the var is not set in the previous lines, or not passed to the script as env vars while executing (psql_username="postgres" ./myscript.sh), then this line will fail the script. It’s compact than testing with [[ -z psql_username ]] and echoing the error msg and exit 1ing.

Another use would be to use it inside empty branch in a conditional to make code readability better.

Yet another use: Use it as multi-line comments. Bash doesn’t have it. But you can do this:

: <<CMT
if [[ $VERBOSE ]]; then
  echo "verbose"
else
  LOG_FILE=/dev/null
fi
CMT

$() - command substitution

Run any command inside it, and its stdout will be captured and replaced in its place. That means you can capture it into a variable, or use directly in a print statement.

today=$(date +%Y-%m-%d)
echo "Today is $today"

result=$(ls -la)
echo "$result"

echo "You are in $(pwd)"

$(()) - arithmetic expansion

Do math in bash with these! Refer variables inside without the $ prefix! But do so with only integers! It fucks up floats!

You can use this to capture the result and assign it to a variable:

a=4
b=5
c=$((a+b))
echo $c # 9

(( )) thingy

It’s $(())’s cousin. Arithmetic family. But is used in conditionals and to perform arithmetic calculations where the result is not needed right away.

if (( x > 3 )); then echo "big"; fi

x=0
((x++))
((x += 5))
echo $x

Can also be used to run a loop n times:

for ((i = 0; i < 5; i++)); do
    echo $i
done

<() and >() - Process substitution (read and write)

If a command needs only a file as its argument, but you only have the content in stdout, then you can use <():

diff <(sort a.txt) <(sort b.txt)

I don’t care about the other thing: >().

< and > - Input and Output redirection

psql -U user -d dbname < some.sql

This feeds the contents of the sql file as stdin to psql command. It’s as if the commands were typed interactively one line at a time. Use it for cmds/programs that don’t take command line arguments, but only read from stdin.

stdout and stderr redirection

know the file descriptors first:

  • 0 is stdin
  • 1 is stdout
  • 2 is stderr

Now you can do magic:

  • stdout to file: cmd > out.log
  • stderr to file: cmd 2> out.log
  • stdout, stderr to same file (posix): cmd > out.log 2>&1
  • stdout, stderr to same file (bash): cmd &> out.log
  • silence stdout: cmd > /dev/null
  • silence errors: cmd 2> /dev/null
  • silence both (posix): cmd > /dev/null 2>&1
  • silence both (bash): cmd &> /dev/null
  • append with >> on all above

tee is a cmd that reads from stdin and writes to both stdout and to a file given to it as arg.

We can use tee with above commands, to both send the std streams to terminal, as well as to a file.

./deploy.sh | tee "deploy-$(date +%F).log"

This will show the deploy output in the terminal, as well as log it in a file.

You can even pipe it to yet another cmd, which will read from stdin the tee’s output:

pg_dump mydb | tee backup.sql | gzip > backup.sql.gz

Here’s a simple cmd to understand tee:

# tee's stdin is fed through pipe
echo "one two three" | tee digits.log

# same as above, but stdin is fed through herestring `<<<`
tee words.log <<< "aa bb cc"