Know Your Tools

Us old timers love to share our knowledge. We taught ourselves through trial and error. Extensive googling, being ‘taught’ by hackers to type sudo followed by rm -rf / followed by reinstalling whole 8 or so CDs of SUSE Linux…(yes reader, I really fell for that). We learnt our skills the hard way.

Fast-forward a few years, and I see junior devs struggling to perform basic tasks. They’d come out of uni knowing f… all, having never executed a basic command in a terminal (or even opened one for that matter). So you teach them the basics - the standard Git, Rake, Gradle, Cocoa Pods, Kitchen Plan… What can they quickly learn themselves beyond that, though?

To make things easy, I’m going to present a series of walkthroughs. All the useful tools and tricks every effective software engineer should master. Let’s start with the basics i.e the terminal

Episode 1: Terminal & Bash

What if I told you that terminal is not just this black/(white if you don’t change the colour) window into the entrails of an *nix box that hackers use, but contains a proper, programming language? Mind=blown.

Terminal by default runs something called Bash, which is short for bourne again shell. Read more about its history here. here.

What the article doesn’t tell you is that Bash is a proper programming language. (They will claim it’s a scripting language but it’s all the same thing really). Bash is Turing complete. It allows for variables, functions, control statements and and you can pretty much build anything with it (whether you should build stuff with it is a different question altogether).

It operates on plain text files. Text goes in, text goes out. Simple as that. One could even claim that since everything operates as text, you can use it in a functional programming manner as well, piping output of a command (which behaves kind of like a function) into another command and then on and on and on, without any side effects. Functional programming by definition.

Mind=blown. Again.

Proper (mis)use of Bash.

The real power of a scripting tool comes from making our lives easier by automating repetitive tasks. After all, we’re in the business of making people work less, so tendency towards laziness is one of the first signs of a good software engineer. Automation allows us to be lazy. Automation is good.

So, what really happens when you type ls -al and press enter?

You’re telling your terminal to execute the program called ls, passing in the flags a and l, where a means include hidden files (the ones beginning with a dot .), and l gives us more details on these files, such as their sizes and permissions set for them. More on that later.

The output I get is something like:

Serenity:zan blog$ ls -al
total 144
drwxr-xr-x  26 zan  staff    884  4 Oct 14:10 .
drwxr-xr-x  61 zan  staff   2074 31 Jan 09:40 ..
drwxr-xr-x  14 zan  staff    476 12 Feb 07:36 .git
-rw-r--r--   1 zan  staff     13  4 Oct 13:45 .gitignore
-rw-r--r--   1 zan  staff     13  4 Oct 13:45 .ruby-gemset
-rw-r--r--   1 zan  staff     11  4 Oct 13:45 .ruby-version
-rw-r--r--   1 zan  staff     33  4 Oct 14:10 CNAME
-rw-r--r--   1 zan  staff     48  4 Oct 13:45 Gemfile
-rw-r--r--   1 zan  staff   2777  4 Oct 13:45 Gemfile.lock
...

Also worth noting is that pretty much all programs will output by default back to the command line in text format.

Note that programs also take their input in the same plain text format… if only you could chain them, like you can chain functions in programming languages…

Something like that:

Serenity:zan blog$ ls -al | grep ruby
-rw-r--r--   1 zan  staff     13  4 Oct 13:45 .ruby-gemset
-rw-r--r--   1 zan  staff     11  4 Oct 13:45 .ruby-version

the | character, also called a pipe, allows you to rout the output of the program on the left into a program to the right of it, in our case that’s the grep tool, that grabs a regular expression (ruby) and prints all lines containing that match in the input stream.

Trust the man, man

For man can tell you great many things… about other things. You query him using man followed by almost any program in unix, like this:


Serenity:zan blog$ man grep

GREP(1)                   BSD General Commands Manual                  GREP(1)

NAME
     grep, egrep, fgrep, zgrep, zegrep, zfgrep -- file pattern searcher

SYNOPSIS
     grep [-abcdDEFGHhIiJLlmnOopqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
          [-e pattern] [-f file] [--binary-files=value] [--color[=when]]
          [--colour[=when]] [--context[=num]] [--label] [--line-buffered]
          [--null] [pattern] [file ...]

DESCRIPTION
     The grep utility searches any given input files, selecting lines that
     match one or more patterns.  By default, a pattern matches an input line
     if the regular expression (RE) in the pattern matches the input line
     without its trailing newline.  An empty expression matches every line.
     Each input line that matches at least one of the patterns is written to
     the standard output.
...

Vim (and most importantly, how to exit it)

I've been using Vim for about 2 years now, mostly because I can't figure out how to exit it.— I Am Devloper (@iamdevloper) February 17, 2014

Fact of life is, as you spend time in the terminal, you’re probably going to launch Vim at some point, whether willingly or not.

You quit it by typing :wq. This will save the current file and quit. To quit without saving you just press :q!. For all other intents and purposes, just friggin use nano. Or sublime.

Where do the commands come from?

Serenity:zan blog$ which grep
/usr/bin/grep

The which tool tells you where the program is located, namely in the /usr/bin directory. Bash found it by exploring the $PATH environmental variable, and going through all and any folders specified in it, searching for the command. If it’s found, it’s going to use it, if not, it’s going to say something like this:

Serenity:zan blog$ snafu
bash: snafu: command not found

So what goes into your $PATH?

Serenity:zan blog$ echo $PATH
/Users/zan/.rvm/gems/ruby-2.1.2@zmarkan-blog/bin:/Users/zan/.rvm/gems/ruby-2.1.2@global/bin:/Users/zan/.rvm/rubies/ruby-2.1.2/bin:/Users/zan/.rvm/bin:/usr/local/heroku/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Users/zan/bin:/Users/zan/dev/sdks/android_sdk/platform-tools:/Users/zan/dev/sdks/android_sdk/tools:/Users/zan/dev/sdks/scala/bin

The answer is - whatever the heck you want. In my machine, I’ve got rubies, heroku toolchain, Android SDK, Scala SDK, alongside several other bin directories containing most of other programs/commands.

How do I add stuff to my $PATH? You call something like export PATH=$PATH:{/your/directory/here}. Boom, job done. $PATH now contains your directory, appended to it… until the end of your terminal session - i.e. you close the terminal window or tab. More on how to persist your variables later.

Another thing on these environment variables, they are accessible from pretty much everywhere, in a Ruby context, you’d get their values like this: ENV["VARIABLE_NAME"], in Java like System.getEnv("VARIABLE_NAME"); and there is a way to do it in just about any high level programming language.

Saving the same stuff every single time is lame.

So how do I keep my directories in my $PATH even if I close my terminal tab? That’s done by setting up bash profile scripts.

They are located in you home directory and named either ~/.bash_profile or ~/.bashrc. On a Mac, you’re probably more interested in the former, and on Linux machines in the latter. Read this for a further explanation.

So your ~/.bash_profile is a bash script (convenient, eh?) that gets executed when a tab is opened. That’s where you set your $PATH and other things so that you never lose them again.

Other things you might want to put into your profile script are other variables, such as $JAVA_HOME, $SCALA_HOME, and settings for terminal colours, as well as scripts like git completion and the like.

My colours are set like this:

export CLICOLOR=1
export TERM=xterm-color
export LSCOLORS=ExFxCxDxBxegedabagacad

and that’s what I have for my prompt:

PS1='\h:\u \W$ '

giving me the following output (hostname, user, current directory):

Serenity:zan blog

Users, Super Users and H4x0r5

Most of the time when in the terminal you’re logged in as your own user. In my case, that’s the zan in the prompt.

As that user, you’re confined to changing stuff in your own home directory and its subdirectories.

On a mac, it’s this one:

Serenity:zan blog$ cd ~
Serenity:zan ~$ pwd
/Users/zan

(Here, the tilde ~ character is a shortcut to your home directory)

But what if I want to change something in other directories? Like add a program to the /bin or /usr/bin directories?

Serenity:zan ~$ touch myfile
Serenity:zan ~$ cp myfile /usr/bin/
cp: /usr/bin/myfile: Permission denied

A quick google will tell you to just sudo it. So let’s do it.

Serenity:zan ~$ sudo cp myfile /usr/bin/
Password:
Serenity:zan ~$

That’s basically telling the computer to Shut Up and DO and I didn’t just make that up. Really, sudo allows any user to use their own password to perform any system wide commands. (On Mac, if you have administrator account on the computer, on Linux if you are listed in the sudoers file).

Aliases and links

These are technically different things but still kind of work in a similar way.

First the aliases, as they most often allow you to be lazy. Tired of writing git checkout branchname? Specify an alias for it, such as aliast gco='git checkout' and here you go, you just saved 9 characters. In an average month of work, that’s 27 minutes saved. Incidentally, 27 minutes is also the average time needed to finish a pint of beer (stats I have totally not just made up)

Next are links, they are pointers to a location on disk. We have 2 types of them, symlinks and hard links. Symlinks, or symbolic links are basically shortcuts to any file or folder. I most often use them to create links to my projects in my home dir, the syntax is like this:

ln -s ~/dev/deep/project/hierarchy/project_dir ~/project

So that now, I can just open up a terminal tab in my home dir, and cd project to skip all that file system stuff. Handy.

Symlinks are pointing to a specific path in the file system, nothing else. Hard links, on the other hand, point to a file itself, regardless of where it actually is on the file system, so when you create one, you can move the target file around, and the hard link will still know where to go.

You create them the same as you’d create a symlink, without the -s flag.

The limitations of the hard links are that they can only point to a location on the same file system, and only to a file (no folders).

This stack thread gives a nice explanation about links.

Other niceties

TAB key autocompletes. Try it. It’s good.
!! repeat the last command. Plays nice with sudo.
control-r lets you search for the last command.
source ~/bash.profile reloads bash profile in your current terminal tab. Cool when you’re editing your bash profile script.
$EDITOR environmental variable allows you to specify something else than vim as default editor.