Working On The Terminal
Getting comfortable with working on a terminal (also knows as a command line prompt or a shell) is quite a valuable skill for any software professional. First of all, it's more practical to have everything in one place, rather than switching windows (and visual context) all the time.
In addition, a lot of the terminal tools and commands are consistent across different distros, operating systems or versions. Most of them are already pre-installed, and change very little with time; some work the same way they used to 30-40 years ago. And, in cases such as logging into a headless server, the shell will be your only option!
I spend a big deal of my time on the terminal, so I thought I'd share a few of the things I do there, and the tools I use. I am by no means a terminal ninja though; I've seen people (mainly sys admins) do some crazy stuff there. Also, this article is not a tutorial at anything; there's plenty of material online that does a good job at that. It'll be more of a general overview based on my daily development routine. I don't intend to cover the most basics commands either (such as cd, mv, ls, mkdir, rm, cp, cat, ps and so on) as I assume most people reading this are already familiar with them and there's not much interest going over them.
Let's start with bash, which stands for Bourne Again Shell. Shell is a generic name for any program that gives the user a text-based interface to interact with the computer. You basically type a command and get an output on the screen. Many shells though have scripting abilities and/or additional programming constructs similar to that of a general-purpose programming language.
On most Unix/Linux systems, multiple shells are available: bash, csh, ksh, zsh just to name a few. They serve more or less the same purpose, but do differ in certain aspects, such as the complexity and capabilities of the scripting language.
Bash is the default shell on many Linux distros, and it's the only one I've ever used. I haven't dwelt in other shells, because it covers all my needs. Bash offers features such as redirection of standard input and output, piping of commands, multi-tasking with background jobs, and a powerful scripting language. You can tweak it to behave in certain ways by modifying the .bashrc file, which makes it easy to transfer the same setup to another machine: all you have to do is transfer this file and source it. This is a
recurring theme in the Unixverse by the way. Look up the term dotfiles.
Note: for the rest of this article, when I say shell or terminal, I'm talking about bash.
Next one is vim. Vim is a very powerful text editor. While you can use a simpler and more intuitive one (such as nano) for quick editing, if you find yourself spending a lot of time editing text on the terminal, vim will massively enhance your productivity. It does have a steep learning curve, but that's because you can do so much with it.
Vim has a large community of users and a rich ecosystem of plugins. You can transfer the .vimrc file and hit the plugin install command, and in seconds you replicate the same setup in another machine. I must say that vim is still just a text editor, not an IDE. While you can make it "IDE-like" by installing plugins, you might be better off sticking to an actual IDE (i.e. VS Code) for some parts of the development process. The good thing is, most modern IDEs offer a vim plugin on their own, so your vim skills are easily transferable. But on the terminal, vim remains unbeaten, and there's nothing the emacs gang can do about it.
Then there's package managers. One of the most common activities on the terminal is installing, updating or deleting packages. Therefore, knowing some basic commands from the package manager of your OS can be very handy.
The one I use is apt, which stands for Advanced Package Tool and is a popular package manager in the Debian world. Don't let the name fool you though; apt is a trimmed down version of apt-get, keeping only its basic and most used features. It has less command-line options, because it is supposed to be more user friendly to the average user.
While apt-get is not being deprecated, and in fact you'll still have to resort to it for more complex use cases, apt should be good enough for most of your needs. Whatever your package manager of choice is though, it's a good idea to get comfortable with using it.
Let's talk about git on the command line. Most IDEs will have a simplified visual interface to perform common git operations, but for various reasons, I prefer to interact with git strictly through the terminal.
The first one being, I don't like to leave the terminal unless I have to. If I'm already editing code on the terminal, why not stage, commit, and push it also from the terminal ? But probably a bigger reason is that from the terminal you can utilize git's whole spectrum of power. There's only so many buttons you can put on a GUI!
I find it easier to work with git logs, diffs and merges from the terminal. Furthermore, the git command can be piped with other commands. In addition, git commands usually are followed by a very descriptive output message, which could get lost in the GUI implementations.
And lastly, you really get used to working with these commands. Keep in mind that IDEs change! They change from one to another, and they change between different versions of the same IDE. The commands rarely ever change.
Another really useful tool is curl, which stands for Client URL. Curl is used to transfer data from and to servers.
It can be used to perform user authentication, HTTP POST (or other methods), SSL connection, FTP upload and what not. I mostly find myself using it to test REST endpoints. At its most basic usage, all you have to do to get a response is write curl followed by the endpoint URL. But it supports all sorts of switches.
And sometimes curl is the most practical way to get a package, if your package manager's repositories don't have it and you have to hit the creator's website endpoint directly.
The find command is used to traverse the directory tree from a given starting point to every leaf. Passing in just the -print switch will print every file name starting from the cwd.
You can already imagine how useful this becomes when piped with other commands. Very often it is piped with grep, which we'll look at later on. But find on its own supports many switches. The following example:
find / -name passwd 2> /dev/null
searches for all the files starting from the root directory, whose name is passwd, and suppresses the error output. The path /dev/null is a special place used when we want to throw things away. Something to note here is that the -name switch will look for complete words, which is why the output is often piped to grep which can also search for substrings.
Speaking of grep, one of the most popular commands..it stands for General RegEx Parser and as the name suggests, it is a command used to search for string patterns in its input.
Its output are all the lines from the input which match the pattern. The following example:
grep julian /etc/passwd
will print the line in the passwd file which contains my name. You could also just list the files which have a match, instead of printing the actual lines that contain
the match. As is the case with many commands, that's just a single switch away. If you use grep without specifying files to search, it will expect its input from the standard input, like in the following example:
ls -l / | grep '^d.......w'
Here we're listing all directories under root which have write access for the Other permission class. Something to keep in mind here is that grep by default uses the POSIX standard of regular expressions and not the Perl-compatible standard (like Python does), but that can be changed with a simple switch.
If we want to edit text in a file, we can easily do it with a text editor. But if the text we want to edit only exists in a pipe between 2 commands, we need to use sed.
For obvious reasons, sed does not offer the entire editing capabilities of a text editor. It can take files as input, but it usually receives it from standard input. And when it does take files, sed by default will just return the modified text and not change the file at hand, although you can enforce the change using a switch.
Tools like find, grep and sed might not seem like much on their own, but when piped together, they really turn into a powerful chain of commands. Since we saw an example with the other 2, here is one with sed:
sed "s/c/C/g" my_file.txt
The first s stands for substitute, and the g in the end for global, meaning every instance. So we'd be replacing every occurrence of a lowercase c with an uppercase C in that file.
Another popular command is ssh, which stands for Secure Shell. This is actually a communication protocol used to log into remote machines in a secure way, encrypting all the traffic. But it's also a command with the same name.
You can't simply connect to any remote server using ssh; the server must have the SSHD (ssh Deamon) set up and running, listening for ssh connection attempts. But usually the server already has it set up, either by the cloud provider or the OS itself (i.e. Ubuntu Server).
The default way to connect to a server is using a password. For example I'd hit:
ssh julian@192.168.1.25
where julian is my username on the server, followed by the server IP. After that I'd be prompted for the password, and if I enter the right one, I'd get shell access to the server.
A better, that is safer and more practical, way to log in is using a ssh key pair instead of a password. You generate a pair of keys once, copy the public key to the server, and disable password login altogether.
Now I'm not gonna cover the steps for that, because this is not a tutorial, but it's a pretty straightforward process. By disabling password login, you secure yourself from a whole class of cyber attacks.
I use ssh a lot to log into VPSes. While we're on the topic, if it's a new server, it's a good idea to create a new user with your name and su privileges, and disable root user login as well. All these configs are made in the /etc/ssh/sshd_config file.
Alright, this post is getting a bit long, so let's quickly mention some other tools:
- df to see the amount of free and used disk space on a per filesystem basis
- free to see the amount of free and used main memory and swap area
- top to show how resource usage changes with time (it's like a mix of ps + free)
- tr to perform character translation (similar to y in sed but more powerful)
- whoami to show who you're logged in as
One last thing I'd like to mention is that often you'll find programming languages, database systems, cloud services and other kind of systems, provide their own shell-like interface. For example, PostgreSQL comes with the psql shell, Python has the REPL, Heroku has the Heroku CLI, Docker CLI etc.
This means more and more stuff can be done from within the terminal, and it's not a thing of the past. A rich variety of shells is an even bigger motivation to stay on the terminal and get good at it. Of course, it's not for everyone. Some people, for whatever reasons, may simply work better with graphical interfaces and there's nothing wrong with that.
But I advocate giving the terminal a shot to see where you stand on that, and I hope this post serves as an encouragement towards it.