Unix Shells and Terminals

Is a Unix shell and a Unix terminal the same thing?

You have probably seen a Unix terminal emulator before. The image below is from the macOS Terminal application. There are many of these. My favorite is iTerm2. Linux's users may use the GNOME Terminal, or the KDE application Konsole.

Listing contents of Godot directory using Terminal application

A more generic term for these interfaces is CLI (Command Line Interface). My first exposure to a CLI was not actually a Unix system or Microsoft DOS, but on an Amiga 1000 computer running AmigaDOS in a blue CLI window.

Command line interface on Amiga OS

So, the CLI concept is quite generic, but this article will focus on Unix command line interfaces (CLI) because they dominate modern operating systems such as Linux, macOS, and even Windows. Yes, Windows has its DOS and PowerShell CLIs, but the Unix command line is also taking over Windows through the Windows Subsystem for Linux (WSL).

The Unix command-line can be confusing to get for beginners. People get confused by the difference between a shell, such as the Bourne Again Shell bash, a computer terminal, such as VT100, and a terminal emulator such as Konsole or GNOME Terminal. To add more confusion, we can talk about Pseudoterminals.

A couple of questions naturally pop up from these examples:

1. Why on earth does it have to be so complicated to have a command-line-interface?

2. Do I even need to know any of this? Can I just live in ignorant bliss?

For a long time, I did not know the nuts and bolts of the Unix command-line and was still using it without too many problems. Yet, without understanding the underlying technology, you will frequently struggle with understanding important things.

For complete beginners: Unix Command Line Crash Course

I will take an example from the Git version control system: For a long time, I tried to stick with only knowing the commands to issue. I tried to only relate to Git as a user. As someone who only knew what commands to use. That didn't work. I got extremely frustrated and was ready to give up on Git entirely. A friend of mine implored me to not give up. So, I made a last ditch effort by reading a book on Git under the hood. Suddenly, everything clicked. I actually made a talk based on this experience which helped many people who struggled with Git: Making Sense of the Git Version Control System.

The Unix command-line is much the same. By understanding the underlying concepts, you will be able to do more with the command-line.

Why is the Unix CLI So Complicated?

If you built a command-line interface from scratch today, you could make something much simpler. It would be designed for a mouse, window, and keyboard from the start. Copy-paste would on Windows and Linux work with Ctrl-C and Ctrl-V just like any other application. Hotkeys for jumping one word or line at the time would work like any other text editor. You would be able to place the cursor easily with the mouse between letters. Yet, you normally cannot do any of these things in a Unix terminal, and it is probably not obvious why these seemingly arbitrary restrictions exist.

The problem is that large amounts of Unix tools such as ls, telnet, ftp, ed, awk, sed, grep, tar, gzip and many others got made for computer hardware which no longer exists. People would rather not abandon all the software they had grown accustomed to using. They would rather not rewrite all this software from scratch. Thus, as Unix hardware and software evolved, they created various forms of emulation on top to keep existing tools such as grep and tar thinking they are still running on an old Unix system.

It is the same reason why we use Intel x86 processors today. The instruction-set architecture is quite outdated. It does not look like what we would have made if we could have designed a microprocessor from scratch. Yet, the design has not stood still. It has evolved and expanded with a lot more features to stay relevant. A modern instruction-set architecture would be closer to Arm used in M1 and M2 Macs today, or the RISC-V instruction-set.

We see the same with programming languages. C++ has turned into a monstrosity. Nobody would design anything like C++ today if given a clean slate. The language gained ground initially because developers could reuse their old C code. Later, it has remained relevant by adding features while keeping old C++ code running.

Thus, understanding any widely used technology today is equally about archaeological digging as it is about understanding engineering and program design. I used to work on a 3D modeling tools written in C++ from 1992. Frequently understanding the architecture and design could not be done by considering engineering trade-offs but by asking the old timers to give you a history lesson about the good-old-days.

It may seem like a waste of time and energy to learn, but I can promise you that understanding the history of how Unix terminals came to be, will greatly help you understand why the system works the way it does.

History of the Unix Terminal

My aim in this section is to get you to grasp how a terminal, shell and terminal emulator relate to each other.

The original Unix mainframe computers did not have electronic displays like you are used to today. They were not individual personal computers, but huge boxes meant to be used by multiple users. Computer users instead had something called a teletype on their desk. You may also have heard them referred to as a teletypewriter, teleprinter, or TTY. These machines worked like a glorified typewriter and telegraph all rolled into one. It had no memory, microprocessors, or anything like that. It was primarily an electro-mechanical device. You typed letters on it that showed up on paper like a real typewriter, so you could see what you were typing. The key difference is that the letters you typed got send over a serial cable (RS-232) to a Unix computer.

Linux running on 1930s teletype

To understand the relation between Unix computers and teletypes, it helps to make an analogy with our modern web-driven world. Google, Amazon, and others have numerous servers running web server software. Users can connect to these computers through the internet from their smartphones, tablets, laptops, and desktop computers running web browsers such as Firefox, Chrome, or Safari. Ultimately, you have multiple users connecting to the same computer over network cables. The server responds with requested web pages.

In the Unix mainframe setup, everything is more primitive by our modern standards. Instead of a smartphone, you connect with an electro-mechanical device, the teletype, which isn't even a real computer. A teletype cannot perform any calculations or run any software. All it can do is register what letters you press and send them over a serial cable to the Unix mainframe. The Unix mainframe will respond by sending letters back across the serial cable. These letters will get punched out on paper on your type-writer-like machine as they arrive.

There are several notable differences from our modern web and internet world. Networks send data in packets, such as TCP/IP packets. Even a humble USB cable actually sends packets. A packet is a self-contained little chunk of data. You can think of a packet like a letter: It says where it is from, where it is going to and contains some data. That way, data from multiple different clients can move along the same network cable to a server. When received the server can separate out the different packages so that it can handle communication between itself and different smartphones, tables and laptops all with just one physical cable.

Comparing relation between Unix terminals and modern Web clients

A Unix computer did not work like that. Every teletype needs a separate serial cable. It doesn't send packets of data. It just sends binary digits representing the keys you press as soon as you press them. It is a basic system. It was not actually made for computers in the first place. Teletype machines got made as a replacement for the telegraph. Instead of tapping a telegraph and hearing beeps on the other end, you had the luxury of being able to type whole texts. No need to learn morse code.

In the 1970s, teletypes evolved into more modern electronic variants such as the VT100. You can consider it to be a monitor and keyboard joined together as one device. Hence, you can think of old Unix mainframes as a computer which allowed you to plug in multiple screens and keyboards. Each keyboard and screen combo served a different user.

VT100 terminal. Not a PC. Only communicates with computer.

These terminals did not have to be connected to Unix mainframes. They could be connected to many other systems, such as VAX; thus they were quite generic devices. When Unix computers shrunk and became desktop computers for individual users such as a PC, the relationship between terminals and the computer changed. Unix got a windowing system called X Window System, where you run different software. One type of applications you could run were terminal emulators. An early variant was simply called xterm.

Terminal emulators pretend to be old physical terminals like the VT100. The underlying Unix system still thinks you are typing in a physical terminal connected by a serial cable. But how exactly do you trick Unix into thinking that the window of a terminal emulator is a "real" terminal? That is the next question we will explore.

Unix Device Files, TTY and PTY

On Unix, the filesystem is more like a namespace for different kinds of operating system resources, rather than simply physical files on disk. Most files map to files on your hard drive. However, on Unix files can also represent physical devices like your keyboard, mouse, audio card, hard drive, floppy disk drive, serial communication to a modem or terminal.

You find files representing these devices in the /dev directory. These files provide a way to communicate with the device drivers handling communications with the underlying physical device. The device driver is software which makes the underlying hardware look like a file in the /devdirectory. In the old days, Unix would communicate with each serial interface through /dev/ttyS0, /dev/ttyS1 and /dev/ttyS2 files (The Sstands for serial port).

A program could open one of these files and read from it to get what was being typed on a teletype machine. The programs could write to the file to send letters to the paper on the teletype machine.

Representing this connection to the outside world as just a file proved to be a powerful abstraction. Unix vendors could write new drivers to make communication with a terminal emulator look the same. Unix programs would read and write to the /dev/tty files without knowing that they now represented one of many windows in a graphical user interface. Unix has lived on to the modern era in large part because it created powerful abstraction early on.

While Unix users with physical terminals connected through a TTY device, they would often run commands such as telnet, rsh, rcp and rlogin (see r-commands). All these commands imply connecting to another Unix machine. Several Unix machines could be connected through a TCP/IP network. Say user Bob is sitting by an old teletype terminal and connects through a serial port to a Unix mainframe called Asterix. On this computer, he runs the telnet program to connect to another Unix computer called Obelix, which is connected to Asterix through a TCP/IP network connection.

While on the Obelix machine remotely, Bob issues commands such as ls. How does ls know where to send its output? None of the TTY devices would work as they relate to physical ports, but Bob is connected over a TCP/IP connection.

The solution is pseudo-terminals written PTY for short. These are represented as files under /dev which get created on the fly by the network applications. In fact, a terminal emulator is handled as a pseudo-terminal since it does not represent an actual physical port and you can make as many terminal windows as you like. On macOS, which I use they have names such as /dev/ttys000, /dev/ttys001 and ttys002. They get created in order as I make new terminal windows and tabs.

On macOS, you can check what TTY device your terminal emulator window is communicating with by issuing the tty command.

❯ tty
/dev/ttys000

You can use the process info command with the -a switch to get an overview over all the TTY devices currently used and what programs are being run on them. You can see from this overview that the first thing done after ttys000 was created was to run the login -fp erikengheimcommand.

❯ ps -a
  PID TTY           TIME CMD
25797 ttys000    0:00.02 login -fp erikengheim
25798 ttys000    0:00.10 -fish
25898 ttys000    0:00.00 ps -a
25849 ttys001    0:00.02 login -fp erikengheim
25850 ttys001    0:00.09 -fish
25897 ttys001    0:00.00 nc -l 1234

Unix programs when run create what we call processes. A process is the representation of a running program in an operating system. A process can create a child process, or spawn a child process, as we would normally call it. The login process spawns the fish shell process, which finally spawns the ps -a process, which gives this overview.

Meanwhile, on the second window I opened, /dev/ttys001 we got a number of other processes spawned. The beginning is the same, but I chose to start the NetCat program to listen for connections on port 1234.

That means I can send message and receive message from NetCat by doing a regular network socket connection to port 1234 on localhost. However, all it will end up doing is forwarding data to the /dev/ttys001pseudo-terminal in which NetCat is running and listening on port 1234. We can test all this to see how it works. Create two separate terminal windows:

• Server - Launch nc -l 1234 in this window.

• Client - Connect using telnet localhost 1234.

You can write a message in the client window to see if it pops up in the server window:

❯ telnet localhost 1234
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello world

On the server side, you should receive the following:

❯ nc -l 1234
hello world

If you are connecting from another computer you need to connect using telnet, but since we are local, we can cheat and write and read directly from the /dev/ttys001 pseudo-terminal. To get out of telnet press Ctrl-]. This gives you the telnet prompt, allowing you to issue different commands. You will just type quit:

❯ telnet localhost 1234
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

❯

Let us restart NetCat to do our little cheat. First, we use the tty command to make sure we got the correct TTY device. On my macOS I would get /dev/ttys001, but if you run on Linux you would get something different.

❯ tty
/dev/ttys001

❯ nc -l 1234

We switch to another terminal window and try sending text to the /dev/ttys001 device or whatever TTY you use.

❯ echo we are cheating > /dev/ttys001

You should see "we are cheating" pop up in the other window.

❯ nc -l 1234
we are cheating

You can equally well read from the TTY as if it were a file using cat. In this case, cat will block until you write a message.

Using NetCat to demonstrate TTY communication

You can signal end of communication by sending a Ctrl-D from NetCat. The cat command at the other end will interpret that as a hangup and exit. Technically, it maps to an EOF (End-of-File).

❯ nc -l 1234
we are cheating
A message from NetCat

At the other end, we will then see:

❯ cat /dev/ttys001
A message from NetCat

Any program allowing you to read or write to a file would work. For instance, editors such as vim, kak and emacs would work as well. When you save the file, it will write to the TTY device, and you will see what you edited in your NetCat window.

Serial Communication over USB Cables