Breaking Big Applications into Small Applications
Reflections on modern software complexity and how to solve it, by using inspiration from the past.
Modern software is increasingly complex, inflexible and ironically too minimalist, all at the same time. Understanding the reasons is a huge and complex topic. That is why I will focus on a small part of the problem, which, I hope, will illuminate some underlying drivers of this complexity.
Let me start by giving you some observations to clarify the issue I intend to explore.
A lot of the software used daily follow a very common pattern: A large number of objects to organize and some kind of interface to manipulate and view each of these individual objects. In an email application, for instance, these objects are individual emails. Tools are provided to organize, search, sort and remove emails. Another part of the email application allows you to view, compose and alter individual emails. Essentially, we have what is called a master-detail relation in the user interface.
A photo album application follows a similar pattern: A way to organize photos and put them into different albums or give them different tags. Next there are ways of viewing these photos, zooming, perform a slide show or do simple touchups.
This very article is written using Ulysses, a Markdown tool, which allows both sophisticated organization of markdown files, and editing and viewing of individual markdown files.
A modern integrated development environment is a variation of the same idea: Some kind of way to organizing and locate different source code files based on their content. At the detail level you have editors for viewing, searching and modifying these individual source code files.
Hold on, don't we have a dedicated tool specifically designed to organize data objects? We do. It is called a file manager, and the objects are called files. Some 15-20 years ago, it used to be an essential skill to learn how to use a file manager. Not so much anymore, but previously users frequently organized their photos, videos, documents, and music using a file manager.
Today, things have changed for users across all platforms. On my Mac for instance I organize my books through Apple's Books application. My music files through their Music app, my personal photos through Photos. My Markdown files are organized inside Ulysses. This is just at the personal level. The way we exchange data has significantly changed as well. Earlier, we used file manager-like systems such as ftp and remote network drives to share and organize data. Many still do, but increasingly various web interfaces such as SharePoint, Box, and other replace these solutions.
For a long time, I was actually quite fond of this approach: Let dedicated applications each give you a unique perspective on your files suiting the particular needs of that application. But then I started seeing many of the problems.
These curated views into the filesystem are prone to frequent changes and deprive you of using the rich set of generic tools which already exist to help organize and search for files. Many of my old iPhoto libraries can no longer be opened because Photos use an entirely different format. Many of the apps from Books to Ulysses have at times weird and annoying syncing issues. Resolving these issues involves dealing with a black box because you are deprived of using regular file manipulation tools to fix the issues. I have gotten ePub books stuck, which I cannot seem to remove or to update. How often do you get a file stuck, which you cannot find any way of removing? I cannot remember a single instance of that happening ever. Ulysses keeps running backups, which bloats my hard drive. Yet, the UI offers no way of locating either the backup site or the individual files it manages. You can in principle, but only through detective work. The app itself will not help.
The fundamental problem is that every application is increasingly turning into its own little island. VS Code running on Electron is almost like an Operating System onto itself. Single apps can consume hundreds of megabytes.
Back in the 1980s, I had a fully Windowing system with multiple applications running off a measly 880 KB floppy disk fitting into 256 KB of RAM. Godot is pulling off, putting a whole complex game engine and sophisticated game development tools all within around 30 MB.
Enough complaining. What exactly can we do about this development? Can the past teach the present something?
The Unix Philosophy
I have a strong belief in the Unix philosophy of making simple tools that do one thing well and which can be combined with other tools. That way you can solve more complex problems by assembling multiple simple tools you already master, rather than relying on complex specialized tools. How do you make tools composable, so this solution becomes possible? The Unix innovation was to allow pipelines to be inserted between executing Unix commands. You might want to look at my Unix Commands, Pipes, and Processes story for a more in-depth discussion.
Here is the catch: How do you adopt this philosophy in a modern UI? A command line interface is quite limiting. Answering that question was really my motivation for writing this story.
Let me start with one example close to my heart: The TextMate editor. I have used this editor for 18 years. One of the reasons for that is that it was the first time I saw the Unix philosophy brought to a modern UI application. TextMate today is a bit like the LISP and Smalltalk programming languages. Neither are widely used but both had tremendous influence on how other languages evolved later. The TextMate plugin system got popular enough that even modern editors such as VS Code still base a lot of how their plugins are made on the patterns established by TextMate 18 years ago.
The TextMate Bundle System
Let me use an example from my the Unix command line story to explain the TextMate plugin or rather bundle system. In one example we combine a number of Unix tools such as cat
, tr
and cut
to transform CSV data about pizza sales into something more readable.
"","id","date","time","name","size","type","price"
"1","2015-000001","2015-01-01","11:38:36","hawaiian","M","classic",13.25
"2","2015-000002","2015-01-01","11:57:40","classic_dlx","M","classic",16
"3","2015-000002","2015-01-01","11:57:40","mexicana","M","veggie",16
After running a series of Unix commands through some pipes we get the following output:
size type price
M classic 13.25
M classic 16
M veggie 16
You may have already mastered these Unix tools and want to reuse you skills to add similar functionality to you editor. In TextMate you can do just that. A TextMate bundle contains a collection of snippets, language grammars and commands. The following screenshot shows a command pretty table
implemented. It is a simple Unix shell script.
The question is how does the shell script know what data to operate on. The cat -
command starting the pipeline indicates that the shell script reads input from stdin, before writing output to stdout. But where is stdin and stdout connected? That is what the configuration panel on the right specifies. It says that the input to the command is the current text selection in the editor. Further we are specifying that the output from the shell script replaces the input selection.
Let me show another example. This is the Assemble
command for my Calcutron-33 assembler. If you want to assemble the file foobar.ct33
you write cutron asm foobar.ct33
. In the shell script below the same command is written as $TM_CUTRON asm $TM_FILEPATH
.
Whenever TextMate runs an external command defined in a bundle it fills in numerous Unix environment variables with info about the state of the editor such as the current line number, current line, the path to the currently edited file and so on. TextMate calls these dynamic variables. The TM_FILEPATH
is an environment variable which always contains the file path of the currently edited file. TM_CUTRON
in contrast is a custom defined environment variable. You can create custom environment variable. That is commonly used to let a bundle command know where external commands are located.
Notice how the right panel uses a different configuration. No input is read and the output is written to a new window rather than being used to overwrite the currently selected text. In other words TextMate gives you a lot of options with respect to feeding data into a command and using the data that comes out.
The beauty of the TextMate approach is that all programming languages have facilities for reading environment variables, stdin and stdout. We get several benefits from this approach:
Plugins can be implemented in any language
Process separation. Plugins run as separate programs
Such as approach was quite different from what was normal at the time. In the Windows world plugins were usually loaded as dynamically linked libraries. It meant you had to abide by a brittle binary interface and your plugin ran in the same memory space as the rest of the editor. Consequently if there was a bug in your plugin it could make your whole program crash.
This flexibility and robustness caused TextMate to very quickly get more plugins for all possible language quicker than almost any other programming editor at the time.
Covering all the things you can do with the TextMate plugin system is beyond the scope of this article. The rest of my story will instead be focused on how the Unix philosophy is applied the Haiku operating system, and what we can learn from that.
The Haiku OS Messaging System
Let us compare the email system in Thunderbird with the Haiku email system, because it is a good way to contrast very different approaches to building software. You can see in the little gif animation how Thunderbird gives you access to a calendar, address book, emails, account setup, and much more.
In the Haiku world all the same functionality is handled by five different applications:
Mail – For composing and reading emails messages
Tracker – File manager used to organized and search for emails
People – Used to write or lookup information about a contact
mail_demon – Handles sending and receiving of emails
E-mail – Configure and setup email accounts
All these apps are located in different locations. The first three are in the appsfolder, while the email preferences is placed together with other Haiku preferences. But I collected all the relevant apps into one window on the top right to give you an overview.
Mail and Tracker Integration
In Haiku, looking through your emails is done with the file manager Tracker. You simply navigate to the directory containing the emails you want to look at. Double-clicking an email file opens up the Mail app, which lets you read the selected email and respond to it. The app has arrows to move up and down to the next and previous email.
Thus rather than having a dedicated application to show all your emails and allow you to sort them and organize them, you use Tracker, the standard file manager. You might ask why this was never done in the past. The simple reason is that most operating systems did not traditionally have extendible file attributes.
It would be awkward to browse emails by the name of the email file and the date that file was created. Instead, we want to see things such as sender, subject and when the email was received.
You may notice that in the screenshot that the Haiku file manager is somehow able to show exactly those things. The Haiku file system makes that possible by pioneering extendible attributes.
Emails are simply files with subject, from, when, and other attributes attached. In addition, the file manager will remember what attributes you configured to show for each directory. Thus, once a directory has been configured to show attributes useful for email it will remember that until next time it is opened.
Users of Haiku are actually free to create whatever file types they want with whatever attributes they want, as this example creating a DVD file type shows. Look at the Haiku user guide for more details on how this is done.
Email contacts are handled in a similar manner. Every contact is represented with a file with different attributes for name, email, address, and country. The file manager is used to show list of all people. The application for editing and adding a person is extremely simple. It just edits a single person.
It has no UI for browsing or searching contacts. All those needs are outsourced to the file manager. The Tracker file manager has the ability to search for files based on attributes. These queries can be stored as separate files and reused. A query acts much like a smart folder. You can open it later and look at the results as if it were a normal directory.
Every contact added as a file becomes available in the email application as a completion when I try to write the email of the recipient.
You may wonder how all of this integration works. On one level, it is the use of extendible attributes in the filesystem. The other aspect is the fact that the mail application can modify selections in the Tracker file manager. How is that accomplished? Every Haiku application is highly scriptable because the whole UI system is built around a flexible messaging system. Haiku was built around what they call pervasive multithreading. To avoid race conditions, applications internally coordinate by sending messages between different threads. Messages are of type BMessage
.
Interestingly the same message system used internally is also used to send messages between applications. A similar message pump used to handle clicks and other UI actions can be used to handle messages from external applications. For instance here is an excerpt from the Mail source code, which shows how the Mail application sends a message to the Tracker application about currently selected email.
void
TMailWindow::SetTrackerSelectionToCurrent()
{
BMessage setSelection(B_SET_PROPERTY);
setSelection.AddSpecifier("Selection");
setSelection.AddRef("data", fRef);
fTrackerMessenger.SendMessage(&setSelection);
}
This system makes it possible for users to script applications. You got a command line application called hey which allows users to send messages to any Haiku application.
So, I have given a sense of how many simple applications can cooperate, but at this point you might be asking: Doesn't all of this get very messy? Do you really want all these different windows floating around? The reason many people want integrated applications is so they can have everything they work on clustered. However Haiku has some excellent ways of solving that problem which will be the next topic.
UI COMPOSITION AND INTEGRATION
If you work on email, you would want to make it easy to switch between different workflows without having a big maze of windows to deal with. Fortunately Haiku has a clever system for gluing windows together in a variety of ways.
Holding the option key down while moving windows around allow you to snap them together in different ways. Here I have glued together Tracker and the Mail application so that I can move both windows around as if they were one unit. You can even resize one window and the other window expands to match.
Keep reading with a 7-day free trial
Subscribe to Erik Explores to keep reading this post and get 7 days of free access to the full post archives.