Originally posted 2016-01-25.
1 Introduction
There are a huge variety of programming languages out there, an infinite number of concepts, and numerous ways to think about any given problem. How can one logically discriminate among these choices to make wise programming decisions?
First off, there exist a number of software-design camps, each with their own design methodoligies and folk-loreish guidelines.
Some people claim that you should thoroughly document your code with comments, others claim that you should use comments sparsely lest they become out of date and misdirect future readers.
Most people think it's important that programmers "write understandable code." But how does one define what is understandable and what is not? Is it better for code to be concise and dense (ala APL) or verbose and wordy (ala COBOL)? Some people argue that verbose code is easier to understand, as it can sometimes be "self-documenting", while others claim that dense code is easier to read, as it is shorter.
Object Oriented designers love applying acronyms such as SOLID and GRASP to software. Functional designers love applying principles and techniques such as immutability, declarativity, currying, and higher-order functions.
Every programmer has a preference for a certain programming language – probably the one which she has spent the most time with. People get into passionate debates comparing languages and software ecosystems.
Almost all programmers seem to have internalized the concept of "code smell" – we say things like "that code doesn't seem quite right" and "in Java, it's best practice to implement that feature differently".
2 What are the Objective Criteria?
All of the above statements, standing on their own, are subjective in nature. Is there a provable basis for any opinion with regards to software design?
My answer is that there are a number of objective mathematical criteria which affect software design, but that actually applying these criteria to real world problems involves making subjective judgements, just as everything in the real world involves making subjective judgements.
Below are some core objective principles which can be used to assess the runtime quality of software relative to criteria. The runtime principles are commonly discussed, I do not think my sourcecode principle is commonly expressed, in the form I express it at least.
3 Runtime Principles
3.1 Correctness
This principle is, essentially tautologically, the most important principle w.r.t software implementation.
Given a set of goals G, I define correctness for an algorithm A as whether or not A successfully produces the right output and invokes the correct side effects to satisfy G. There can be subjectivity in creating and assessing the goals. Undecidability is a potential choking point for many algorithms, and must be handled in the definition of G.
3.2 Runtime Guarantees
My definition of this principle is different than my definition of correctness, in that, rather than assessing merely whether the correct output/side effects are produced within standard execution of the code, we are also assessing guarantees and/or probabilities of runtime "performance" in the areas of reliability, space, and time usage.
Examples of features which I would categorize under this feature include uptime probability guarantees, fault-tolerance, bounded response time guarantees, and bounded space usage guarantees.
I do not include "runtime computational complexity" (asymptotic measures of how the performance of an algorithm changes as the size of its input changes) in my definition of "Runtime Guarantees".
3.3 Runtime Complexity
"Complexity" in various forms is a core topic of Computer-Science curricula, and is extensively discussed and researched. In computer science, we typically consider runtime time complexity and space complexity as primary criteria of the quality of an algorithm. One can also consider concepts such as runtime "energy complexity" as a loose concept defining bounds on likely energy usage.
4 Sourcecode Principles
Each of the above principles has been extensively discussed in academic and engineering literature. However, notice that NONE of the principles above say anything about how we should organize or write our code. Below I put forward a principle which I think is the core principle affecting code organization and writing.
4.1 Occam's Razor – Minimum Description Length
4.1.1 Introduction
Occam's Razor has time and time again shown its power in choosing models to explain phenomena. It can be shown to be naturally emergent in a wide range of fields including Bayesian probability, natural philosophy, and Theology.
Occam's Razor is simply the statement that "the simplest explanation is most likely to be the best."
4.1.2 Justification
The most convincing mathematical argument for Occam's Razor which I have seen comes an analysis (link to Chapter 28 of "Information Theory, Inference and Learning Algorithms" by David Mackaye) in Bayesian probability which concludes that "simpler" explanations in a given explanation-space have higher posterior probability after any given observation.
4.1.3 Bridge to Software
But how does Occam's Razor apply to software development? After all, its commonly stated purpose is in model selection.
I think the bridge from Occam's Razor for model selection to Occam's Razor for code organization is recognizing that choosing among code organization options is a form of model selection.
The phenomena we are trying to explain (match) are the runtime requirements of the software system. The models are the different code layouts and structures which we can choose among.
Software development is about solving problems with computers. Therefore, we can view the program as a model estimating the desired solution to the underlying problem. If two models (programs) perform roughly the same function, then we can choose between them by the one which more "sharply" models the problem, e.g. the one which is shorter by the principle of Occam's razor.
4.1.4 Application
If you agree that Occam's Razor is a reasonable principle to direct code composition, and even perhaps that it encompasses such commonly uses principles such as DRY (don't repeat yourself) and YAGNI (you ain't gonna need it), then the next question is: how can we apply Occam's Razor to make code organization choices?
Applying the principle, of course, is where we run into some trouble and must deal with subjectivity. In order to apply Occam's Razor to software, I propose that we look at Occam's Razor from a slightly different perspective which is often used in machine learning and statistics. Let's make more exact what we mean by "simpler" and instead discuss "minimum description length".
To compare two pieces of code with the same functionality, we can usually say that the code which has a smaller minimum description length more sharply models the desired feature set.
How shall we compare description length. Shall we compare lines of code? Bytes? Number of tokens? Function points?
I think each of the above approaches can be useful, but for most applications number of bytes or number of lines should be relatively interchangeable, when used together with common-sense.
Another alternative, which may be more useful to compare programs written in different languages or with different style conventions, is to compress the text of the program and compare the compressed size. This gets us closer to the "minimum description length" of the program and helps elminate differences caused by whitespace, descriptive variable names, and the like.
5 Conclusion
In this paper, we discussed a number of principles for writing software which are objective in their core nature and are based in mathematics. We discussed the standard principles which can be used to assess a program's runtime characterestics. In addition to those, however, we also discussed how the "Minimum Description Length Principle" can be used to make choices about code organization and design. Not only does the "Minimum Description Length Principle" encompass other commonly used principles such as "DRY" and "YAGNI", but it also provides a general framework to assess the "sharpness" of a codebase's match to a given problem. For each of the principles we discussed, we touched on the various difficulties in applying it to real-world problems.
While the principles discussed in this paper do not change the subjective nature of software development, they do correspond to core features of a software program that can be measured objectively or pseudo-objectively, and that can be strongly supported by simple arguments in mathematics and philosophy.