Structural Beauty: A Structure-Based Computational Approach to Quantifying the Beauty of an Image

To say that beauty is in the eye of the beholder means that beauty is largely subjective so varies from person to person. While the subjectivity view is commonly held, there is also an objectivity view that seeks to measure beauty or aesthetics in some quantitative manners. Christopher Alexander has long discovered that beauty or coherence highly correlates to the number of subsymmetries or substructures and demonstrated that there is a shared notion of beauty – structural beauty – among people and even different peoples, regardless of their faiths, cultures, and ethnicities. This notion of structural beauty arises directly out of living structure or wholeness, a physical and mathematical structure that underlies all space and matter. Based on the concept of living structure, this paper develops an approach for computing the structural beauty or life of an image (L) based on the number of automatically derived substructures (S) and their inherent hierarchy (H). To verify this approach, we conducted a series of case studies applied to eight pairs of images including Leonardo da Vinci's Mona Lisa and Jackson Pollock's Blue Poles. We discovered among others that Blue Poles is more structurally beautiful than the Mona Lisa, and traditional buildings are in general more structurally beautiful than their modernist counterparts. This finding implies that goodness of things or images is largely a matter of fact rather than an opinion or personal preference as conventionally conceived. The research on structural beauty has deep implications on many disciplines, where beauty or aesthetics is a major concern such as image understanding and computer vision, architecture and urban design, humanities and arts, neurophysiology, and psychology.

Keywords: Life; wholeness; figural goodness; head/tail breaks; computer vision

Beauty is commonly conceived to be in the eye of the beholder, which means that perceptions of beauty are subjective and vary from person to person. However, this commonly held view on beauty has long been challenged by researchers who sought to measure beauty in some quantitative manners (e.g., Birkhoff 1933, Eysenck 1942, Koenderink et al. 2018). In philosophy, beauty has started to be recognized as an objective concept (Scruton 2009). The most notable researcher among others is probably Christopher Alexander, who devoted his entire career in pursuit of true beauty in our gardens, buildings, streets, and cities, as well as in artifacts (e.g., Alexander 1979, 1999, 2002–2005, Alexander et al. 1977, 2012, Gabriel and Quillien 2019). He found through human perception experiments that beauty or coherence correlates very well to the number of subsymmetries or substructures (Alexander and Carey 1968, Gabriel 1998). In his life's work The Nature of Order, Alexander (2002–2005) demonstrated that beauty lies in the deep structure, so our feelings on beauty are largely shared regardless of our faiths, cultures, and ethnicities. It is essentially the deep structure—or living structure—that evokes a sense of beauty in the human mind and heart.

A living structure has numerous substructures or subsymmetries with an inherent hierarchy that retains two distinct properties: far more smalls than larges across the hierarchical levels or scales, while more or less similar on each level or scale of the hierarchy. For example, a tree as a living structure has far more small branches than large ones, while the branches on each scale (or each level of its hierarchy) are more or less similar sized. The concept of living structure means structurally living, not necessary to be biologically alive, so a dead tree can be a living structure as long as these two properties remain. These two properties—far more smalls than larges across the hierarchy, and more or less similar on each level of the hierarchy—constitute respectively two fundamental laws of living structure: scaling law (Jiang 2015a) and Tobler’s law (1970). Beauty is therefore—first and foremost—about the physical and mathematical structure that pervasively exists in things or their images (see Section 2 for more detail) and then the structure can be well reflected in the human heart and mind to have a sense of beauty. In other words, it is largely the underlying living structure that triggers the perception or cognition of beauty in the human mind and deep psyche. This paper is an attempt to develop a computational approach for assessing the goodness or beauty of an image based on the living structure.

An image is conventionally represented as a large set of uniform pixels (e.g., 1024 x 1024 pixels), but our perception of the image is hardly pixel-based. Instead, any meaningful image (which is not a noise image) can be perceived as a coherent whole or living structure, which consists of far more small substructures than large ones. These substructures with their inherent hierarchy are perceived as a coherent whole or living structure (see Figure 1 for an illustration). The most salient substructures at the top of the hierarchy receive the highest visual attention, while the least salient ones at the bottom of the hierarchy receive the lowest visual attention. Thus, under the perspective of living structure, an image is viewed as an iterative system that consists of the structure of the structure of the structure and so on. To further clarify this point, consider the same example of a tree consisting of trunks, big branches, middle branches, small branches, and numerous leaves, so the tree is with five hierarchical levels or scales. In other words, the notion of far more smalls than large recurs four times, while things are more or less similar on each of these five scales. From the point of view of human perception, the trunks receive the highest visual attention, while leaves receive the lowest visual attention; alternatively, the leaves (due to its largest amount or the highest density) receive the highest attention, while the trunks receive the lowest attention. It is the living structure view that motivates us to develop the computational approach to the goodness or beauty of an image.

This paper is further motivated by the research effort for better understanding images and human perception of beauty across a range of disciplines such as artificial intelligence (AI), computer vision, psychology, neurophysiology, and cognitive science. Related research questions in the effort include: What are the salient features or objects of an image? What is the mental image of a city? How can different images be ranked and compared in terms of their aesthetics? A commonly used approach to these questions is to use human subjects to assess a series of images on their goodness to reach a kind of inter-subjective agreement among people. The basic assumption of the conventional research approach is that beauty is in the eye of the beholder. This commonly used method is essentially a black-box method by taking the majority of the responses as the answer, albeit without asking why an image is beautiful. As a matter of fact, it is the living structure that lies behind the goodness or beauty of images, or it is the living structure that evokes a sense of beauty in the human mind and heart (Alexander 2002–2005). The goodness or beauty of images can be effectively evaluated through the so-called mirror-of-the-self experiment. Given two images side by side, the human subject is asked to pick one that better mirrors him/herself, or with the image the person has a higher degree of wholeness (e.g. Alexander 2002–2005, Wu 2015, Salingaros and Sussman 2020). The mirror-of-the-self experiment is to seek the objective existence of living structure rather than the inter-subjective agreement, so it differs fundamentally from human perception tests that are commonly used in psychology and cognitive science.

The contribution of this paper is four-fold: (1) an organic and holistic way of understanding an image, which is perceived as the figure (conspicuous part of an image) of the figure of the figure and so on with respect to the figure-ground perception (Rubin 1921); (2) the degree of structural beauty or life (L) measured by the multiplication of substructures (S) and the inherent hierarchy (H), thus making it possible to rank different images in terms of their goodness or structural beauty; (3) finding among others that Jackson Pollock's Blue Poles is more structurally beautiful than Leonardo da Vinci's Mona Lisa; and (4) discussions on the potential application and implication of structural beauty in a variety of sciences, and digital humanities and art.

The remainder of this paper is structured as follows. Section 2 introduces the concept of living structure and its fundamental laws—scaling law and Tobler's law—using a human face image as a working example. Section 3 presents the computational approach to the goodness or beauty of an image, and in particular the measure of structural beauty or life as the multiplication of substructures and their inherent hierarchy. Section 4 verifies the computational approach and reports our experiment and the results of case studies applied to 16 images including Blue Poles and the Mona Lisa. Section 5 further discusses the implications and applications of the approach in a variety of disciplines in both science and art. Finally, Section 6 concludes the paper and points to future work.

As mentioned above, human perception of an image is not based on individual pixels, but rather on the overall gestalt of the whole image, the overview, the broad nature of the image, according to Gestalt psychology (Koffka 1936). This overall gestalt is a de facto living structure that consists of many substructures or subsymmetries with far more smalls than larges. To illustrate the concept of living structure, let us take a gray image for example (Figure 1a). The gray image has 512 by 512 (262,144) pixels, each of which is with a gray scale between 0 and 255. The image can be converted into a binary one (Figure 1b) through the average pixel value 123, calculated from all the pixel values; this means that all the pixels darker than the average pixel (123) are set to black, while all the pixels lighter than the average pixel (123) are set to white. The binary image has 120,324 black pixels (46%) and 141,820 white pixels (54%), as shown in Figure 1b. The black and white pixels respectively constitute the figure (conspicuous part of an image) and ground of the binary image (Rubin 1921). The figure consists of 779 individual pieces or segments, which are formally called substructures or subsymmetries. Interestingly, there are far more small substructures than large ones (Figure 1c); more specifically, 1 largest substructure (red), 3 second largest (yellow), 10 third largest (light blue), and 765 fourth largest (or the smallest in blue). In other words, there are four hierarchical levels for the figure as a living structure, perceived at the four levels of salience. It is the living structure—or, more specifically, the substructures with an inherent hierarchy of far more smalls than larges—that evokes a sense of beauty in the human mind and heart.

The notion of living structure, seen from the above working example, is supported by two laws: scaling law across the four scales (or hierarchical levels) ranging from the smallest (blue) to the largest (red), and Tobler's law on each scale or level (of the four). These two laws are the fundamental laws of living structure. They are complementary to each other in many aspects, as shown in Table 1. For example, the ratio of smalls to larges is dispositional across scales: (1) large substructures are very large, while the small substructures are very small, and (2) the number of smalls is far greater than the number of larges. Along with the two laws, there are two design principles, namely differentiation and adaptation (Alexander2002–2005, Jiang 2019); the substructures can be said to be differentiated from the whole structure, yet they are well adapted to each other to constitute a cohere whole or living structure.

Any living structure has a certain degree of beauty or livingness, which can be characterized by many substructures and their inherent hierarchy. Thus, a useful rule regarding the degree of beauty is the more substructures, the more beautiful, and the higher the hierarchy of the substructures, the more beautiful

(Jiang 2019). As mentioned earlier, the living structure of the human face image has 779 substructures, which can be put into four hierarchical levels. It becomes less beautiful if the lowest level (all the blue substructures) is removed, because the number of substructures is dramatically reduced, while the level of the hierarchy is decreased from four to three. This rule on beauty constitutes the major criterion for comparing and ranking the goodness of images. The living structure provides an objective measure to quantify structural beauty. This is the foundation of the computational approach to the goodness of an image to be developed in this paper.

(Note: A gray image (a) is – through the average pixel value – converted into a binary one (b) that constitutes the figure (as a living structure) of the image. The living structure consists of our levels of substructures indicated the four colors (c): one largest substructure (red), many smallest substructures (blue), and two levels of substructures (yellow and light blue) between the largest and smallest. Apparently, there are far more small substructures than large ones across the four scales (scaling law), while on each of the four scale substructures are more or less similar (Tobler’s law).)

(Note: Scaling law is available across scales ranging from the smallest to largest, while Tobler’s law is on each of the scales (Jiang and Slocum 2020).)

Underlying the notion of living structure is wholeness, which stresses that things or images in particular should be viewed holistically rather than analytically, because the whole is more than the sum of the parts. Ever since Gestalt psychology (Koffka 1936) was developed, the general idea of wholeness has been extensively studied in philosophy and in a variety of sciences such as quantum physics, biology, neurophysiology, medicine, cosmology, and ecology (e.g., Bohm 1980, Alexander 2002–2005). Alexander was the first to turn the general idea of wholeness into a physical and mathematical concept in some precise mathematical language. He discovered that the holistic way of seeing things—unconsciously or subconsciously—is more correct (Alexander and Huggins 1964), although most people tend to see things analytically as fragmented pieces.

The theory of living structure was initially conceived and developed to create beautiful buildings and cities, but also to help explain many symmetry-breaking phenomena (Alexander 2002–2005, 2005). Living structure exists pervasively to some degree or other in any space and matter. The degree of living structure or structural beauty is real and measurable, very much like temperature; the living structure is to beauty what temperature is to warmth. However, at the time when his life’s work was published (Alexander 2002–2005), no mathematical models could capture his definition of living structure, so he used hundreds of pictures, paintings, and drawings to clarify his theory and design thoughts. Recently, Jiang (2015b) has developed a mathematical model of living structure that is able to address not only why a structure is beautiful, but also how beauty the structure is. In summary, living structure is a physical phenomenon that exists pervasively in our surroundings, and can be defined mathematically, and can be well reflected in the human mind and heart psychologically.

The computational approach to be introduced is based on the figure of an image rather than the image itself for computing its goodness or structural beauty. Before introducing the approach, we need to first introduce head/tail breaks (Jiang 2013), a classification scheme for data with a heavy-tailed distribution. The computational approach is, to a large extent, an application of head/tail breaks into an image.

We applied the computational approach to eight pairs of images to compute the degree of goodness or structural beauty. All of the images have the same number of pixels (262,144), but the length and width ratio may vary from one to another (Figure 4). The primary goal of the experiment was to compare the goodness or structural beauty of images one from another. We first report on our overall results on the eight pairs of images in terms of their goodness or structural beauty and based on the computed score $L$, and then take a look at three pairs to show that (1) Blue Poles is more structurally beautiful than the Mona Lisa, (2) the Tower of the Wild Goose is more structurally beautiful than the modernist house (or in general traditional buildings are more structurally beautiful than their modernist counterparts), and (3) the weather-beaten face is more structurally beautiful than the posed model. We also show that the average cut is better than all other alternative cuts for the computation.

(Note: For each pair of the images (P1–P8), the left is more living than the right, which is shown through their living structures (L1–L8). The first four pairs (P1–P4 to the left) were previously studied by Alexander (2002–2005), so they have the information on which one is more beautiful than the other, which is consistent with our computational result. The second four pairs (P5–P8) are without a ground truth, but according to our computation, the left is more structurally beautiful than the right.)

The computational approach developed in this paper works very well for measuring structural beauty and further differentiating two images in terms of their goodness or structural beauty. The approach is very much in line with Alexander's vision about the mathematics of beauty or life as cited in the epigraph. To be more specific, the levels of scale property Alexander mentioned is a de facto inherent hierarchy of substructures, or the recurring notion of far more small substructures than large ones. The void property can be viewed as the largest or the most salient substructures, while the inner calm property refers to each level of scale (or hierarchy) in which substructures are more or less similar sized. This section further discusses the computational approach and its implications in a larger context.

The computational approach is developed under the third view of space or the new organic cosmology, in which space is neither lifeless nor neutral, but a living structure capable of being more living or less living (Alexander 2002–2005, Whitehead 1929). Under the new organic view of space, an image is viewed as a living structure that consists of far more small substructures than large ones, and a painting in its course of making can become more living or less living, but ultimately towards the most living. All individual substructures of an image are naturally and organismically defined based on the average cuts. This way of representing an image or painting mimics our perception but differs fundamentally from the pixel-based representation that is essentially a nonliving structure under the mechanistic world view (Descartes 1637, 1954). Different from the pixel-based representation, the living structure can capture visual hierarchy of an image, which well reflects human perception of the image holistically.

Structural beauty has many synonyms, including living structure, wholeness, life, organized complexity (Jacobs 1961, Salinger 1997), and fractal (Mandelbrot 1983), all of which focus on the structural aspect. This structural aspect has been previously studied in the literature, albeit without explicitly referring to the notion of living structure, simply because it did not yet exist. For example, the hierarchical model that underlies the central place theory (Christaller 1933, 1966), is essentially about the kind of structural beauty over a country among all the settlements at different levels of hierarchy. Not only the settlements as a whole, but also an individual settlement (or city) demonstrates the kind of structural beauty, because there are far more small substructures than large ones in the city. It is the living structure or structural beauty that helps shape the mental image of the city (Lynch 1960). In this regard, the computational approach also provides an effective and efficient measure for computing the image of the city.

The measure of structural beauty ($L$), as defined by $S$ (substructures) times $H$ (hierarchy), reminds us of the classic work on aesthetic measure (Birkhoff 1933). The classic work had the same motivation as our computational approach; it was aimed to quantify the degree of beauty by disregarding colors and materials, as well as human aspects such as cultures, education, and ethnicities. The aesthetic measure ($M$) considers the two notions of order ($O$) and complexity ($C$) and combines them together into a single formula: $M\; =\; O/C$. The formula shows an inverse relationship between the degree of beauty and that of complexity, $M\; \backslash propto\; 1/C$, which is against the notion of organized complexity. Eysenck (1942) changed the initial formula to $M\; =\; O\; \backslash times\; C$, which makes better sense, at least from the point of organized complexity because the more complex something is, the more beautiful it is. The biggest problem of the classic measure is that it has never been verified by any psychological study (Douchova 2015). On the other hand, the degree of beauty based on living structure is well supported by the mirror-of-the-self experiments mentioned above.

The research on structural beauty has many applications and implications on various disciplines where beauty or aesthetics is a major concern, such as landscape, architecture and urban design (Lothian 1999,

Workman et al. 2017, Building Beauty 2017), mathematics (Birkhoff 1933, Koenderink et al. 2018), humanities and arts (Gardiner and Musto 2015, Snow 1959, Taylor et al. 1999), neurophysiology (Kawabata and Zeki 2004), psychology (Valentine 2015), philosophy, and more recently on AI, big data, and image understanding and computer vision. The work has provided hard evidence of the trend in philosophy that beauty has started to be accepted as an objective concept (Scruton 2009). Artificial neural networks have been used to train a massive number of crowdsourced images for assessing their goodness or scenicness (e.g., Seresinhe et al. 2017, Bodini 2019, Cetinic et al. 2019). The result developed by Seresinhe and her colleagues on scenicness of areas, although harvested from subjective judgement of individual people, is consistent with our results from the previous section. For example, large flat areas of greenspace, such as ‘grass’ and ‘athletic field’, are associated with lower scenicness, while the areas or images tagged with ‘mountain’, ‘water’, and ‘castle’ are associated with higher scenicness. Apparently, the areas with lower scenicness lack detailed intricate substructures like modernist buildings, whereas areas with natural scenes possess intricate details, like Blue Poles. The objective nature of beauty has enormous impacts in many design fields, because the goodness of a design is no longer considered to be an opinion, but a matter of fact. The computational approach provides an organic or living structure means for understanding computer vision and conducting image processing, because it mimics our perception of images.

Structural beauty, as defined and computed in this paper, presents a radical mindset change from subjective to objective beauty, thus significantly contributing to the effort on aesthetic measures and image understanding. We develop a computational approach to structural beauty or goodness of an image based on the living structure, a new way of image understanding. An image is commonly represented mechanically by many individual pixels, but human perception of the image is hardly pixel-based and is instead oriented towards a coherent whole (e.g., the figure of the figure of the figure and so on) or living structure. As a natural and organic representation, a living structure derived from an image constitutes the backbone or configuration of the image from a holistic perspective. It is governed by two fundamental laws: scaling law and Tobler’s law, which are respectively available across different levels and at each level of the hierarchy. There are far more small substructures than large ones, according to scaling law, whereas substructures are more or less similar in terms of Tobler’s law. These two laws of living structure underlie the computational approach to the goodness or structural beauty of an image. The living structure of an image is composed of many substructures with the inherent hierarchy of far more smalls than larges. The figure of the image can be further composed of many substructures with the inherent hierarchy of far more smalls than larges. Therefore, structural beauty or life (L), given as $S$ (the number of substructures) times $H$ (the number of hierarchical levels), is computed based on the rule that the more substructures, the more beautiful, and the higher hierarchy, the more beautiful. The measure of structural beauty or the computational approach in general is shown to be simple, effective, and efficient for ranking different images.

Seen from the recursive perspective, an image can be perceived as an iterative system that consists of the figure of the figure of the figure and so on. In this connection, the computational approach resembles very much – in spirit, but not in detail – the head/tail breaks that represents a heavy-tailed dataset as the head of the head of the head and so on. This recursive way of understanding images is probably the most significant contribution of this paper. Based on the computational approach, we (re-)discovered that (1) traditional buildings are more structurally beautiful than their modernist counterparts, (2) Blue Poles is more structurally beautiful than the Mona Lisa, and (3) the weather-beaten face is more structurally beautiful than the posed model. These findings may sound controversial, but they are purely based on the structural point of view without considering cultural, social, racial, and other biophilic factors. Our future work will seek to integrate these other factors into our model.

Data and Code Availability Statement: The data used and generated in this study are available at https://figshare.com/s/09304653edcd8e8adc82. The code is based on python 2.7 (https://www.python.org/download/releases/2.7/) and uses the following libraries: Arcpy 10.8 (ArcGIS

10.8, https://www.esri.com/en-us/arcgis/products/arcgis-desktop/overview), Numpy 1.16.5 (https://numpy.org/), Powerlaw 1.4.6 (https://pypi.org/project/powerlaw/) and Pillow 6.2.0 (https://pypi.org/project/Pillow/). Additionally, head/tail breaks have been calculated with a head/tail breaks calculator which is included in https://figshare.com/s/09304653edcd8e8adc82.