Living Images: A Recursive Approach to Computing the Structural Beauty of Images or the Livingness of Space

According to Gestalt theory, any image is perceived subconsciously as a coherent structure (or whole) with two contrast substructures: figure and ground. The figure consists of numerous auto-generated substructures with an inherent hierarchy of far more smalls than larges. Through these substructures, the structural beauty of an image (L), or equivalently the livingness of space, can be computed by the multiplication of the number of substructures (S) and their inherent hierarchy (H). This definition implies that the more substructures something has, the more living or more structurally beautiful it is, and the higher hierarchy of the substructures, the more living or more structurally beautiful. This is the non-recursive approach to the structural beauty of images or the livingness of space. In this paper we develop a recursive approach, which derives all substructures of an image (instead of its figure) and continues the deriving process for those decomposable substructures until none of them are decomposable. All of the substructures derived at different iterations (or recursive levels) together constitute a living structure; hence the notion of living images. We have applied the recursive approach to a set of images that have been previously studied in the literature and found that (1) the number of substructures of an image is far lower (3 percent on average) than the number of pixels and the centroids of the substructures can effectively capture the skeleton or saliency of the image; (2) all the images have the recursive levels more than three, indicating that they are indeed living images; (3) no more than 2 percent of the substructures are decomposable, implying that a vast amount of the substructures are not decomposable; (4) structural beauty can be well measured by the recursively defined substructures, as well as their decomposable subsets. Despite a slightly higher computational cost, the recursive approach is proved to be more robust than the non-recursive approach. The recursive approach and the non-recursive approach both provide a powerful means to study the livingness or vitality of space in cities and communities.

Keywords: Substructures, living structure, wholeness, structural beauty, head/tail breaks, livingness of space

All space has some degree of livingness in it, according to its structure and arrangement (Alexander 2002–2005), so the livingness is commonly sensed in our surrounding such as rooms, gardens, buildings, streets, and cities, as well as in tiny ornaments. The livingness sounds like a kind of human experience of space or a sense of place attachment (Tuan 1977, Goodchild and Li 2012), synonymous with the vitality or organized complexity (Jacobs 1961), and the imageability or legibility (Lynch 1960). Unlike these concepts, however, the livingness is defined mathematically through the underlying living structure. The living structure is a mathematical structure with an inherent hierarchy (see Section 2 for an introduction), which can trigger the feeling of livingness in the human mind and heart. The inherent hierarchy of living structure is commonly recognized in a series of urban and geographic theories such as the central place theory (Christaller 1933, 1966), space syntax (Hillier and Hanson 1984, Hillier 1996), and the fractal cities (Batty and Longley 1994). The hierarchy may reflect spatial heterogeneity, one of the two spatial properties, the other being spatial dependence (Goodchild 2004, Anselin 1989, Tobler 1970). As shown later in Section 2, the hierarchy or spatial heterogeneity is better characterized by the recurring notion of far more smalls than larges across different levels of scale.

Any image is perceived by human eyes subconsciously as a coherent structure (or whole) with two large, contrasting substructures: figure and ground (Koffka 1936, Rubin 1921). The figure, which is also called the foreground, constitutes the focus of the visual field, while the ground is the background. The figure is a living structure, for it can be decomposed into many substructures with an inherent hierarchy of far more smalls than larges. The substructures are auto-generated segments (or sets of pixels) out of a gray-scale image by vectorizing the individual sets of pixels that are darker (or lighter) than the average pixel. There are far more small substructures than large ones across the hierarchy or the different levels of scale, yet the substructures on each of the hierarchy are more or less similar in size. It is essentially the recurring notion of far more small substructures than large ones that triggers a sense of livingness in the human mind and heart (Jiang 2019). This sense of livingness is called structural beauty (Jiang and De Rijke 2021) and it is shared among people, and even different peoples, regardless of their cultures, gender, and races. Thus, the livingness (L) or structural beauty is defined by the derived substructures, or more specifically the multiplication of their number (S) and their inherent hierarchy (H); that is, $L\; =\; S\; \backslash times\; H$. Instead of working with the figure, we, in the present paper, work directly with the image itself and derive its substructures, and the substructures of the decomposable substructures recursively until all substructures are no longer decomposable. All the substructures at different iterations (or recursive levels) together constitute a coherent whole or a living structure: hence the notion of living images, the central theme of this paper.

In this paper we consider an image – or space in general – to be a living structure that is composed of recursively defined substructures. This is a holistic view of perceiving an image or space as a coherent whole, so it differs fundamentally from conventional thinking (e.g., Umbaugh 2017, Davies 2017). Conventional image understanding tends to identify a few features or objects that can be named by words or recognizable by human eyes – so-called computer vision. For example, a human face image consists of numerous substructures with far more smalls than large ones, but our natural language can only name certain features, such as the eyes, the nose, the mouth, the ears, and the hair. In other words, a vast majority of substructures cannot be named by words. In general terms, a gray-scale image can be decomposed, around the average pixel value $m\_1$, into dark pixels (darker than $m\_1$) which may be called the figure (say, 52 percent) and light pixels (lighter than $m\_1$), which may be called the ground (for example, 48 percent). Although the figure (or the dark pixels) is perceived as a whole, it consists of numerous substructures with an inherent hierarchy of far more smalls than larges. Interestingly, the figure can be further decomposed in a recursive manner, around the average pixel value ($m\_i$) of the figure, into dark and light pixels, leading to numerous substructures with far more smalls than larges. This decomposition process is referred to here as recursion.

We have mentioned two concepts so far: hierarchy and recursion. The hierarchy refers to the recurring notion of far more small substructures than large ones on each of the recursion, whereas the recursion refers to the decomposition process for those substructures that are decomposable at different levels of the recursion. Here we use a parable to further clarify these two concepts. Imagine a tree with five hierarchical levels: one trunk, two limbs, eight branches, 24 twigs, and 96 leaves. According to the above mentioned formula, $L\; =\; S\; \backslash times\; H$, the degree of structural beauty is calculated through the multiplication of substructures ($1\; +\; 2\; +\; 8\; +\; 24\; +\; 96\; =\; 134$) and their hierarchy (5); that is, $134\; \backslash times\; 5\; =\; 670$. We know that the leaves have their own textures that have a similar hierarchy to the tree itself (for example, five levels). This means that each of the leaves can be decomposed into five hierarchical levels (Jiang and Huang 2021). Thus, there are two levels of recursion, and 96 decomposable substructures (or leaves), which means there is an alternative way of calculating the degree of structural beauty is $96\; \backslash times\; 2\; =\; 192$ (to be introduced as a new formula in Section 3.2). It is this new insight about the recursive nature of substructures that motivated us to develop this paper.

The contribution of this paper lies in the living structure perspective, a holistic and comprehensive approach to measuring the structural beauty of images or the livingness of space in a recursive manner. More specifically, there are four major findings from this study. First, all images are living images that have recursive levels more than four. Second, the centroids of the recursively defined substructures effectively capture the skeleton or saliency of the images, and the number of centroids (or substructures) is far fewer than the number of pixels. Third, among the derived substructures, no more than 2 percent are decomposable. Fourth, not only substructures but also their decomposable subsets can be used to measure the structural beauty of images or the livingness of space.

The remainder of this paper is structured as follows. Section 2 introduces the concept of living structure and explains why one structure is more living or structurally beautiful than another, not as an idiosyncratic opinion, but as a matter of measurable fact. Section 3 uses a working example to illustrate a recursive approach to computing the livingness of space or structural beauty of images. Section 4 reports our case studies for verification of the recursive approach and major findings out of the case studies. In Section 5 we further discuss on the livingness of space in terms of related works, its application, and implications on geography and even beyond. Finally, in Section 6 we draw a conclusion and point to future work.

Living structure is what underlies the notion of structural beauty or the livingness of space. As mentioned above, the degree of structural beauty of an image (L) is determined by its substructures, or their number (S) and their hierarchy (H) to be more precise. In other words, the more substructures an image has, the more living or more structurally beautiful it is, and the higher hierarchy of the substructures, the more living or more structurally beautiful the image is. This rule about livingness or structural beauty was distilled from the 15 properties (cf. Figure 1), originated from the 253 patterns (Alexander et al. 1977). These properties recur in natural things – naturally occurring things – as well as in human-made things. Those natural and human-made things are genuinely beautiful and can trigger a sense of livingness in the human mind and heart (Alexander and Carey 1968, Alexander 1993, 2002–2005). In this section, we first compare two drawings – as two living structures – in terms of their livingness through calculating the L score. Eventually, the one with the higher L score is more living or more structurally beautiful. These two working examples help us to understand why one structure is more living than another, and importantly two fundamental laws of living structure.

Two drawings are conventionally seen as two sets of four squares and four octagons, respectively (Panels a and b of Figure 2). Instead of this static or LEGO-like assembly view, we consider these two drawings to be evolved from an empty square of size one through the three iterations (Panel d). The two drawings have a clear figure distinguished from the white background: the four squares and the four octagons, respectively. The four squares (or four substructures in general, $S\; =\; 4$) have two hierarchical levels ($H\; =\; 2$), so the four squares as a whole have structural beauty of $4\; \backslash times\; 2\; =\; 8$ (Panel c). Alternatively seen from the perspective of 16 sides, the structural beauty is $16\; \backslash times\; 2\; =\; 32$. For the right drawing (Panel b), there are four octagons ($S\; =\; 4$) that are defined at three hierarchical levels: 1, 1/3, and 1/9 ($H\; =\; 3$), which means that structural beauty is $4\; \backslash times\; 3\; =\; 12$. Alternatively, there are 32 edges of the four octagons defined at the three hierarchical levels, so structural beauty is $32\; \backslash times\; 3\; =\; 96$. From the calculation, there is little doubt that drawing (b) is more living or more structurally beautiful than drawing (a). The above computations are shown in Panel c.

Having shown the two examples of living structure, let's see how it can be defined formally. Living structure is a physical phenomenon pervasively existing in our surroundings and is a mathematical structure that consists of or is transformed in a step-by-step fashion into numerous substructures with an inherent hierarchy. Across different levels of the hierarchy there are far more small substructures than large ones, yet on each level of the hierarchy substructures are more or less similar. These two notions – far more smalls than larges, and more or less similar – underlie the scaling law (Jiang 2015) and Tobler’s law (1970), respectively. These two laws are complementary rather than contradictory to each other (Table 1) to characterize the various kinds of living structure in natural and man-made things. These two laws will be further illustrated in the next section.

Strictly speaking, neither of the two drawings is a living structure because they both violate Tobler’s law. The four squares or four octagons are precisely the same rather than more or less similar. In addition, the left drawing violates the scaling law, which states that the notion of far more smalls than larges recurs at least twice or with the hierarchy being three. Nevertheless, the two drawings are simple enough to illustrate various concepts such as living structure, substructures, hierarchy, two fundamental laws, and the livingness of space.

In this section, we introduce a recursive approach to the structural beauty of images by extending the substructures to recursive ones. Before that, we will also illustrate the head/tail breaks (Jiang 2013) as a recursive function for deriving the underlying living structure or the inherent hierarchy of a dataset. The dataset contains 100 numbers as a working example to mimic the rank–size distribution: the largest size is about twice as large as the second largest, approximately three times as large as the third largest, and so on (Zipf 1949). This working example also helps to illustrate the two laws of living structure.

We applied the recursive approach to the eight pairs of images to verify whether the recursive approach is better or more robust than the non-recursive approach. The same set of images had been used to verify the non-recursive approach previously (Jiang and de Rijke 2021), so they were convenient for verification of the recursive approach and for comparing the non-recursive and recursive approaches. The first four pairs had previously been studied by Alexander (2002–2005), who used the 15 properties to examine their livingness, so they are with ground truth on their livingness or structural beauty. In addition to the verification and comparison, we found that (1) the number of the recursively defined substructures of an image is far smaller (3 percent on average) than the number of pixels, and the centroids of these substructures can well capture the skeleton or saliency of the image; (2) all the images have the recursive levels more than four, indicating that they are indeed living images; and (3) no more than 2 percent of the substructures are decomposable, but they can well characterize the structural beauty.

Any space or image has a certain degree of livingness or structural beauty, and it can be sensed in the mind and heart. Conventionally, the degree of livingness was judged by the mirror-of-the-self experiment (Alexander 2002–2005, Wu 2015). That is, two images are put side by side and the human subject is asked to choose the one of the two, to which he/she has a better sense of wholesome feeling. The experiment is intended to capture the subject’s genuinely liking from the bottom of his/her heart, so it provides an objective judgement. In other words, the judgement is a kind of human feeling triggered by the underlying living structure rather than idiosyncratic feelings. Recently, eye-tracking and other biometric data (Salingaros, 2020, Lavdas et al. 2021) have been used to capture the kind of objective judgement. Essentially, the livingness of space or living structure in general is a mathematical concept and physical phenomenon, and it can also be reflected in the mind psychologically. Based on the living structure, Salingaros (1997) provided a rough mathematical approximation for computing the degree of livingness of the 24 famous buildings. The line of research can be traced back to the classic work on aesthetic measure (Birkhoff 1933, Eysenck 1942, Douchova 2015) and it continues to be of interest to scientists and scholars; see the recent survey paper (Perc 2020) that has offered a comprehensive overview across a range of the social sciences and humanities.

The livingness of space or structural beauty has significant applications in geography and geographic information science (GIScience). It can help better understand human experience of space such as place attachment and place making (e.g., Tuan 1977, Lewicka 2011, Seamon 2018). This is because the livingness of space or the sense of place is no longer conceived as opinion or personal preference, but a matter of measurable fact. In human geography and GIScience, there is a long history of research interest in humanistic perspectives of geographic space, which requires going beyond the division between science and humanities toward the third culture (Sui 2004, Jiang and Sui 2014, Mennis 2018, Young and Kelly 2017). The recursive approach or the livingness of space in general may offer a new perspective on why natural scenes or living places have healing and nurturing effect on human beings (e.g., Ulrich 1984, Wilson 1984, Kaplan and Kaplan 1989). This of course warrants further research.

This study or the livingness of space in general adds a deep implication on image understanding and computer vision. For example, the morphological skeleton (Umbaugh 2017) and saliency of images (Kadir and Brady 2001, Borji and Itti 2013) can be effectively captured by the centroids of substructures. However, research questions remain regarding how the living structure perspective is comparable to previous approaches to morphological skeleton and saliency maps. While conventional image understanding concentrates on things (objects or features) that human eyes can recognize or natural language can describe with “words” (Deng et al. 2009), the recursive approach to structural beauty captures all substructures that may or may not correspond to human recognizable or language describable things by “words”. The auto-generated substructures, and the recursively generated substructures in particular, capture the wholeness of the image. In this connection, the substructures represent a new way of image understanding. While the human recognized objects or features tend to be fragmented as “words”, the substructures represent a coherent whole or living structure. To make this point clear, let's examine in detail another living image the fishpond image (Figure 9, Table 6) with which structural beauty is calculated from both the figure and the ground.

Different people may perceive this image differently, depending on which objects or features they are familiar with or interested in. The fishpond image contains many objects or features, which are termed as centers (Alexander 2002–2005) or substructures. Identifying these substructures is probably the first step of image understanding, but the livingness of image lies on how these individual substructures constitute a coherent whole. The following passage (Alexander 2002–2005) is a typical description of the livingness of the image.

“Here we have the water, the waves, the fish, the plants, the overhanging bushes, the lilies in the pond, the lily leaves, the mud on the bottom, the caddis, the moss on the rocks, the slime in the water, the flow of water through the pond. Each of these centers, too, is brought to life, and has its life intensified, by the other centers. The fish themselves, for instance, live more intensely when the stream is flowing, so that the dissolved oxygen is constantly replenished. The shade, within the water, formed by the rocks and lily pads, allows the fish a place to cool themselves. The stream flow itself is brought to life, intensified as a center, by the eddies and turbulence at the edge, which make the pond.”

A strong sense of livingness is triggered by the fishpond image, and the livingness can be revealed by the underlying substructures. In fact, there are far more substructures (or centers) than what was described in the above passage through words or phrases such as the water, the waves, the fish, and the overhanging bushes. Many of them are beyond what can be described by any natural language. As shown in Figure 9 and Table 6, there are a total of 25,718 and 26,665 substructures from the figure and the ground, respectively. Among the substructures, there are 200 and 232 decomposable substructures. The describable centers in the above passage, such as the water, the waves, the fish, and the overhanging bushes are just a small subset of the most salient of the massive auto-generated substructures. Those describable ones constitute the surface order, while the massive number of substructures constitute a sort of the deep order. It is the deep order that underlies the quality without a name (Alexander 1979). Put simply, the deep order is the quality without a name.

Any space possesses a certain degree of livingness or structural beauty, although the degree varies from one to another depending on its internal geometry of substructures. Reflected in an image, a substructure is a set of pixels whose pixel values are greater (or less) than the average pixel value. Thus, an image can be viewed as a set of recursively defined substructures, rather than a set of pixels or a set of human recognizable objects as conventionally conceived. The major difference between substructures and objects lies in the fact that substructures are defined by pixels themselves from the bottom up, while objects are defined by human eyes. In this paper, we developed a recursive approach to the structural beauty of images by considering all recursively defined substructures. Through the case studies, we have verified that the recursive approach is better or more robust than the non-recursive approach, although both approaches are based on the same principle: the more substructures the more beautiful, the higher hierarchy of the substructures the more beautiful. We have also verified that the decomposable substructures alone can be used to differentiate two images in terms of their structural beauty, by multiplying the number of decomposable substructures and their levels of recursion. This implies that the more decomposable substructures the more beautiful, and the more levels of recursion the more beautiful.

In addition to the verification of the recursive approach, we have made three major findings. First, the number of substructures of an image is far lower (for example, three percent on average) than the number of pixels, and the centroids of the substructures can capture very well the skeleton or saliency of the image. Second, all the images have the recursive levels more than four, indicating that they are indeed living images. This second finding implies that that any space or matter has a certain degree of livingness or life, according to its internal geometry. Third, no more than 2 percent of the substructures are decomposable, which means that there are far more less-living substructures than more-living ones. To echo the epigraph, we have the following statement about living structure and substructures of an image: in a living structure, every substructure is unique, and the different substructures also cooperate, with no substructures left over, to create a global whole – a whole that can be identified by everyone who is part of it. It is essentially the global whole or wholeness that triggers a sense of livingness in the human mind and heart. The livingness or structural beauty entails that there is a shared notion of livingness among people and even different peoples. It will open a new horizon for research on human experience of space – a sense of places and place attachment – and more importantly on place making. Our future work will concentrate on how the livingness is reflected in the human mind, and whether or how the kind of reflection varies from people to people in terms of their culture, gender, and races.

The data used and generated in this study are publicly available at Figshare, which is accessible with the following link https://figshare.com/s/e4a69ee23511de986239. The file package includes (1) an Excel file with results out of the eight pairs of images' analyses, (2) Shapefiles for all pairs both the decomposable substructures and non-decomposable substructures, and (3) all output results of the living structure algorithm for each individual image. The living structure algorithm is based on Python 2.7 (https://www.python.org/download/releases/2.7/) and uses the following libraries: ArcPy (10.8) and Numpy. (The Python scripts process an input image into recursively defined substructures, and into a tree-like network of decomposable substructures.) The software tools used in the study include ArcGIS 10.8 (https://www.esri.com/en-us/arcgis/products/arcgis-desktop/overview) for the processing and analysis of spatial data, Microsoft Excel (https://www.microsoft.com/en-us/microsoft-365/excel) for storing and analyzing numerical results, and Gephi 0.9.2 (https://gephi.org/) for network analysis and visualization.

The recursive approach developed in this paper was initially verified by ordinary images that all have the same size and same resolution. Given the same condition, the verification should logically apply to any pair of georeferenced images as well. In this Appendix, we chose two pairs of georeferenced images to demonstrate that how the livingness of space can be well captured (Figure A1). The first pair is about two nighttime images: Belgium, the Netherlands, and Luxburg (Benelux, Panel a1) in contrast to Stockholm region (Panel b1). The central European area including the three countries is much more populated than the Stockholm region, so the former is full of human settlements (far more smalls than larges) therefore more living than the latter; see Panels a2 and b2 or a3 and b3. This fact is clearly reflected in the livingness scores both LR and V.

(Note: The source images (a1–d1), their corresponding living structures both decomposable and undecomposable (a2–d2), and the decomposable living structures in a graph format (a3–d3). For the living structures shown in the second row (a2–d2), there are numerous first-iteration substructures (in blue), a very few last-iteration substructures (in red), and some in between the first- and last-iteration substructures (in other colors of the spectrum between blue and red). Some of the substructures are nested to each other, with different colors showing the nested relationship, shown in both the second and third rows.)

The second pair is about two satellite images: one from countryside of Sweden (Panel c1) and the other from the center of Stockholm city (Panel d1). Again, the two areas are with the same physical size and their images are with the same resolution. Different from the nighttime images that capture human settlements at night, the satellite images capture geographic features of various kinds. In this case, the natural scene is supposed to be more living than the urban scene (Panels c2 and d2, or c3 and d3). It is indeed true that the natural scene has higher livingness scores than the urban scene.