Enhancing Scenario-Based Modeling Using Large Language Models

Chat Generative Pre-trained Transformer (ChatGPT), by OpenAI [7, 43], is a large language model (LLM) based chatbot. ChatGPT is sufficiently powerful to conduct an iterative conversation of variable format, style, length, level of detail, and language. At each stage of the conversation, the user presents ChatGPT with a fresh prompt, and the chatbot then replies—based on all previous prompts that appeared in that conversation (also referred to as the context). Since its debut in 2022, ChatGPT has quickly become profoundly successful, and has inspired several other companies to create their own chatbots [3, 13, 42].

Internally, ChatGPT is comprised of a proprietary series of generative pre-trained transformer (GPT) models. These models are, in turn, based on Google’s transformer architecture [46]. ChatGPT has been fine-tuned for conversational applications, through a combination of reinforcement and supervised learning techniques, followed by manual adjustments by human engineers. The chatbot’s training, and also its inference, are considered very costly in terms of processing resources and power consumption.

As far as functionality is concerned, ChatGPT is extremely versatile. Some of its numerous uses include writing and debugging computer programs [45], generating student essays [2], and also composing music [40]. However, it may sometimes produce plausible-sounding but completely incorrect answers—which is a known limitation for large language models [16].

Scenario-based modeling [33] (SBM) is an approach for modeling complex, reactive systems. The main building block in a scenario-based (SB) model is called a scenario object, and it describes a single behavior of the system being modeled—whether desirable or undesirable—which can be specified as allowed, necessarily, or forbidden. Each of the model’s scenario object does not interact with its counterparts directly, and is created in isolation. Cross-scenario interaction is supported, but only through a global execution mechanism, which may execute a set of scenarios in a way that produces global, cohesive behavior.

Multiple flavors of SBM have been proposed, each with its own mechanism for cross-scenario interactions. Here, we focus on a particular set of idioms, which are common in many of the SBM frameworks: the requesting, waiting-for and blocking of discrete event sets [33]. As the mode is executed, each of the scenario objects repeatedly visits designated synchronization points; and in each of these points, the global execution mechanism triggers a single event. Each scenario object may declare sets of events that it wishes to see triggered (requested events), events that it would like to avoid (blocked events), and events that it does not request itself, but which it would like to monitor (waited-for events). The global execution mechanism then collects these declarations from each of the scenario objects in the system (or, possibly, a subset thereof [19]), selects a single event that is requested and not blocked, and then broadcasts this selection to all relevant scenario objects.

In each synchronization point there may be multiple events that are requested and not blocked; and multiple strategies have been put forth for selecting one of these events. Possible strategies include arbitrary or random event selection, a round-robin mechanism, and also look-ahead schemes that simulate the possible progression of the execution, and then select events in order to optimize an objective specified a-priori (e.g., the avoidance of deadlocks). Executing a scenario-based model in this way is termed play-out [31]).

Figure 1 illustrates a simple SB model. It represents a system that controls the water level of a water tank, which is equipped with cold and hot water taps. Each of the scenario objects is depicted as a transition system, wherein nodes correspond to the predetermined points of synchronization. The ADDHOTWATER scenario object repeatedly waits for events of type WATERLOW, and whenever such an event is triggered, the scenario object requests three times the event ADDHOT. Symmetrically, the ADDCOLDWATER scenario object requests the addition of cold water. When the SB model includes only the objects ADDCOLDWATER and ADDHOTWATER, three ADDCOLD events and three ADDHOT events may be triggered, in any order, during the model's execution. If maintaining a more stable water temperature within the tank is important, this could be achieved by adding the scenario object STABILITY, which enforces the interleaving of ADDHOT and ADD COLD events—through the use of the event blocking idiom. An execution trace of the SB model that contains all three objects appears in the event log.

Scenario-based modeling has been implemented on top of several high-level languages, such as C++ [21], Java [32], JavaScript [4], Python [47], and ScenarioTools [14]. Moreover, scenario-based modeling has been applied to various complex systems, including cache coherence protocols [23], a web-server [21], and robotic controllers [17]. To keep the presentation simple in the coming sections, we will usually describe SB models as transitions systems.

We follow here the formal definitions of SBM as they appear in [36]. A scenario object $O$ over event set $E$ is given as a tuple $O\; =\; \backslash langle\; Q,\; \backslash delta,\; q\_0,\; R,\; B\; \backslash rangle$, where the components are defined as follows:

A set of individual scenario objects can be composed, in a pairwise fashion, according to the rules that appear below. Let $O^1\; =\; \backslash langle\; Q^1,\; \backslash delta^1,\; q\_0^1,\; R^1,\; B^1\; \backslash rangle$ and $O^2\; =\; \backslash langle\; Q^2,\; \backslash delta^2,\; q\_0^2,\; R^2,\; B^2\; \backslash rangle$ be two scenario objects, specified over a common event set $E$. These two objects are composed into a single scenario object $O^1\; \backslash parallel\; O^2\; =\; \backslash langle\; Q^1\; \backslash times\; Q^2,\; \backslash delta,\; (q\_0^1,\; q\_0^2),\; R^1\; \backslash cup\; R^2,\; B^1\; \backslash cup\; B^2\; \backslash rangle$, where:

Through the $\backslash parallel$ composition operator, we define a behavioral model $M$ as a set of scenario objects, $M\; =\; \backslash \{O^1,\; O^2,\; \backslash dots,\; O^n\backslash \}$. The executions of model $M$ are defined to be precisely the executions of the composite object $O\; =\; O^1\; \backslash parallel\; O^2\; \backslash parallel\; \backslash dots\; \backslash parallel\; O^n$. Consequently, each execution of $M$ starts from $O$'s initial state, which, in turn, is the $n$-tuple of initial states of the individual objects of which $O$ is comprised; and throughout $M$'s run, in each state $q$ one enabled event $e\; \backslash in\; R(q)\; -\; B(q)$ is selected for triggering, if such an event exists. The execution then transitions to a state $\backslash hat\{q\}\; \backslash in\; \backslash delta(q,\; e)$, and the process is repeated.

As the first step towards integrating SBM and ChatGPT, we present a straightforward methodology for generating scenario objects from free-text, using ChatGPT. To get ChatGPT to present its output in scenario object form, we propose to include, in the prompts presented to the chatbot, a preamble that describes SBM's semantics, as presented in Sect. 2. This preamble is used to instruct ChatGPT to produce its output as transition systems, wherein each state is labeled with requested/blocked events; and has also a set of transition rules that indicate its successor state, for each event that could be triggered. Note that such a preamble only needs to be designed once, and that it can then be used with any subsequent query. Whenever a specific SBM implementation is in use (e.g., C++, Java), the chatbot's output can be adjusted to match the required syntax of that implementation.

The simple preamble, which we used in our some of our initial experimentation, is as follows:²

I would like you to help me create a scenario-based model. A scenario-based model is comprised of a finite set of events, and scenarios. Each scenario is a sequence of states. In each state the scenario can request events, wait-for events, and block events. Afterwards, one event that is requested by at least one scenario, but is blocked by no scenario, is selected for triggering. Scenarios that requested or waited-for this event may then change states, and the process repeats itself. Scenarios cannot have internal loops or variables; these must be expressed using additional states.

To display scenarios, please use the following format. For each state, mention the state's name, and the sets of requested and blocked events. Also, mention the transitions to other states. For example:

Through this preamble, we could get ChatGPT to generate the model we described in Sect. 2. For the ADDHOTWATER scenario, we used the following prompt:

Consider the event set WaterLow, AddHot, AddCold.

Please suggest a scenario that waits for event WaterLow (without requesting it). When WaterLow is triggered, the scenario performs the following, 3 times in a row: request the event AddHot, and wait for it to be triggered (both in a single state). When the 3rd AddHot event occurs, the scenario transitions back to its initial state.

The chatbot then replied to this prompt as follows:

Which is indeed equivalent to the transition system that appeared in Fig. 1. The ADDCOLDWATER scenario could also, be generated using the symmetrical prompt. Next, if we wished to incrementally modify the system through the introduction of the STABILITY scenario, we could issue the final prompt:

Please suggests a scenario that uses blocking to ensure that no two consecutive AddHot events can be triggered, and that no two consecutive AddCold events can be triggered; that is, once AddHot is triggered, AddCold must be triggered before AddHot can be triggered again, and vice versa. This scenario should not request any events, and should work regardless of any WaterLow events.

And in response, ChatGPT would produce the STABILITY scenario, precisely as described in Fig. 1.

We point out that there is a subtle difference between the prompts we used to obtain the first two scenarios, ADDHOTWATER and ADDCOLDWATER, compared to the prompt used to obtain STABILITY. For the first two cases, our prompt included information that approximately described the transition system that we desired; whereas in the last case, our prompt was more high-level, and did not contain the word "state". Still, in all cases, the chatbot successfully produced the desired outcome. This further demonstrates the wide specification range that ChatGPT can successfully handle; and it suggests that a chatbot can be used even in cases where the engineers themselves are not entirely certain of what the scenario that they desire looks like. While it is reasonable to assume that the more accurate description, the more accurate result, it seems that even high-level descriptions can prove useful, especially when enhanced by techniques for automated analysis, as we discuss next.

Further extending the basic integration between SBM and ChatGPT, we now propose an outline for a structured, language-agnostic and LLM-agnostic methodology for creating complex models of reactive systems that interact with their environment repeatedly, and may receive external inputs [35]. Many critical, modern systems can be regarded as reactive [1], and as a result there has been extensive work on devising methods and tools for modeling such systems. In spite of this tremendous effort, there still remain significant gaps; and these could result in models that are either inaccurate or difficult to maintain, or both. In this paper, which can be regarded as an element within the Wise Computing vision [25], we seek to mitigate these gaps, by creating advanced and intelligent tools, which will begin to undertake system development tasks that are traditionally reserved for humans. The approach is based on having system components generated, incrementally and iteratively, through the use of an LLM; and to have the LLM's outputs checked systematically, and semi-automatically, using various methods and tools (see Fig. 2).

Next, we elaborate on some of these steps, and provide illustrative, simple examples.

Code generation is one of the most straightforward capabilities of chatbots that we propose to integrate into a system’s development cycle. In Sect. 3 we demonstrated that ChatGPT can generate an executable, scenario-based model—and similar capabilities have been demonstrated with other languages [6, 39, 45]. One advantage in the context of scenario-based systems is that it is possible to generate stand-alone scenarios, which can then be tested and reviewed separately, and later be added, incrementally, to the system at hand. In our preliminary experimentation for this paper, we tested code generation for requirements in the realms of algorithms on data structures, autonomous vehicles, control systems, and simulating natural phenomena. For each of these realms, the chatbot/SBM integration proved useful.

Once ChatGPT successfully digested the underlying principles of SB models, it was indeed successful in combining logic with its knowledge of the problem domain and of the world at large, in order to enhance or develop an SB model. For example, it was successful in introducing new environment events, in describing the sensor scenarios required for triggering such events, and also in adding the corresponding application scenarios that react to these events. In one case, we considered a robot moving numbered tiles on a sequence of cells, and asked the chatbot to generate SB code for a bubble-sort algorithm to be used by this system. ChatGPT was able to introduce events corresponding to the arrival of a tile at the tail-end of the array, and it was also able to generate scenarios for reacting to such events.

After a few attempts, we succeeded in getting ChatGPT to produce execution logs for arbitrary SB specifications. Initially, we observed “wishful thinking”, where the chatbot would describe an execution as it should be per the problem description. By iteratively refining our prompts, as illustrated in Fig. 3, we eventually got ChatGPT to follow the execution steps correctly and to display the event and transition leading to each synchronization point; as well as a table of all scenarios, indicating for each scenario its event declarations and whether or not it had requested or waited-for the event that was eventually triggered.

A key advantage of SB modeling is its amenability to automated formal verification, and the availability of tools that perform this—either through exhaustive model checking that traverses all paths, or by using a domain-specific

Even in cases where an existing model addresses all stated requirements perfectly, it is possible that there are additional areas of the broader problem domain that are currently not handled properly—possible due to synthetic or reduced problem descriptions. When presented with appropriate prompts, ChatGPT was able to significantly improve an already developed model, or, conversely, to help in explicitly stating the expected scope and boundaries of the intended system.

In one example [18], we presented the chatbot with an SB model that had been developed for science teaching purposes. The model contained information on how bees depend on the ambient temperature in order to switch between being active and hibernating; how flower plants rely on hours of daylight in order to start blooming; and how active bees depend on blooming flowers for their survival. In the original model, the number of daylight hours and the average temperatures were mutable parameters, which students could adjust in order to check how certain combinations of settings could sustain the bees, whereas other settings could cause the bees to die out due to hunger.

When we prompted ChatGPT for possible enhancements to the education model, it returned with a suggestion that we consider also weather-related variations, as these might modify the pre-specified fixed number of daylight hours and average temperature for a given time of year.

Once the chatbot is provided with a certain expected or observed behavior of the system, it is often able to explain the cause of that behavior, in natural language. Further, it is often able to reasonably balance between providing the overly detailed, complete execution trace, and the overly simplistic explanation that “this is what the system is supposed to do”.

In one case we provided the chatbot with details on the semantics of SB modeling, and then prompted it as follows (borrowed from [26]):

In a system with two scenarios, one waits for time tick events and after each time tick requests a short beep (100 ms). The other scenario waits for time ticks, and after each such event requests a long beep (500 ms). Each of these scenarios waits for both the short and long beep events, so that after each time tick, exactly one beep occurs. Actuator scenarios actually cause the respective beeping action. Time tick events are injected by the environment every one second. A user reported that it watched the system for 20 s, and never heard a long beep. Can you explain why this may have happened?

ChatGPT’s response was:

The user’s report of not hearing a long beep in a system with two scenarios—one requesting a short beep after each time tick, and the other requesting a long beep after each time tick—can be explained through the principles of Scenario-based Modeling (SBM).

In SBM, when multiple scenarios request different events after a shared trigger event, the execution environment selects one of these requested events to occur. The selection can be arbitrary or follow a specific strategy, such as alphabetic or lexical order, priority, random, round-robin, etc. In this case, if the event selection strategy prioritizes the short beep over the long beep (for example, due to alphabetic order, where “short beep” comes before “long beep”), then the short beep will always be selected to occur after each time tick, and the long beep will never be executed. Since time tick events are injected by the environment every second, and both scenarios are waiting for this event, each time tick leads to both scenarios being activated. However, if the system consistently selects the short beep event over the long beep event after each time tick, the scenario requesting the long beep will never progress to the state where it actually triggers the long beep. As a result, only short beeps will be heard.

In summary, the user's observation of not hearing any long beeps can be attributed to the event selection mechanism within the SBM framework, which in this case consistently favors the short beep event over the long beep event. This could be due to the selection strategy used by the system (like alphabetic order) or other operational logic specifics.

We then challenged the chatbot again:

Thank you. Assume that the developers changed the selection algorithm and they now select events randomly. The user tested the system many times, for hours, and never heard a long beep. Can you try to explain?

And in response to this, ChatGPT returned with a highly comprehensive answer, which covered many possible issues with the execution infrastructure, human-machine interactions, and the application scenarios. Here are a few excerpts, borrowed from [26]:

If the event selection algorithm was changed to randomly select between the short beep and long beep events, but the user still never hears a long beep, there are several potential explanations:

Implementation Error in Random Selection: The most straightforward explanation could be a bug or implementation error in the random selection algorithm. It might not be truly random, or it could be incorrectly favoring the short beep event due to a coding mistake.

Issue with the Long Beep Scenario: There might be a problem with the scenario that requests the long beep. This could be a coding error or a logical flaw in the scenario itself, preventing it from properly requesting the long beep event.

User Perception or Reporting Error: Finally, there could be an issue with how the user is perceiving or reporting the beeps. They might be missing the long beep due to distraction, ambient noise, or misunderstanding what constitutes a long beep in the context of this system.

To diagnose the issue accurately, developers would need to conduct thorough testing and debugging, looking into each of these potential areas to find where the problem lies.

Although these explanations were quite impressive, they were also imperfect. In some cases, we presented the chatbot with well-specified scenarios accompanied by a detailed description of an observed event sequence, but ChatGPT was unable to describe a step-by-step execution (play-out) of the scenarios that would result the given event sequence; instead, it provided incorrect explanations for how the event sequence might occur. Such issues could sometimes be resolved by repeatedly providing ChatGPT with reminders and pin-pointing its mistakes. This is a significant caveat that serves as a reminder that outputs produced by the LLM must be formally inspected and verified.

Many software engineering teams are tied to specific programming and modeling languages, and their associated semantics. In our experimentation, ChatGPT was in many cases able to discuss and accommodate alternative semantics, demonstrating flexibility that human engineers sometimes lack.

For example, consider again the water tap example, first described in Sect. 2. Observe the ADDHOTWATER scenario: when it is in any of the states where it requests ADDHOT, it is not waiting for event WATERLOW, and consequently cannot react to its triggering. However, if we consider the semantics of the Live Sequence Charts [9], which is another SB formalism, every scenario constantly waits for all events that were waited-for in its starting state. When such a waited-for event occurs, the LSC infrastructure instantiates a fresh copy of the scenario. As it turns out, from our initial textual description of SBM, ChatGPT concluded that this semantics were the default.

In another example, we tried to get ChatGPT to generate a scenario-based implementation of Quicksort. Before starting, the chatbot remarked that this will be hard, because classical solutions are recursive. To assist the chatbot, we then pointed out that there was actually an alternative, published implementation for Quicksort, which was iterative instead of recursive [34]. Further, this implementation was structured as a set of instructions to human workers that needed to arrange cars in an automobile dealership parking lot, according to window-sticker prices. Each employee in this implementation was assigned one, narrow role, which coincides nicely with how SB modeling works. ChatGPT readily accepted this new “mindset”, and was able to produce the sought-after SB specification.

Through our experimentation with the proposed approach, we observed that ChatGPT could often achieve better results when it was provided with multiple prompts, explorations and discussions, compared to when it was provided with a single, concise, detailed description. We believe that this might be a general rule worth following when a chatbot is used as part of the software and system development cycle. Thus, this style of work might be a good fit for an agile development process, which inherently involves a great deal of trial and error as it spirally converges towards the specified (or, sometimes, under-specified) goal. From the chatbot’s point of view, this is a highly constructive process, in which developers and stakeholders iteratively describe their wishes and plans, as they slowly refine their own understanding of the system and its environment, as well as their own needs and intended future interactions with the system.

An integral part of this refinement is the production of increasingly explicit definitions of elements that are within the scope of the system’s environment.

Such definitions, which may be highly useful to both chatbots and humans, are often quite absent from classical system specifications.

Given an LLM-generated component, generating test cases based on the original specification is a straightforward activity. Furthermore, it is only natural to ask the LLM to generate the code for such a test case. However, the generated programs, or scenarios, must be tested also from additional perspectives. Specification and execution of these tests may be carried out both with and without the help of the LLM.

To illustrate these additional perspectives, consider the following examples.

We have asked ChatGPT to produce a stand-alone Python program that flashes a green light for half a second every 3 s, i.e., in seconds 0, 3, 6, 9, etc., when counting time from zero. ChatGPT returned an acceptable program. When asked to generate a test case, it created a reasonable program that examined an execution log and confirmed that indeed one would observe repetition of periods of half a second of the green light being on, followed by 2.5 s of that light being off. However, as may often happen, if the on and off periods are not exactly 0.500 and 2.500 respectively, but say, 0.501 and 2.501, the program may still pass the test, but over time, the timestamps of when the light is flashed will be quite far from a multiple of 3. This is another property that must be checked as it was in the specification. Generating the code for this test may be done by a human or by ChatGPT (of course, the test case must be checked in either case).

One technique for generating new perspectives is to generate alternative designs and implementations, and examining the differences. We have asked ChatGPT to generate code for a robot to navigate a two dimensional grid of cells from an origin cell to a destination cell with steps of right, left, up and down, and where some cells are marked as obstacles, and can be sensed only from an adjacent cell. We then asked ChatGPT to generate a materially different program. ChatGPT generated a breadth-first search, replacing its choice of depth-first search in the first program. When asked about the externally visible effects it listed resource usage and performance aspects, opportunity for parallelization, and more. We then clarified to ChatGPT that the grid data structure is not to be used as a map in advance route planning, and that the scenarios should simulate actual movement, including back-tracking. While such simulation seems very similar to a depth first search, the path that the program outputs would be different: in the simulated traversal, the backtracking steps are part of the path, where in the search they are only part of the planning until a path with no backtracking is found. A developer can examine these issues and determine whether any of them should be included in the specification or in the testing or formal verification of the component.

Another approach for generating new perspectives does not involve code generation, but explores possible behaviors. Going back to the light flashing program from the first example in this subsection, we asked ChatGPT whether a program could comply literally with every aspect of the specification, and still produce undesired results. In its first response, ChatGPT suggested that a program that flashes a red light between consecutive flashing of the green light may comply literally with the specification, but may not behave as expected. Similarly ChatGPT suggested that if the device producing the green light also creates a sound, the behavior may be undesired. The developers can then consider whether to enhance the specification accordingly to avoid these cases, and/or whether to create test cases, using cameras, microphones, and appropriate software, to check that indeed the behavior of the system in the world is as intended.

The properties to be checked can be formulated by the LLM in natural language, as executable test cases, or as we show below, as formal assertions for verification tools. For example, we prompted ChatGPT to analyze the following scenario: An autonomous vehicle (the ego car) is at a red traffic light. It senses that car B, that was approaching fast behind it, noticed the red light late, and hit its brakes. B is screeching to a halt. It is not clear whether B will be able to stop before it hits the ego car. Further, parts of the intersection are “clear”. With some interaction, ChatGPT produced reactive code for the ego car to move forward when both needed and possible. Clearly, it is not enough that the ego car should eventually reach a point beyond the stopping point of car B, and we asked ChatGPT to refine and formalize this requirement. Referring to variables in the program, it offered two variants, one for SPIN/Promela:

and one for NuSMV:

The use of an LLM in building software applications is already wide-spread, and will likely grow, in analysis, requirement specification, design, coding, testing, etc. The methodology discussed here focuses particularly on the development task (which is largely code development), as contrasted, say, with requirement specification or integration testing; it can be employed within an agile-development sprint, and within other approaches to the system development cycle.

What should a prompt that requests an LLM to generate one scenario look like?

Ideally, one would like to specify the desired function in a natural manner, as in “Do not allow the robot to move into a grid cell that is marked as an obstacle”, and let the LLM take care of the details of SBM. Alternatively, one could specify “Please create a scenario that, before the robot moves out of a grid cell, checks the cells to the right, left, above and below the current cell, and if any of those adjacent cells is an obstacle, block the respective, Right, Left, Up, and Down move events, until the robot exits the current cell.”. However, if the LLM is able to add fine details to the actual code, it means that the code is more distant from the requirement specification, and may contain decisions and choices that a domain specialist may not be aware of. For example, following multiple prompts, the LLM may make application-related assumptions (as in the case of carrying out a map-based search instead simulated traversal as described in Sect. 4.9). Of course, such LLM decisions would turn out in code review, and then either be accepted or rejected by the developer. Clearly, in a methodology that aims for intuitive executable specifications, the natural language prompts should be as close as possible to the actual code. Hence, when the LLM indeed enhances the generated module beyond the original prompt specification, the developer should update the prompt to either request or exclude these enhancements specifically.

One could also start with a functional description of a scenario, and once this is complete, after several iterations, have the LLM generate, in addition to the SBM code, a natural language description of the scenario. In addition to serving as documentation, the description may serve as an LLM prompt in another scenario-generation iteration (which of course must be tested).

The output of an LLM is never predictable. Small and focused changes in a prompt, or new prompts requesting a small change in an existing code block may cause larger, or unpredictable, changes in one or more scenarios. Our conclusion is that with each iteration of producing one or more artifacts one should (a) compare the result with prior artifacts; and (b) run tests and a comprehensive manual (i.e., human-driven) review over a larger set of artifacts and components. Tools that readily compare and display all differences between multiple LLM-generated code blocks would be very useful for this purpose. It is not enough to observe the desired change in the expected location in the code, or the changed behavior in the problematic section in an execution log. While this is an intuitive methodological requirement and common practice, recall that the entire methodology is about the coding stages in application development and not about application testing stages.

As stated earlier, scenarios implicitly start with the word “always”, as in: “Always when event E1 occurs, request event E2”. However, this universal rule may have exceptions that depend on the context, i.e. a variety of system and environment conditions. For example, when hurrying to a hospital, transporting a patient in a critical condition, on a deserted street, when no other vehicle is in sight, a human driver may (cautiously) run a red light, and an autonomous, SBM-based vehicle might need to do the same. Or, in our grid traversal example, the robot creates its own context, and the path should not loop into itself. Should the necessary information be tracked in each scenario separately? Should there be a data structure that is shared between the scenarios? Or, should the environment inject external sensor events to indicate relevant information?

There has been research about context oriented programming in general [8], and implementing context awareness in behavioral programs in particular [11]. In the present example, we chose to have several shared data structures for this purpose, but the more general question of what are the best ways to weave context dependencies SB designs, and therefore into LLM prompts, is still open.

The LLM may generate incorrect code even for the most precise prompt. Here are a few examples from our experiments: (i) when checking certain properties of adjacent cells, the scenario used the previous location of the robot (which was accessed for another function in the same scenario) instead of the current location; (ii) when needing to block multiple events, the scenario incorrectly issued several different YIELD commands instead of accumulating multiple blocked events in a set, and passing the set to the single yield; (iii) an IF statement that should have controlled only a few code lines, instead dominated and conditioned the rest of the scenario; (iv) a scenario that should have had a single YIELD command, waiting each time for the next move of the robot, incorrectly waited for the move both at the beginning and the end of each of its iteration. These (and other) errors were caught in testing, and the code was corrected using revised or new, corrective prompts. However, since these errors were not a result of prompt vagueness or ambiguity, one may ask how the principle of not manually modifying the code, and only changing the prompt, should be applied in this case. One approach could be to add these originally corrective actions as emphases and notes in the prompt; such notes may be reminiscent of when an experienced programmer explains a code module to a novice, sharing stories about past bugs and omissions, to emphasize the need for close attention to details.

It is often very tempting to change the code directly and not via an LLM prompt. Two examples from our experiments are changing a printed string, from “CURRENT LOCATION” to “CURRENT LOCATION:”, or inserting a PRINT() in a particular point for debug purposes. Unfortunately, such manual code changes are lost the next time the scenario is generated using the LLM, or, conversely, they cause the abandonment of the methodology.

While in SBM scenario specifications can align with requirement specifications, our experiments show that in incremental scenario development this is not always the case. For example, initially, we separated the three functions of physically moving the robot, updating the robot's location, and printing the robot's actual path, into separate scenarios. To enforce a particular event order, we combined these three functions into one scenario (an alternative solution was to keep the scenarios separate and rely on event blocking). To generate the scenario, we prompted the LLM to create one scenario that consolidates the functions of these three scenarios in a certain order, without repeating the details of each operation. In another case, ChatGPT, on its own, added the logic that prevents the robot from stepping outside the grid boundaries, to the scenario that checks for obstacles; this created a redundancy with the scenario that was responsible for checking for grid boundaries; hence, we prompted ChatGPT to delete the unneeded scenario.

We also occasionally prompted for application-wide changes affecting all scenarios, such as changing the parameters to the YIELD command from the format that appears in the SBM preamble that we used (borrowed verbatim from [48]), to the format required by the BP Python execution environment of [47]. The entire chat dialog can then serve in having the LLM generate a concise description of each of the scenarios in the final application.

One of the characteristics of agile approaches to software development is the emergence of new requirements as part of iterative sprints. In the first block of code for the robot-on-grid application, the coordinate system for grid cells that ChatGPT chose was different from how the user tacitly visualized them: (row,column) as is common in matrix description versus (horizontal,vertical), as in an (x,y) Cartesian system; in the former, a move up increases the second coordinate, and in the latter, it decreases the first coordinate. This choice, or decision, by the LLM, for an application detail that was not specified, triggered the documenting of a new requirement.

Some of the dialogs with ChatGPT were on specific programming style issues, like whether to rely on particular semantics of the Python language (e.g., when the compiler or interpreter exits the computation of an OR condition when one of the sub-conditions is computed to be true), or when and how to use temporary variables for streamlining the program's appearance and flow. This question is outside of the scope of the current methodology; the LLM may be taught to comply with the developer's preferences as a rule, as opposed to dealing with specific changes to already-generated code blocks.