Why Human Touch Still Matters in Automated Process Testing | HackerNoon

Emerging patterns in the coded items can be brought together to form tool and methodological suggestions. This analysis of patterns in the coded interview data can be seen as an application of the GT phase advanced coding [4]. In the following, we present the inferred suggestions and concepts in more detail. These serve as both concrete suggestions and examples of how the requirements can be used when implementing AB-BPM tools.

Integrated Process Model Repository. The idea of integrating the ABBPM tool with a process model repository (PMR) came up in one interview. It is implicitly supported by statements of most of the other study participants. The main aim of IPMR is to provide a fast and efficient way of introducing new process variants to process participants while allowing them to provide feedback. Once a new business process variant is added, the responsible process expert would need to add additional material to help process participants understand the new version. Suppose a specific process participant is part of an experimental process case (or a new version is permanently rolled out). In that case, they would receive a notification (e.g., via email) with the most important information regarding the process update, with a link to more material. This could be extended with the need for process participants to pass a short test to avoid improvement effort failures due to misunderstanding. They could also provide feedback, which could then be evaluated by the process experts or even considered for the RL experiment.

Instance Recovery Mechanism. The analysis of the interviews made it clear that the danger of failed process instances has to be mitigated in some way. Based on that data, we propose an instance recovery mechanism (IRM). Before starting an experiment, process experts could set thresholds for specific KPIs. The expert would be notified when a process instance reaches such a threshold. They could then intervene in the process execution to make sure that the process consumers still get their desired product or service. The thresholds should be set relatively high, to only intervene in erroneous cases. Otherwise, it could create a bias in the experiment. This is why one also has to consider how such instances are evaluated in calculating the rewards of the RL. Leaving them out of the RL calculations could lead to problems: Imagine only one process version sometimes exceeds the thresholds and has to be salvaged manually. By leaving these instances out, we might misjudge that version. A better approach could include these manually salvaged versions in the model with a certain penalty value. Considering the scientific state-of-the-art, IRM is closely related to the notion of (real-time) business process monitoring, as for example covered in [18,10].

Another pattern that can be observed throughout the interviews is the need for more human control. This can be seen in the perceived risks (e.g., BLE, LRE), as well as in the desired tool features (e.g., EES, CPS, PSN, MRL, XRL). In a previous paper, we have already presented an initial prototype that enhances the AB-BPM methodology with additional features for human control [13].

The presented study has several limitations. One possible threat to the validity of results, especially the ranking, is that we employed a shortened RTDM. Usually, there are multiple ranking rounds until the concordance among the experts has reached a satisfactory level. As mentioned above, we decided not to conduct multiple ranking rounds due to the exploratory nature of the study and the extensive interview process, which makes it difficult to be certain in the ranking. Preliminary statistical analysis of the rating differences between the items shows that there is no significant difference between the listed risk items, but there is a significant difference between the ratings of the different feature items. However, given the qualitative nature of the study the rankings should only be seen as rough guidance. The focus of the work is the qualitative elicitation of items. This has been taken into consideration for the discussion and conclusion, meaning that we took all the items into account, not just the most highly ranked.

Another possible threat to the validity of the results is that the participation in the validation round dropped from ten to five. We tried to mitigate validity issues by ensuring higher participation in the ranking round (eight experts participated) and giving study participants the option to also give feedback on the coding in that round. Since no more remarks were made about the coding, we conclude that the coding was satisfactory for all eight study participants of the ranking round.

Furthermore, all study participants are employed by the same company. We tried to attenuate this by selecting experts with extensive experience from various teams and backgrounds. Additionally, the consultants brought in their experience with business process improvement projects from additional companies.

The main aim of this study was to obtain practitioners’ perspectives on the ABBPM method. Using mostly qualitative research methods, we shed light on the requirements for the further development of AB-BPM tools and the underlying method. Overall, the study participants perceived the methodology as advantageous in comparison to the status quo. The three main conclusions of the study are: i) more possibilities of human intervention, and interaction between the RL agent and the human expert, are a core requirement,; ii) transparency and features for the participation of process participants are needed to make AB-BPM culturally viable; iii) integrated process execution is necessary to facilitate the seamless deployment of parallel process variants and deliver the real-time data needed for dynamic RL and routing. The openness of the semi-structured interviews facilitated the discovery of future research opportunities, e.g., studying companies carrying out unstructured process tests in a production environment, and tool-driven training of process participants. While we focused on a single process improvement method, practitioners’ insights on different kinds of business improvement methods [15] could provide insights for studies that compare AB-BPM to other methodologies.