International Journal of Healthcare Simulation

Introduction

Many of the constructivist methodologies and methods used in simulation-based research (SBR) involve coding of text, otherwise known as thematic analysis. Examples of thematic analysis methods used in constructivist SBR include template analysis [1], framework analysis [2], content analysis [3] and reflexive thematic analysis [4]. Methodologies are a system of methods which incorporate philosophical underpinnings, data collection methods and data analysis methods. Methodologies used in SBR that incorporate thematic analysis include [5] constructivist grounded theory [6], ethnography [7] and interpretive phenomenology [8]. While the stand-alone methods allow for the use of pre-defined codes (template analysis, framework analysis and reflexive thematic analysis), all of the methodologies recommend that codes are derived only from the data (constructivist grounded theory, ethnography and interpretive phenomenology). Content analysis is actually a group of methods and the strategy recommended depends upon the specific type of content analysis used.

Within those methods that allow for the use of pre-defined codes, some researchers have elected to use a fully pre-defined theoretical framework to inform the initial codes, some researchers insist that no codes should be pre-defined and others fall somewhere in the middle (using only a few pre-defined codes and deriving others from the data). The extent to which the above-named methods and methodologies allow for the incorporation of pre-defined codes is depicted in Figure 1.

Figure 1:

The extent to which methods and methodologies used within simulation-based research allow for the use of pre-defined codes. In template analysis, the original description suggests that use of a few a priori codes is justified, but within SBR and medical education research there are many examples where researchers use a fully pre-defined theoretical framework. This common usage, beyond the original description, is depicted by the striped portion.

Positioning of the debate within SBR

Examples of some of the approaches to deriving codes for analysis used within constructivist SBR are shown in Table 1.

Table 1:

Examples of approaches to deriving codes for analysis in constructivist simulation-based research

Authors, year, reference	Aim	Stated data analysis methods	Origin of codes
Approaches involving pre-defined codes
Kerins et al. , 2020 [9]	To explore how non-technical skills training might facilitate transformative learning in final-year medical students	Template analysis	Pre-defined codes based on transformative learning theory, modified by the data
Tallentire et al. , 2022 [10]	To use the social identity approach (SIA) as a lens to explore the impact of interprofessional simulation on the identities and professional relationships of trainee pharmacists and medical students	Template analysis	Pre-defined codes based on the social identity approach, modified by the data
Tallentire et al. , 2015 [11]	To explore the subject areas in which junior doctors; acute care errors occur, and how the errors made in each subject relate to the types of error classified in a modified generic error modelling system framework	Framework analysis	Cross-referencing data coded into a set of pre-defined codes (based on a modified generic error modelling system) versus the same data coded into a template derived from the data
Purdy et al. , 2022 [12]	To explore how real-world team psychological safety influences simulation experience, and how simulation impacts real-world team psychological safety.	Thematic analysis	Pre-defined codes based on enablers of psychological safety in healthcare, modified by the data
Approaches using only codes derived from the data
Nestel, 2017 [13]	To explore the editorial themes of a simulation journal	Reflexive thematic analysis	Fully derived from the data, though influenced by educational theory (communities of practice) for meaning-making
Corr et al ., 2017 [14]	To explore medical students’ experiences of living with simulated melanoma	Template analysis (using principles of phenomenology)	Fully derived from the data (template is produced on the basis of a subset of the data, applied to the remaining data and modified as needed)
Cooper-Ioelu and Jowsy, 2022 [15]	To explore experiences of students in an interprofessional clinical simulation learning environment	Thematic analysis (ethnography)	Fully derived from the data
Sullivan et al ., 2023 [16]	To develop a conceptual model of simulated patients as educators.	Constructivist grounded theory	Fully derived from the data
Coutinho et al ., 2016 [17]	To explore nursing students’ perceptions of structured debriefing	Content analysis (constructivist approach)	Fully derived from the data

One of the most cited SBR data analysis methods that uses pre-defined codes is template analysis [1]. In his original description, King describes the use of a template produced using a subset of the data which is then applied to the whole data set and modified as needed [1]. He suggests that a few a priori codes may be included (i.e. codes derived from theory rather than from the data). In later descriptions, template analysis has evolved to incorporate either the use of ‘strong, well defined a priori themes’, or ‘a bottom up approach ’, (i.e. deriving codes from the data) or somewhere in the middle [18]. In medical education research, it is common within template analysis to use a priori themes based on a theoretical framework, and this has also been seen within SBR [9–11], as shown in Table 1.

Pro argument: use of pre-defined codes is best (Samantha Eve Smith)

Social scientists have long been criticized for their tendency to focus on the production of studies without due consideration as to how these studies might build on existing knowledge and add to overall understanding in the field [19]. In 1963, Forscher developed the analogy of throwing bricks at a wall [19], and this analogy has resonated with so many others that it has been cited and added to many times over the years [20]. When deriving codes directly from the data, researchers are at risk of producing yet another ‘brick’ of research to throw on the pile. So, what is the alternative?

I would suggest that the vast majority of constructivist research should be conducted using an explicit theoretical framework. Varpio et al. (2020) define a theoretical framework as ‘a logically developed and connected set of concepts and premises—developed from one or more theories—that a researcher creates to scaffold a study’ [21]. Frameworks have been lauded as useful for both illuminating and magnifying aspects of a problem [22], and it has been suggested that failing to explore all the framework options for a problem can ‘short-change ’ the research [22]. A theoretical framework, incorporating multiple theories if needed, can be created following a literature review, and this is the ideal basis for a set of pre-defined codes. In the SBR examples above, pre-defined codes are based on existing theory from the fields of education (transformative learning theory), sociology (the social identity approach), psychology (Reason’s generic error modelling system) and even from our own field of healthcare simulation (enablers of psychological safety in healthcare).

Constructivism is, by its definition, transactional and subjectivist in its epistemological stance [23]. This means that knowledge is constructed by the researchers through their interactions with the world [23]. Constructivist researchers must therefore incorporate reflexivity within their research [6], i.e. they must consider how their own knowledge and experience might shape the research outcome. I would argue that defining a theoretical framework is part of making explicit the researchers’ knowledge and understanding prior to commencing the research. This seems more efficient than entering a project without reading any of the relevant literature, and more honest than reading the literature but failing to fully acknowledge the influence of prior studies on the researchers’ own work. It is a fallacy to believe that we can start our research ‘stripped of all knowledge’, as proponents of grounded theory might suggest [5]. When a study is explicit about the researchers’ pre-existing knowledge, this should be considered a strength.

Critics of the use of pre-defined codes often state concerns about shoe-horning the data into an ill-fitting framework, or that we might close our eyes to interesting data that does not fit our coding template. I think that these criticisms are based on a misunderstanding. In constructivist research, using a pre-existing coding template does not mean sticking rigidly to it. The template is a starting point, and we can modify the template to better fit the data by adding to it, removing from it, or changing the hierarchy or structure, all of which is encouraged in template analysis [1]. In the example in the table in which template analysis is used to explore medical students’ learning of non-technical skills (now called behavioural skills [24]) transformative learning theory forms the initial coding template. One of the codes from transformative learning theory is ‘self-examination with feelings of guilt or shame’ [25]. These feelings are thought to be important for the process of transformative learning. Within the medical student data set, additional feelings were thought to be important in this process. These included fear, anxiety and frustration [9]. Thus, the use of template analysis allowed transformative learning theory to inform the data, but the data also informed a modification of the theory (i.e. the addition of other emotional states which may invoke the same transformation). It is therefore important that as researchers, we are open to new ideas that do not fit the pre-defined codes. We can enhance this vigilance by specifically seeking alternative codes and by incorporating coding from a number of different researchers with different perspectives on the research question. We must also be willing to throw out a theoretical framework which does not fit, and willing to go back to the drawing board at an early stage in the study (e.g. by carrying out pilot interviews and by conducting data analysis alongside data collection).

In the past, when the field of health professions education was relatively new, many papers used codes derived exclusively from the data, referring to concepts now considered contentious such as ‘emergence of the themes ’ [26]. While SBR remains a relatively young field, it is rare to find a topic so novel that it is not possible to build explicitly on knowledge, be that from the fields of health professions education, quality improvement, clinical medicine, psychology, sociology, education or beyond. I would therefore recommend that, when at all possible, constructivist researchers search the existing literature, construct their own theoretical framework and use this to produce a pre-defined coding template. This will ensure a thorough, honest basis for their research, and will avoid producing just another brick to throw on the pile.

Contra argument: codes should be derived from the data (Victoria Ruth Tallentire)

In contrast to the argument above, I believe that when conducting simulation-based constructivist research, deriving codes exclusively from the data is often preferable to utilization of a pre-existing coding framework. As described in the introduction, some constructivist methods require codes to be derived only from the data as an integral step in their utilization. However, all of the methods and methodologies included in Table 1 include a version whereby codes are derived exclusively from the data, so taking this approach gives constructivist researchers many options. The positive aspects of such an approach are discussed below.

It has been suggested that utilization of a pre-existing framework helps to illuminate or magnify certain aspects of the data set [22]. However, in doing so, the framework also obscures or de-emphasizes other characteristics of the data set. The resultant analysis predictably aligns closely with the framework used for analysis, producing a research echo-chamber that amplifies pre-existing understanding, regardless of how flawed or incomplete that may be. In his well-known book, The Four Agreements, Don Miguel Ruiz says, ‘We only see what we want to see; we only hear what we want to hear. Our belief system is just like a mirror that only shows us what we believe ’ [27]. We must be cautious as researchers to avoid the predictable and pre-formed conclusions that stem from the use of pre-existing coding frameworks, and take care to preserve the richness of the data we have so diligently collected.

In order to truly understand the phenomenon being researched, particularly if it is a novel subject, or being conducted in a novel context, then the data is best viewed from the perspective of the participants. The complexity of human behaviours, based on individual value-systems and experiences, cannot and should not be reduced to fit a unidimensional coding framework. Constructivist methods that value such complexity include phenomenology and ethnography. Phenomenology may be the optimum way to explore the lived experiences of participants, as in the example above in which medical students’ experiences of wearing a simulated melanoma for a day are explored [14]. Ethnography may help us to understand participants’ complex, multidimensional social worlds, such as students’ experiences of the interprofessional clinical simulation learning environment [15]. When utilizing such methods, there is no aspiration to produce results that are transferable to any other group or context, but instead a desire to promote a deep and meaningful understanding of the research topic. In such circumstances, a pre-existing framework is not only inappropriate, it also confines the analysis and limits understanding.

The approaches to data analysis within constructive research that use codes derived from the data have been heavily influenced by the development of grounded theory. The Discovery of Grounded Theory was written by sociologists Barney Glaser and Anselm Strauss in 1967. This seminal work was published as a description of the methodology they had used to study the experiences of palliative patients in hospitals. At the time, qualitative research was not viewed as scientifically credible by the wider healthcare research community. Prior to the publication of this methodology, qualitative theory development of a phenomenon was conceived from a priori knowledge. Grounded theory introduced the new concept of ‘the discovery of theory from data’ [28]. As Kelle describes, ‘One of the main purposes of Glaser and Strauss’ “Discovery book” was to challenge the hypothetico-deductive approach which demands the development of precise and clear-cut theories or hypotheses before the data collection takes place ’ [29]. The concept of allowing theory to develop from within the data is known as ‘emergence’ [29]. In his later work, Glaser argued that a researcher should be able to put aside all preconceived ideas so that they can ‘remain sensitive to the data by being able to record events and detect happenings without first having them filtered through and squared with pre-existing hypotheses and biases ’ [28]. The argument that a researcher should avoid ‘filtering’ the data through a sieve of their own preconceived ideas (manifest as their chosen framework for analysis) remains equally powerful today.

Constructivist grounded theory, as detailed by Kathy Charmaz in her widely utilized book ‘Constructing Grounded Theory’ takes a more pragmatic line, with the acknowledgement that ‘…we are part of the world we study and the data we collect. We construct our grounded theories through our past and present involvements and interactions with people, perspectives, and research practices ’ [6]. However, in contrast to the use of a pre-existing framework that leads researchers down a single track, Charmaz advocates careful consideration of the researcher’s influence on the collection and analysis of the data, in a holistic and reflective way. A recent SBR study [16] (Table 1) has embraced Charmaz’ approach and adopts her recommendation to include a reflexivity statement delineating how the researchers’ own preconceptions influenced the study.

In order that SBR avoids becoming an echo-chamber for pre-existing conceptual ideas and readily available frameworks, I would advocate that more constructivist research is performed using codes exclusively derived from the data, whether as part of an existing methodology or not. By generating more research in this way, the field of SBR will deepen its understanding of the complex emotional and social phenomena at play during simulation-based educational activity, and bring new and exciting ideas to the wider research community.

The middle ground

Analogies abound in our debate, and researchers may feel disheartened when faced with either ‘throwing bricks in the rubble’ or becoming an ‘echo-chamber for pre-existing beliefs’. However, a third option, alluded to within our initial arguments, may give cause for hope. It is possible to consider a hybrid approach to analysis, by either using a pre-defined framework but vigilantly seeking data that does not fit [30], or by deriving the codes directly from the data and performing a post hoc comparison with a theoretical framework [31]. Indeed, it is possible to perform both types of analysis concurrently, thus drawing on the strengths of each [32].

Concluding remarks

Table 1 displays a variety of different SBR topics and approaches. Studies that used pre-defined codes incorporated aims relating to a specific conceptual lens (such as the social identity approach or transformative learning) or were related to a large body of knowledge within the SBR field (psychological safety). Studies that used codes derived exclusively from the data were concerned with a deep exploration of experiences or perceptions (lived experiences), or asked questions for which there was an absence of pre-existing theory (e.g. editorial themes). In the SBR studies displayed here, the study aim has therefore been a crucial factor in determining the type of analysis chosen.

Some methodologies demand a specific approach, whereby codes are derived only from the data. For other studies, researchers are faced with a choice of whether or not to utilize pre-existing frameworks in their coding. The best approach for them may depend on their own philosophical stance, or the experiences of their research team. If researchers hope to draw conclusions that are transferable to other contexts, use of a theoretical framework may be beneficial. For truly novel topics, or for research questions that focus in depth on participants’ experiences, deriving codes exclusively from the data may be the best way forward. Hybrid approaches should also be considered, to utilize the advantages of each type of analysis. Whichever method is chosen, researchers must be aware of the underlying arguments for and against these conflicting approaches, and ‘reflect on the match between purpose, problem, method and tradition of research ’ [33].