Andreas Berner, Tariq Saleem Alharbi,,, Eric Carlstr?m,, Amir Khorram-Manesh,*Prehospital and Disatser Medicine Center, Gothenburg, Region V?stra G?taland, SwedenInstitute of Health and Care Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, SwedenDepartment of Health Services Administration, Faculty of Public Health and Health Informatics, Umm Al-Qura University, Mecca, Saudi ArabiaDepartment of Surgery, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden

STREET: Swedish Tool for Risk/Resource Estimation at EvenTs. Part two, resource assessment - face validity and inter-rater reliability

Andreas Berner1, Tariq Saleem Alharbi1,2,3, Eric Carlstr?m1,2, Amir Khorram-Manesh1,4*
1Prehospital and Disatser Medicine Center, Gothenburg, Region V?stra G?taland, Sweden
2Institute of Health and Care Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
3Department of Health Services Administration, Faculty of Public Health and Health Informatics, Umm Al-Qura University, Mecca, Saudi Arabia
4Department of Surgery, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden

ARTICLE INFO ABSTRACT

Article history:

Received 21 January 2015

Received in revised form 26 January 2015 Accepted 29 January 2015

Available online 13 February 2015

Keywords:

Events

Mass-gatherings

Assessments

Face validity

Inter-rater reliability

High reliability organizations

Healthcare

Rescue

Police

Objective: To develop a validated and generalized collaborative tool to be utilized by high reliability organizations in order to conduct common resource assessment before major events and mass gatherings.

Methods: The Swedish resource and risk estimation guide was used as foundation for the development of the generalized collaborative tool, by three different expert groups, and then analyzed. Analysis of inter-rater reliability was conducted through simulated cases that showed weighted and unweight κ-statistics.

Results: The results revealed a mean of unweight κ-value from the three cases of 0.44 and a mean accuracy of 61% of the tool.

Conclusions: A better collaboration ability and more accurate resource assessment with acceptable reliability and validity were shown in this study to be used as a foundation for resource assessment before major events/mass-gathering in a simulated environment. However, the result also indicates the challenges of creating measurable values from simulated cases. A study on real events can provide higher reliability but needs, on the other hand, an already developed tool.

E-mail: amir.khorram-manesh@surgery.gu.se

1. Introduction

Collaboration between organizations is one key factor in management of major incidents/disasters and crisis[1,2]. The need for such collaboration is more evident at the time of resource scarcity, when the available resources are not enough and when the staff is not trained to handle the incident[3].

In most countries, emergency medical services (EMS), rescue teams (RT e.g. firefighters) and police departments (PD) are those organizations that deal with a large number of emergency responses on a daily basis and consequently collaborate regularly[4-6]. However, Mass-gatherings (MGs) such as sport activities, festivals and musical events have proven to be a challenge beyond the everyday collaboration due to the fact that these large gatherings are exposed to common risks such as collapsing buildings, fire, and trampling, high temperature, storm, and aggressions[7]. The large number of participants in such given scenario will then exceed available resources and the ability of high reliability organizations (HROs) to manage the incident and the immediate consequence of such failure will be a disaster[8-10]. This calls for improving seamless actions, capacity integration and information sharing between HROs as well as organizers in order to estimate resources needed[11].

As suggested by Arbon[7], MGs are characterized by delays in HROs activities because of difficulties in passing in and out from the area. This calls for careful strategies to guarantee sufficient resources. The variation of risks (potential presence of pyrotechnics, fires, weapons, riots), type of arena (in or out door), the environment (urban or rural) and the distance to emergency hospitals together with earlier history of violence, all contribute to challenges, which should be overcome by adjusting HRO resources to the severity of possible incidents. The need of resources can vary between different HROs due to the nature of event and consequent risk assessment. Another major factor with high impact on resource distribution is the presence of non-HRO organization, recruited by the organizers. Such organizations, private companies, may take care of security, technical issues or healthcare. However, the main responsibilities to manage an incident or major incident remain with HRO. In such scenario, the knowledge and information about all available resources and their focus of action will certainly have a high impact on the need of reinforcement.

Estimation of resources and the needed reinforcement is a crucial step in HRO’s pre-planning of an event regardless of the type of event. However, since each organization pre-plans its resources on its own, there is no collaboration in common resource estimation and no ability of cross-organizational cooperation. Collaboration has often been limited to ad hoc actions on the site, when a need of overlapping resources has become obvious[12]. Other has reported inter-organizational passivity due to avoidance to handle needs that is out of scope of present organizations agendas[13]. Consequently, there is a need to start collaboration early to provide common and overlapping strategies in a dialogue between HROs and organizers. A common resource estimation and reinforcement planning will enable unlimited cooperation and increase the ability of using right resources on right area.

Such preparation can potentially distribute resources in anoptimal way. Sharing information before a MG can also benefit the outcome of an incident[14]. When different competences, such as EMS, RD and PD contribute to a common interorganizational strategy the outcome of incidents can be improved[15]. Knowledge communicated by RT about weakness of bleachers in an arena or information from PD about threats aimed at an artist can call for a higher alert on EMS. On the other hand, EMS predictions on the amplitude of medical activities and expected complications during evacuation of victims will provide PD and RD with necessary information on the need of giving support to EMS. Common HRO staff pools, joint unit leadership areas, inter-organizational incident management groups, shared register and triage areas will increase the efficiency during a crisis[5]. Altogether, the need of collaboration in predicting accidents and crisis, stresses the need for collaborative tools that can be used prior and during different types of events. The aim of such a tool is to enhance the collaboration between PD, RT and EMS before and during an event. In this way all partners get an understanding of all conceivable risks from various perspectives and can utilize all available resources to play in a controlled manner.

Table 1Organization related factors of resource assessment.

In an earlier study we presented a guide to assess potential risks before different types of events/MGs[16]. The reason for the development of such a tool, “STREET” (Swedish Tool for Risk/ Resource Estimation at EvenTs), was the evident need for an instrument, which could engage all HROs as well as organizers in a common assessment to predict all possible risks. It consists of two parts, one for risk assessment and the other for resource evaluation.

The aim of this study is to describe the second part of the STREET instrument (Table 1): to develop a validated and generalized collaborative tool to be utilized of HRO in order to conduct common resource assessments before major events and mass gatherings.

2. Materials and methods

This part of the instrument was divided into three dimensions consisting of all three collaborating HROs, i.e. PD, RT and EMS. The response option range was presented as factors influencing the need for reinforcement, measured as factual data or on a three degree scale of none, moderate and high response need[17]. The instrument was filled in by officers from each organization. PD factors were distributed in seven items such as assessed tactical conditions during confrontations, potential presence of weapons, availability of reinforcement, history of the event, potential risk of riots or assaults. RT factors were distributed in eight items, such as manageability during fire, potential occurrence of campfires and pyrotechnics, closeness to seawater and the availability to the area of the MG by heavy fire trucks. There were six EMS items: availability of heart starters, accessibility to disabled, closeness and size of emergency departments, predicted temperature during the event and injury rates from earlier similar events. The items generated figures indicating low, middle or high need of PD, RT or EMS resources. The scale transformation differed between the three HROs. When indexed to a common scale (0-100) 0-33 indicated low, 34-67 middle and 68-100 a high need of reinforcement. The same fictive case reports were used as in the former study[16]. They were chosen from case studies used in the literature[18,19]. The cases were as follows: 1) a concert based on the Bruce Springsteen concert in Gothenburg 2012, visited by more than 50 000, mainly middle-aged spectators; 2) a three days music festival arranged at the countryside, visited by almost 15 000 spectators and 3) a hockey game final from the Swedish championship tour, visited by supporters known as provocative and violent. The hockey arena was fully booked with 12 000 spectators mainly consisting of families.

The HRO-STREET instruments were developed by two Expert Groups I and II, consisting of five academically skilled experts with extensive experience in instrument development (I) and nine senior HRO specialists (PD, RT and EMS) who developed and tested the tool independently and in collaboration. Expert Groups I and II did also test the tool on the fictive scenarios and adjusted the data to current contexts in Region V?stra G?taland, with a population of 1.7 million inhabitants and over 90 events every year of which around 30 are high risk events.

The tool was assessed and the statistical results were based on assessments performed by Expert Group III; 55 experienced HRO staffs that used the tool to assess the fictive case-reports. They were divided into PD staff (n=10, 30%), RT staff (n=10, 30%) and EMS staff (n=13, 39%). Their professional experience ranged from 4 to 42 years in service (18.0±10.2). They ranged in age from 29-64 years (45.0±9.3) and consisted of 73% men and 27% women. Analysis of inter-rater reliability showed un-weighted κ-values[20-22]. According to Altman[23], a κ -value of 0.21-0.40 is regarded as fair agreement and a value of between 0.41-0.60 is regarded as moderate agreement. Good agreement is between 0.61-0.80 and ＞0.80 is considered as very good agreement. Accuracy was given in percentages[22]. Each participant was invited and informed about the study in a personal letter. The aim of the study and the voluntary basis of participation were carefully emphasized. The completed assessments were returned to the first author. One reminder note was sent out after approximately three weeks if no replies were received (Figure 1).

Figure 1. Development of “HRO-STREET”.

3. Results

The mean value of all assessments of the three scenarios was (25.3±8.8) on a scale from 0-100, i.e. 0-33 indicating a low need for reinforcement, 34-67 an average need for reinforcement and 68-100 a high need for reinforcement. There was a difference between the three HROs. The PD showed a steady need for reinforcement, irrespective of the scenario (m=32.1-32.8). The RT demanded more resources for the festival (35.2± 6.1) than the other two scenarios. EMS had substantially higher need of resources for festival (47.6±10.5) than the concert (3.3± 9.8) and the hockey game (6.7±9.9). The festival was generally considered to be more demanding and in need of extra resources of the three scenarios presented (38.3±7.5) (Table 2).

Table 2Assessed need of resources.

Data are expressed as mean±SD. 0-33: Low, 34-67: Middle, 68-100: High.

The tool showed a mean accuracy of 61%. Two of the cases (concert and festival) showed a substantial to moderate accuracy of 68% and 63%. The hockey game displayed a weak accuracy (52%). The accuracy differed between the three professions. The tool appeared to show higher accuracy when used by the EMS (67%) than the other organizations (RT 63% and PD, 53%) (Table 3).

Table 3Accuracy of the tool I scenario cases

The mean κ-value of the three case-reports was calculated as a linear unweight κ-value. In total it showed an interrater reliability of κ=0.44, i.e. moderate agreement. The κ -values did vary between the participating organizations. The instrument displayed κ=0.50, moderate agreement, when it was used by the EMS, it was followed by RT (κ=0.45, moderate agreement) and police (κ=0.25, fair agreement).

4. Discussion

Cooperation between three main actors i.e. police and rescue/fire departments and healthcare may lead to a more accurate risk and resource assessment, saving lives and money[24]. In the earlier study, we presented a validated and generalized collaborative tool “STREET” for risk assessment prior to an event to be used of all HRO partners together with organizers[16]. In this study, we describe the second part of the tool, which will guide HROs through more accurate resource estimation. The results revealed a better collaboration ability and more accurate resource assessment expressed by mean of unweight κ-value from the three fictive cases of 0.44 and a mean accuracy of 61% of the tool.

By using the first part of STREET, consisting general questions, all partners had the possibility to do a risk assessment based on the available information from organizers[16]. In this study we used the second part, consisting of specific questions, that makes an impact on all HRO’s partners and enable them to do a resource assessment for the need of each organization, respectively. By focusing on the resource needs, we identified a clearer diversity in organizational assessment and confirmed our belief in a common assessment tool that offers, every organization, relevant information about how other partners have assessed the situation. The second part of STREET elucidates specific needs in each organization, while the final outcome from each group can be compared to the rest of partners.

Similar to the earlier study (Part one), we used three fictive scenarios, i.e. a festival, a concert and an ice-hockey match[16,18,19]. In the first scenario, the resource assessment by each organization during a festival, resulted in a need for extra resources, however, there was a difference in HRO’s assessment in remaining two scenarios (concert and icehockey match). For the EMS, the estimation of extra resources was influenced by the time it took to get to the event and back to the hospital, if anything happened, while RT foresaw the risk for fire and PD took all threats into consideration. However, all partners needed to consider the time it took to get to the event. This is, thus, indicative for EMS to have extra resources on place. Besides the distance, the geographical location of the event has also more importance for EMS than other partners. This is clearly seen in the assessment of other scenarios (concert and ice-hockey match), when EMS foresaw less needs for extra resources, since these events were held closer to the hospitals (shorter distance).

Part two of the STREET template enables all organizations to deeply engage themselves into the specific organizationrelated risks and estimate the needed resources to handle possible scenarios (fire, threats, healthcare for RT, PD, and EMS, respectively). Therefore, it is obvious that there will be differences in their assessment; however, STREET makes it possible for HROs to adjust their needs much more adherent to the reality. In our earlier study[16], we could show that PD’s risk assessment is somehow closer to the reality and stands between assessments conducted by RT and EMS. Since, PD is much more aware of all risks, it is thus logical to assume that their assessment should be used as foundation for resource estimation. In this way, the needed resources in both EMS and RT organizations would be highly affected. The STREET tool clarifies the need for collaboration between PD, RT and EMS in a joint meeting, in which the event is overviewed in all perspectives and the result will be an adjusted resource estimation which will improve the outcome.

As it was shown in our earlier study[16], the use of this collaboration tool (STREET), needs to be clarified by clear instructions to its users, since there might be a difference in individual assessment within each organization. However, this difference would be diminished with gaining experience. Thus, repeated utilization of the tool increases the accuracy of the assessment. In addition, it is of high importance that all assessments by STREET, is conducted by experienced individuals within each organization, since the outcome of the template and the final evaluation is based on each representative’s knowledge about his/her organization, organizational capacity and limitation. Repetitive and constant utilization of the tool would also increase the user’s self-confidence in each organization. This calls for monthly strategic meetings between HROs to constantly evaluate coming events by using the template (Part I and II), based on available information from organizers. In many of these meetings, the final outcome, will probably lead to an adjustment of resources in all organizations.

The advantage of using simulated cases is comparability since all participants receive similar information. The use of simulated case-reports is also common when instruments are being assessed, particularly when investigating accuracy and inter-rater agreement is on target[25-30]. However, simulated cases do not offer all real facts and information that can be presented in a real environment[31].

The STREET collaboration tool showed acceptable reliability and validity to be used as a foundation for resource assessment before major events and mass gatherings in a simulated environment. The STREET template enables collaboration between HROs. It gives an insight in each organization’s capacity and limitations and has a potential to offer accurate final resource estimation before any major event. However, this study also indicates the challenges of creating measurable values from simulated cases. A study on real events can provide higher reliability but needs, on the other hand, an already developed tool.

Conflict of interest statement

The authors report no conflict of interest.

Acknowledgements

The authors would like to thank all the participants of the different expert groups contributing to the development of the instrument during the pilot study. Special thanks to Mats Kihlgren, director and Dr. Per ?rninge, medical advisor at PKMC for their support and contribution to this study.

References

[1] Powly EH, Nissen ME. If You can’t trust, stick to hierarchy: structure and trust as contingency factors in threat assessment contexts. J Homel Secur Emerg Manag 2012; 9(1): 1-16.

[2] Clarke S. Safety leadership: a meta-analytic review of transformational and transactional leadership styles as antecedents of safety behaviors. J Occup Organ Psychol 2013; 86 (1): 22-49.

[3] Scholtens A. Controlled collaboration in disaster and crisis management in the Netherlands, history and practice of an overestimated and underestimated concept. J Conting Crisis Manag 2008; 16(4): 195-207.

[4] Lateef F. Simulation-based learning: Just like the real thing. J Emerg Trauma Shock 2010; 3(4): 348-352.

[5] Berlin J, Carlstr?m ED. From artefact to effect: the organizing effects of artefacts on teams. J Health Organ Manag 2010; 24(4): 417-427.

[6] Andersson A. Carlstr?m ED, ?gren B, Berlin JM. Managing boundaries at the accident scene - a qualitative study of collaboration exercises. Int J Emerg Serv 2014; 3(1): 77-94.

[7] Arbon P. Mass-gathering medicine: a review of the evidence and future directions for research. Prehosp Disaster Med 2007; 22(2): 131-135.

[8] Larsson L?, Norstedt SA. [The fire in Gothenburg 1998. A study on communication, rumours and confidence]. Stockholm: National Board of Psychological Defense (Rapport 179); 2000. Swedish.

[9] Suserud BO. How do ambulance personnel experience work at a disaster site? Accid Emerg Nurs 2001; 9(1): 56-66.

[10] Meites E, Brown JF. Ambulance need at mass gatherings. Prehosp Disaster Med 2010; 25(6): 511-514.

[11] Perry RW. Disaster exercise outcomes for professional emergency personnel and citizen volunteers. J Conting Crisis Manag 2004; 12(2): 64-75.

[12] Taber N. Emergency response: Elearning for paramedics and firefighters. Simul Gaming 2008; 39: 515-527.

[13] Berlin J, Carlstr?m E. [What is really exercised during collaboration exercises? Effects of first initiative and parallelism]. Nordiske Organisasjonsstudier/Nordic Organization Studies (NOS) 2008; 4: 23-39. Swedish.

[14] Deverell E. [Crisis management and learning: Factors that influence authorities’ ability to learn]. Kungl Krigsvetenskapsakademiens Handlingar och Tidskrift 2012; 216: 117-130. Swedish.

[15] Berlin J, Carlstr?m E. 2013. The mechanistic dominance - a critical study of crisis exercises. The dominance of mechanistic behaviour: a critical study of emergency exercises. Int J Emerg Manag 2013; 9(4): 327-350.

[16] Berner A, Alharbi TS, Carlstr?m E, Khorram-Manesh A. STREET: Swedish Tool for Risk/Resource Estimation at EvenTs. Part one, risk assessment - face validity and inter-rater reliability. J Acute Dis 2015; 4(1): 37-43.

[17] Haugen Ohlsson M, Siwerstam L. [Development of a questionnaire for participation. A pilot study]. H?gskolan V?st. Institutionen f?r omv?rdnad h?lsa och kultur; 2011. Swedish. [Online] Available from: http://www.diva-portal.org/smash/get/diva2:606085/ FULLTEXT01.pdf [Accessed on 24th January, 2015]

[18] Berner A, Khorram-Manesh A, Hedelin A, ?rtenwall P. Avoiding disasters, a tool for estimating the needs of healthcare resources at sporting and other public events. ICU Management 2011; 3: 38-39.

[19] Khorram-Manesh A, Berner A, Hedelin A, ?rtenwall P. Estimation of healthcare resources at sporting events. Prehosp Disaster Med 2010; 25(5): 449-455.

[20] Stemler S. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. PARE 2004; 9 (4). [Online] Available from: http://pareonline.net/getvn. asp?v=9&n=4 [Accessed on 24th January, 2015]

[21] Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med 2005; 37(5): 360-363.

[22] Polit D, Beck CT. Nursing research: generating and assessing evidence for nursing practice. 8th ed. Philadelphia: Lippincott Williams & Wilkins; 2008.

[23] Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991.

[24] Coleman L. Frequency of man-made disasters in the 20th century. J Conting Crisis Manag 2006; 14(1): 3-11.

[25] Dilley S, Standen P. Victorian nurses demonstrate concordance in the application of the National Triage Scale. Emerg Med 1998; 10: 12-18.

[26] Gould D, Goldstone L, Gammon J, Kelly D, Maidwell A. Establishing the validity of pressure ulcer risk assessment scales: a novel approach using illustrated patient scenarios. Int J Nurs Stud 2002; 39: 215-228.

[27] Offredy M. Decision-making in primary care: outcomes from a study using patient scenarios. J Adv Nurs 2002; 40(5): 532-541.

[28] Considine J, Ung L, Thomas S. Triage nurse’s decisions using the national triage scale for Australian emergency departments. Accid Emerg Nurs 2000; 8: 201-209.

[29] Olofsson P, Gellerstedt M, Carlstr?m ED. Manchester Triage in Sweden-interrater reliability and accuracy. Int Emerg Nurs 2009; 17(3): 143-148.

[30] G?ransson K, Ehrenberg A, Marklund B, Ehnfors M. Accuracy and concordance of nurses in emergency department triage. Scand J Caring Sci 2005; 19: 432-438.

[31] Johansson C. ?str?m S. Kauffeldt A, Carlstr?m E. Daily life dialogue assessment in psychiatric care-face validity and interrater reliability of a tool based on the International Classification of Functioning, Disability and Health. Arch Psychiatr Nurs 2013; 27(6): 306-311.

doi:Document heading

*Corresponding author:Amir Khorram-Manesh, M.D., Ph.D., Prehospital and Disaster Medicine Center, Gothenburg, Region V?stra G?taland, Sweden.