Gerard Anthony Sheridan, Carl O’Brien, Bassam A Masri, Clive P Duncan, Donald S Garbuz,Department of Orthopaedics, University of British Columbia, Vancouver V6T 1Z4, British Columbia, Canada

Abstract The demand for revision total hip arthroplasty (THA) is increasing. Information quality on the internet has been extensively analysed in relation to primary THA but no such analysis has ever been performed for revision THA. Our aim was to assess the quality and readability of this information. Three major internet search engines were searched for information on revision THA. All websites were assessed for quality of information using the DISCERN score, the Journal of the American Medical Association benchmark criteria and a novel scoring system specific to revision THA [Vancouver Revision Arthroplasty Information (VRAI)score]. Website readability was assessed, as was presence of the Health On the Net Foundation (HON) seal. The majority of websites (52%) were academic with a post-graduate reading level. Only 6.5% of websites had the HON seal. Twentyeight percent of websites had a ‘good’ DISCERN score and only 28% had a ‘good’score with the novel VRAI scoring system. Health information websites had significantly higher rates of ‘good’ VRAI scores (P = 0.008). Websites with the HON seal had significantly higher DISCERN scores (P = 0.01). All governmental websites were at a reading level suitable for patient review. Information on the internet relating to revision THA is of low quality, much lower than the quality of information on primary THA. We recommend governmental websites for their readability and health information websites for their quality of information specific to revision THA. Websites with the HON seal provide higher quality information and should be recommended to patients as reading material regarding revision THA.

Key words: Revision; Hip; Arthroplasty; Internet; Quality; Readability

INTRODUCTION

Revision total hip arthroplasty (THA) is a commonly performed procedure in orthopaedic practice. The commonest indications for revision THA in modern practice are aseptic loosening, osteolysis and instability[1]. Numerous sources have projected a significant international increase in the demand for this procedure both in Europe and in the United States by the year 2030[2,3]. With the rise in demand for revision THA, we can expect that patients will increasingly reference the internet for sources of information relating to this procedure. Extensive research has already been conducted to assess the quality of internet-based information relating to primary THA[4-7].However, no such analysis has been conducted to assess the quality of information relating to revision THA. The aim of this study is to assess sources of information on the internet relating to revision THA for quality and readability using a number of validated scoring systems, a novel scoring system and statistical analysis.

METHODS

Google, Yahoo!, and Bing search engines were used to browse the internet for the terms ‘Revision’, ‘Hip’, ‘Arthroplasty’, and ‘Replacement’. The search took place on August 29, 2018 at 23.30. Given the 72.68% majority market share of Google, we analysed the first 40 websites returned by the Google search[8]. We then analysed the first 20 websites returned by the smaller search engines on both Bing and Yahoo!. Any website duplicates were excluded from the analysis. All the websites that were reviewed are listed in Supplement Table 1.

All websites were reviewed within 2 wk of the original search by two of the named authors. Each website was allocated to one of the following categories: Academic,commercial, physician, allied-health, media-related, health information website,social/discussion page, governmental, non-profit organisations, and unspecified.These categories have already been regularly used in the literature for assessing information quality on the internet[4,9,10]. Each website was then analysed against a number of scoring systems to assess the quality of the information pertaining to revision hip arthroplasty within.

Each website was assessed for the presence of a ‘Health On the Net Foundation’(HON) seal. The HON seal originated in Geneva, Switzerland in 1995 at the conference entitled ‘The Use of the Internet and World-Wide Web for Telematics in Healthcare’[11]. In an attempt to provide laypeople and medical professionals with reliable sources of healthcare information, this seal was developed for websites that provided users with high quality information.

The next assessment involved the DISCERN tool, as described by Charnocket al[12]in 1999. It comprises 8 questions on reliability, 7 on treatment information and a final question on overall website quality. The maximum score is 80 and a score of greater than 70 is classified as ‘excellent’ while a score of greater than 50 is considered to be‘good’.

The authors developed a novel scoring system to specifically assess the quality and relevance of information regarding revision THA provided on these websites. This 20-point scoring system assesses whether the website addresses the essential preoperative (6 points), perioperative (7 points) and postoperative (7 points) factorsthat patients and healthcare professionals should be aware of when reading information relating to revision THA (Table 1). This score was entitled the ‘Vancouver Revision Arthroplasty Information’ (VRAI) score. A VRAI score of 12 was considered to be ‘good’ and a score of 16 was considered ‘excellent’.

Table 1 A novel scoring system

The interobserver variability was evaluated for both the DISCERN score and the novel VRAI score using Cohen’s kappa co-efficient (κ). This coefficient was described by Cohen in 1960 and has been utilised since then to measure agreement levels between observers for a large range of scoring systems[13]. The below equation demonstrates how Kappa was evaluated, where Po is the observed agreement among raters and Pε is the probability of agreement by chance. When κ = 1, there is complete agreement between the observers: κ = (Po - Pε)/(1 - Pε).

A Kappa value of 0 implies that the scores have no similarity and can be explained by chance. A negative Kappa value implies that the interobserver agreement is worse than what would be expected to occur at random. A good level of agreement determined by the κ ratio is greater than 0.6. Greater than 0.4 is considered to be a moderate level of agreement. Greater than 0.2 is fair and less than 0.2 is poor.Excellent agreement correlates with a κ value above 0.8.

All websites were then assessed using the Journal of the American Medical Association (JAMA) benchmark criteria. These criteria include 4 parameters:Authorship, attributions, affiliations and credentials[14]. It was noted specifically which criteria were fulfilled and which criteria were not. ‘Authorship’ is important in allowing the reader to identify the origin of the information. ‘Attribution’ deals with content referencing. ‘Affiliation’ addresses any potential conflict of interest an author might have and ‘currency’ addresses whether or not the content is current and therefore relevant to the reader.

Finally, every website for inclusion in the study was assessed using the ‘Flesch reading-ease’ test. We defined the readability of each website according to the school grade that the content would be most compatible with. The score was as follows: 5thgrade, 6thgrade, 7thgrade, 8th& 9thgrade, 10th-12thgrade, college student, and postgraduate.

Statistical analysis was performed using STATA? software [Stata/IC 13.1 for Mac(64-bit Intel)]. Fisher’s exact test was used to analyse whether the website type was significantly related to presence of the HON seal and to assess whether the website type was also predictive of a ‘good’ DISCERN score and a ‘good’ VRAI score. The pairedt-test was used to then assess whether presence of the HON seal was predictive of the specific DISCERN and VRAI scores. A one-way ANOVA test was used to assess whether the website type was significantly related to the specific DISCERN and VRAI scores. Linear regression analysis was used to evaluate the relationship between the DISCERN and VRAI scores as they are both interval variables. APvalue of less than 0.05 was taken to be statistically significant.

RESULTS

In excess of 891000 websites returned from the Google search. Eighty websites were assessed in total from Google, Yahoo!, and Bing combined. Of the 80 websites, 18 were dedicated to primary hip arthroplasty exclusively, 15 were duplicates and 1 was dedicated to hip hemiarthroplasty for trauma. Forty six websites were analysed once these were excluded (Figure 1).

Of note, there were no allied-health websites, no media-related websites and no social media websites analysed. Academic websites were much more frequent than other website types (39% of total). There were 18 academic websites, 12 physician, 6 commercial, 6 health information, 3 governmental, and 1 unspecified website (Figure 2).

Health on the net

The HON seal was documented in only 6.5% of websites and was significantly associated with certain website types (P= 0.007). Governmental websites had the highest number of HON seals with two thirds of governmental websites being HON positive. Health-information websites had 1 HON seal out of a total of 6 websites.There were 18 academic websites reviewed in total and none of these had the HON seal.

DISCERN score

The mean DISCERN score overall was 43/80 (σ = 12.7; 8-73 range). Twenty-eight percent of websites (n= 13) had a ‘good’ DISCERN score (＞ 50) whereas only 1 website had an ‘excellent’ DISCERN score (＞ 70). This was a governmental website with a score of 73. The lowest DISCERN score was 8/80 recorded for an unspecified website. It was found that websites with the HON seal had significantly higher DISCERN scores (P= 0.01). The mean DISCERN score for websites with a HON seal was 54 [σ = 13.8; 95% confidence interval (CI): 31.9 to 76.0] compared to a mean score of 41.9 for those websites without the HON seal (σ = 12.3; 95%CI: 38.1 to 45.8).

VRAI score

The mean VRAI score overall was 8.45/20 (σ = 4.5, 0-17). Twenty-eight percent of websites (n= 13) had a ‘good’ VRAI score. Only one website achieved an ‘excellent’score, this was a health information website. Two websites scored 0 including an academic website and a physician website. Health information websites had significantly higher rates of ‘good’ VRAI scores (P= 0.008). Eighty-three percent (n=5) of health information websites had a ‘good’ VRAI score and one of these was considered ‘excellent’. Fifty percent of commercial websites were of ‘good’ quality (n= 3) whereas only 2 of the 18 academic websites had ‘good’ VRAI scores.

Interobserver variability

Cohen’s Kappa coefficient (κ) was calculated for both the DISCERN and the VRAI scores. It was found that the DISCERN scoring system only had a ‘fair’ level of interobserver agreement (κ = 0.37). The VRAI score had a much higher rate of interobserver agreement which was classified as ‘good’ (κ = 0.73). Simple linear regression analysis was performed to compare the Discern and VRAI scoring systems.There was a statistically significant relationship between the two systems (P＜ 0.001).A scatterplot with a regression reference line was also developed (Figure 3). This illustrates how Discern scores tend to increase with an increasing VRAI score,demonstrating the significant relationship between the two scores.

JAMA benchmark criteria

Twenty-eight percent of websites (n= 13) scored 4/4 for the JAMA benchmark criteria. 4 websites scored 0/4. Academic websites had a significantly higher JAMA score compared to other website types (P＜ 0.001). Twelve of the eighteen academic sites scored a JAMA of 4. The remaining websites all scored 3 except for one website that scored a 2. All government, commercial and physician websites failed to score a 4. One health information website scored a 4. Presence of the HON seal was not associated with a higher JAMA score.

Figure 1 Flow chart on study-selection process.

Readability

Using the Flesch reading ease model, it was found that the majority of websites were pitched at a reading level above the 8thgrade (52%). Four websites had a reading level of the 10thgrade. Three websites were written to a college student standard and 36%of all websites (n= 17) were written to a postgraduate standard. The academic websites all had postgraduate reading levels except for one which had a college reading level. Academic websites were found to have a significantly higher reading level than the other websites (P＜ 0.001). All governmental websites were at a 7thgrade level. The majority (10/12) of physician websites were at or below an 8thgrade reading level.

DISCUSSION

Revision THA is a procedure that is increasing in demand as time progresses. Kurtzet al[3]predicted that between 2005 and 2030, the demand for revision hip arthroplasty procedures would increase by 137%. Between 2009 and 2010 in the United States, the total number of revision THAs increased by 10.8%[15]. With the growing demand for this procedure in the future, the public will naturally become more inquisitive regarding the indications, techniques, recovery times and complications associated with revision hip arthroplasty. It is well known that orthopaedic patients are now frequently utilising online resources to research their own conditions[16]. We anticipate that revision THA will become an increasingly searched entity among patients undergoing this procedure in the future. The quality of internet-based information related to primary THA has been extensively described to date[4,5,17]. The purpose of this study was to analyse the quality of the online resources available to patients undergoing revision THA as this has not been evaluated in the current literature to date.

Surprisingly, of the 46 websites that were analysed in this study, not one of them was a social media website. The use of social media by patients by has been shown to improve the doctor-patient relationship by creating more equal communication[18]. It is perhaps concerning therefore that social media are not a prominent source of information for patients undergoing revision THA. It may be the case however, that the higher age profile of this patient cohort is associated with lower levels of social media participation and this is why there is less reference to revision THA on social media sites.

Figure 2 The number of different type of websites.

Academic websites were the commonest website type returned in our search. This is reassuring from a number of perspectives. Academic websites scored the highest consistently on the JAMA benchmark. This means that the source of this information is highly transparent and likely to be current and easily referenced. Unfortunately, the academic websites do have a significantly higher reading level than the other websites that were returned for analysis in the study (P＜ 0.001). Academic websites also tend to have a very specific aim which may not meet the needs of patients looking for general information on revision THA. All of the academic websites were written at a postgraduate level except for one website which was written at a college level. It has been reported that the average United States resident reads at an 8thgrade level and so it is fitting that information intended for public reading should not exceed this level in order to be comprehensible to the vast majority of orthopaedic patients[19].Considering all academic websites were tailored to a much higher reading level, it can be assumed that these websites are not appropriate sources of information for the vast majority of patients. We found that all governmental sources of information had a 7thgrade reading level and so are well suited for relaying information related to revision THR. The majority (10/12) of physician websites also had appropriate reading levels for transmitting information to the public.

The HON seal has been described as a useful tool for physicians to recommend to their patients when searching the internet for information[4]. Previous studies assessing elective orthopaedic information on the internet have found the HON seal present in over 25% of websites[20]. In relation to revision THA information in this study, the HON seal was only found in 6.5% of websites. This demonstrates a much lower standard of quality internet-based information on revision THA when compared to primary THA. Governmental websites had the highest number of HON seals. We found that the HON seal predicts a higher DISCERN score (P= 0.01) and is associated with a higher mean DISCERN score, 12 points higher on average than those websites without a HON seal.

Only 28% of websites were classified as ‘good’ sources of information when using the DISCERN score. When using the novel VRAI score, we also found that only 28%of websites could be classified as ‘good’ sources of information regarding revision THA. Overall, these figures demonstrate a distinct lack of quality information relating to revision THA on the internet. Cassidyet al[21]concluded that the readability and quality of online orthopaedic information is generally poor. We agree with this sentiment as it relates to revision THA and propose that information relating to revision THA is even poorer than information relating to primary THA.

Health information websites appear to be the best sources of information available to patients currently on the internet as they have significantly higher numbers of‘good’ VRAI scores (P= 0.008). Eighty-three percent of health information websites had ‘good’ VRAI scores. The only website to score an ‘excellent’ on the VRAI score was also a health information website. Regression analysis in this study confirms the significant relationship between the validated DISCERN score and the novel VRAI score described (P＜ 0.001). Condition-specific scoring systems are known to be useful in detailing the type of information that should be included by a source providing information to patients on a specific health-related topic[4]. The ‘VRAI’ score described here provides a useful template for websites and other information sources to use if comprehensive information is to be given to patients. This score is significantly related to the validated DISCERN score as demonstrated through regression analysis and it also has a much higher interobserver reliability when compared to the DISCERN score as shown in this study.

Figure 3 The scatterplot with a regression reference line.

We also note that the use of a HON seal is associated with better information quality and higher DISCERN scores. Governmental websites have the most appropriate readability levels on the internet in this field whereas academic websites are not deemed suitable for providing information to patients given the niche focus of their content and the very advanced level of readability. Health information websites are the best sources of information for patients undergoing revision THA as determined by the VRAI score.

This study has a number of limitations. The VRAI score operates on a 20-point scale whereas the DISCERN score has 80. Given the discrepancy between the two scales, it may be more difficult for two observers to agree on a DISCERN score compared to a VRAI score. This may account for the higher interobserver reliability seen with the VRAI score compared to the DISCERN score. We propose that the VRAI score is still a very useful tool in delivering quality information relating to revision THA. The threshold for ‘good’ and ‘excellent’ VRAI scores were arbitrary. It was thought that these were reasonable values to use however, given the maximum score of 20. Only two authors reviewed the websites that were searched, ideally a higher number of reviewers would improve the accuracy of the analysis. Both reviewers were senior orthopaedic trainees with much experience in the quality assessment of health-related information.

CONCLUSION

Information related to revision THA on the internet is of generally poor quality and seems to be of lower quality than information relating to primary THA on the internet. Only 28% of websites had ‘good’ quality information as determined by both the validated DISCERN score and the novel revision THA-specific VRAI score. We recommend that patients use governmental websites as these are the most readable on the internet. We also recommend the use of health information websites as these were of the highest quality overall. Academic websites should be avoided as sources of patient information given their advanced readability and overall lack of patientrelevant content relating to revision THA.