Assessment of quality and readability of internet-based health information related to commonly prescribed angiotensin receptor blockers
Anuoluwapo Oloidi1,&, Sabina Onyinye Nduaguba2, Kehinde Obamiro3
1Medplus Health and Beauty, Saka Tinubu, Lagos State, Nigeria, 2Health
Outcomes and Pharmacy Practice 'Division' College of Pharmacy, University
of Texas at Austin, USA, 3Centre for Rural Health, College of Health
and Medicine, University of Tasmania, Australia
Anuoluwapo Oloidi, Medplus Health and Beauty, Saka Tinubu, Lagos State, Nigeria
hypertension is a global public health burden. Angiotensin receptor blockers (ARBs) have proven efficacy in the management of hypertension and related complications. The Internet has become a major source of health information for patients and healthcare professionals. The study aimed to assess the quality and readability of internet-based information related to selected Angiotensin Receptor Blockers (ARBs).
the three most widely used ARBs were identified from published literature, after which internet-based patient information was identified from the first five pages of three search engines (Google, Yahoo and Bing). Quality of identified websites were assessed using the DISCERN instrument, while readability was evaluated using the SMOG instrument and the Flesch Kincaid readability algorithm. Final ratings were then calculated as described by the instruments developers. Further, inter-class correlation coefficients (ICC) were calculated using the Statistical Package for Social Sciences.
the average overall DISCERN score in this study was 2.99 (SD±1.05). No website received an excellent rating, 15% were rated good, 66% as moderate and 19% as poor. The inter-class reliability for losartan was 0.804 and 0.695 for valsartan. The mean Flesch Reading Ease score for the websites was 48.87 (SD±16.12), mean Flesch Kincaid Reading Grade Level was 9.29 (SD±1.98) while mean SMOG value was 11.29 (SD±1.70).
overall, patient information on the reviewed ARBs websites was found to be of moderate quality and suboptimal readability. Content providers on websites should ensure that health information is of favorable quality and easy to read by patients with varying degree of health literacy.
The internet has rapidly become the largest source of information
worldwide  and an important source of health information
both for patients and health professionals . With
the advent of the internet, medical information is easily assessed with
the click of a mouse for several disease conditions requiring short-term
or long-term therapy . Studies have showed that
in Canada and the United States, more than 80% of the population have access
to the internet , and about 72% of United State users
seek information over the internet . Studies have
also showed that health information are among the most sought after topics
on the internet , not necessarily as a means of replacing
advice given by health professionals, but for validation of information
given and to gather additional information . Increasingly,
patients and health professionals are turning to the internet for information
pertaining health challenges and complicated therapies .
One of such health challenges is hypertension which is a major risk factor
for Cardiovascular diseases (CVD) . In a study conducted
in 2013 at the Pew Research Center, 45% of adults in the United States reported
that they live with one or more chronic conditions including hypertension,
lung conditions, diabetes and heart disease . Sixty-seven
percent of those living with hypertension were internet users, and 58% of
them accessed websites that provided information about a specific medical
condition or problem . Hypertension is a global
public health burden that contributes to morbidity, mortality and healthcare
cost in both developing and developed countries .
Approximately 25% of hypertensive patients worldwide are taking ARBs, and
about 20 million people worldwide take ARBs for both hypertension and other
cardiovascular conditions . The effectiveness and
safety of ARBs are well established . However,
its use has been associated with a number of adverse drug events, especially
in elderly patients. Providing quality patient information and education
is therefore important for safe and effective hypertension management. Despite
its potential as a significant patient information resource, the internet's
usefulness is often limited by the challenges associated with finding good
quality information that comes from reliable and authentic sources .
Previous studies have reported that more than half of websites provide poor
quality health information . Health professionals
are therefore constantly faced with patients who have been informed or misinformed
by the internet . As a result, clinicians, researchers,
and consumers are concerned about the quality and accuracy of online health
information and it is essential to assess the validity .
As much as disseminating health information on the internet can help improve
knowledge transfer from health professionals to the population and help
individuals maintain and improve their health, the rapid development of
medical information on the Internet has its shortcomings which include:
1) uneven quality of medical information available on the internet ;
2) difficulty in reading and understanding this information due to use of
technical language ; 3) potential dangers related
to its erroneous and unsuitable use . In addition
to addressing the issue of quality, accuracy and reliability, it is also
imperative to systematically assess the presentation of online health information
using readability algorithms to ensure that such information is easily read
and understood. We therefore aimed to assess the quality and readability
of Internet-based health information related to select Angiotensin Receptor
Identification and selection of websites
We identified the three most widely used Angiotensin Receptor Blockers from published literature: losartan, irbesartan and valsartan [21
]. Each of these key terms ‘losartan', ‘irbesartan', ‘valsartan' and ‘patient education' were entered into three different search engines (Google, Yahoo, Bing) chosen based on their widespread use [22
], in March 2017. Only the first 50 links reported by each search engine were identified, as studies have showed that web users only visit the top 10 websites listed in search results [23
Websites were included in the research if they were in English Language and free to access, and if they provided patient information relating to any of the search terms. Websites designed for marketing purposes evident by presence of diverse advertisements were excluded [24
]. News feeds, video feeds, abstract listings and duplicate websites were also excluded. Additionally, websites for co-formulated ARB products were excluded. Quality of identified sites was evaluated using the DISCERN questionnaire, while readability was assessed using the SMOG calculator and Flesh-kincaid algorithm.
Assessment of quality of information using DISCERN instrument
The DISCERN questionnaire is a valid instrument that was designed to enable patients (or health consumers) and health care providers to assess the quality of health information. This questionnaire was developed based on the input of an expert panel, health information providers and representative of patient population [25
]. The DISCERN tool is freely accessible online and a downloadable instruction handbook is available from the DISCERN handbook [26
]. It is suitable for anyone who utilizes or produces information about treatment choices. Its uses include an aid for individual consumers who are making decisions about treatment, a screening tool for health information providers, a checklist for authors and producers of written consumer health information, a training tool for health professionals to improve communication and shared decision-making skills.
The DISCERN questionnaire, consisting of 16 questions, was used to evaluate the quality of information on the selected websites. These questions are organized into three sections [26
]. Section 1 (questions 1 to 8) addressed the reliability of the publication and helps in considering whether it can be trusted as a source of information. Section 2 (questions 9 to 15) focused on the specific details of the information about treatment choices while Section 3 (question 16) assessed the overall quality rating. Each question was rated on a 5-point scale ranging from "no" to "yes". A score of "5" indicates a definite yes; a score of "2-4" indicates that the publication meets the criterion in the question to an extent; while a score of "1" indicates a definite no.
It is generally accepted that in evaluating the readability scores of written
information, the use of more than one readability formula improves the reliability
of the readability scores [27
], hence two readability formulae;
SMOG formula [28
], and Flesch-Kincaid algorithm [29
were used in this study. The Smog readability formula was created by G Harry
McLaughlin in 1969 and it estimates the years of education a person needs to
understand a piece of writing [28
]. SMOG readability grades
were measured using the manual SMOG formula [28
]. The SMOG
was then calculated as described by McLaughlin et al.
The readability scores were calculated using Microsoft Office Word. The Flesch
Reading Ease test rates the text on the given website on a 100-point scale. The
higher the score, the easier it is to understand the document. The Flesch-Kincaid
Grade Level test rates a text on a United States school grade level. A score
of 8.0, for example, means that an eighth grader can understand the document.
standard data entry and analysis were done using a Microsoft Excel spreadsheet (edition 2013). Inter-class Correlation Coefficient (ICC) value was calculated using the Statistical Package for Social Sciences, version 23 (IBM, Armonk, New York, US).
A total of 450 websites were reviewed between April 2017 and September 2017. A total of 150 websites for each of the three search engines (Google, Yahoo and Bing). Removal of 363 duplicate websites left a total of 87 unique sites. Thirty-seven out of the 87 unique websites met the inclusion criteria and were eligible to be evaluated. The remaining 50 websites were excluded as 11 were not freely accessible, 2 had broken link and 37 did not provide information useful for patient education. The authors further streamlined the search results leaving only websites with patient information on losartan and valsartan considering they are the most frequently prescribed ARBs as reported in the literature . This left a total of 24 websites for evaluation. The Inter-class reliability for losartan and valsartan was calculated to be 0.804 and 0.695, respectively (Table 1).
Assessment of quality of websites
From the evaluation conducted using the DISCERN tool, none of the websites scored 5 for question 16 (the overall quality rating at the end of the questionnaire), 15% were rated as good, 66% as moderate, while 19% were rated as poor. The overall mean DISCERN score was 2.99 (SD±1.05). The mean quality rating across the websites is shown in Figure 1
. From a maximum score of 5, the mean score for the questions 1 to 8 that addressed reliability was 2.82 while for questions 9 to 15 that focused on information regarding treatment choice was 3.20. The questions with the highest response score were as follows: "Does it describe risk of each treatment?", "Does it describe the benefit of each treatment?", "Does it provide support for shared decision-making?" and "is it relevant?" On the other hand, the lower scoring questions are: "Is it clear what sources of information are used to compile the publication?", "Does it describe what would happen if no treatment is used?" and "Does it describe how the treatment choice affects the overall quality of life?" The highest-rated websites according to our analysis are Mayo Clinic (Valsartan) 3.84 and Patient Info (Losartan) 3.84 [31
Twenty-four websites were assessed for readability Table 1
. The mean Flesch Reading
Ease Score (FRES) was 48.87 (SD±16.12), while the mean Flesch Kincaid
Reading Grade Level (FKRGL) was 9.29 (SD±1.98). The mean SMOG value was
calculated to be 11.29 (SD±1.70).
The internet has the potential to provide patients and health professionals with health information, but with the increase in use, concerns arise as to the quality, reliability and readability of the information obtained from the internet . In addressing these concerns, several solutions have been proposed. These solutions have included electronic filtering of web-based information, creation of ethical codes of conduct for providers of web-based information (currently done on voluntary basis) and assessment of websites by health professionals . Introduction of clear critical appraisal tools with standardized website evaluation systems also appears to be useful in improving the ability of users to differentiate between trustworthy sites and inadequate ones .
This study is the first to have systematically evaluated the quality and readability of internet-based information related the two mostly prescribed ARBs. The results from this investigation provide some insights regarding online health information related to the ARBs. The overall mean DISCERN score for the 26 websites was 2.99 (SD±1.05). This suggests that information on ARBs on most of the evaluated websites was of moderate quality. This analysis is consistent with findings from previous studies which have evaluated the quality of health information on the Internet for a range of different chronic diseases (Table 2
With respect to the quality of health information, most of the websites assessed scored reasonably well in describing the benefits of treatment as well as the risks of using ARBs. Most websites evaluated were also balanced and unbiased, and providing support for shared decision making between patients and physician. This is in agreement with the DISCERN criterion which suggests that a good quality health information resource should include issues for patients to discuss with their healthcare practitioners. However, most of the websites fared poorly in identifying clear sources of information or references, as only five websites met this criterion. This is important because clear sources enable online users examine the credibility and reliability of information on the website or decide to seek further information.
There was consistently a lack of information on the likely consequence of no treatment and how treatment choices affect the overall quality of life. Hence, patients receiving ARB therapy would potentially have poor knowledge on this aspect of their therapy. From the present study, certain websites are Mayo Clinic (Valsartan) 3.84 and Patient Info (Losartan) 3.84, received the highest DISCERN score [31
]. These websites were seen to have clearly met a good number of the DISCERN criteria.
The Inter-class reliability determined in the study was 0.804 for losartan and
0.695 for valsartan. Ideally a value of 0.7 is considered optimal [37
Therefore, the values gotten indicated a good level of consistency for quality
rating measurements between independent raters. This study highlights that most
of the websites had patient information that is potentially difficult to read,
as most of the websites were written at readability grade levels higher than
grade 8 (Table
). Of note, the SMOG rating scale was observed to provide a measurement
of 2-4 grade levels higher than the Flesch Kincaid grade Table
. This is considered
the result of variation between different measurement scales and is consistent
with studies conducted by Wilson, 2009 [38
]. However, for
both scales, most websites consistently scored above "9" indicating the universality
of the readability problem.
The suboptimal readability observed in the present study is consistent with those of the broader literature. Estrada et al.
, reported that patient education materials related to the use of anticoagulants were written at grade levels beyond the comprehension of most patients [39
]. Hutchinson et al.
, have also evaluated websites containing medical information pertaining to nine common general medicine diagnoses, and have reported similar readability levels beyond grade 8 for many of the websites evaluated [40
]. Our finding is particularly relevant because hypertension is common in the elderly, and its prevalence increases with aging which has been associated with decline cognitive function [41
]. As such, patients on ARBs might find it difficult to read and understand ambiguous online materials. Hence, patient information on these sites should be written at approximately school grade 8 or less to enhance easy understanding. The Institute of Healthcare Improvement recommends using simpler words, shorter sentences and avoiding medical jargon, all of which are important considerations in the provision of online health information for public consumption [42
The present study is not without a number of limitations. Only the first 50 search results per search engine were reviewed, and only the two mostly prescribed ARBs were assessed. Further, only websites written in English were evaluated, hence, findings may not be applicable to other languages. The Flesch Kincaid readability tool used may have overestimated the required readability scores as a list of polysyllabic clinical and medical terms such as ‘angiotensin' and ‘aldosterone' may be regarded as one sentence. Finally, readability formulae used was limited by the lay out of the website page. Websites vary dramatically in how they present information separately from the text that is written. Despite these limitations, the present study provides evidence on suboptimal readability and the need for quality improvement for online ARB patient education material.
While the overall quality of health information on the selected ARBs is moderate, the websites were found to possess a suboptimal readability score. Future content developers of ARB patient education material should ensure that health information is of favorable quality and easy to read by patients with varying degree of health literacy.
What is known about this topic
- The internet is an important source of health information;
- Health information on the internet varies widely in quality and a significant number of online health information are written at inappropriate reading levels, which could limit their usefulness.
What this study adds
- Health information on the reviewed ARBs websites was found to be of moderate quality with written language above recommended reading level;
- Improving the quality of health information available to patients will potentially lead to better treatment outcome.
The authors declare no competing interests.
KO, AO and SN conceived the idea and were involved in the design of the study. AO collected the data and together with KO and SN participated in data analysis and interpretation. AO did the drafting of the paper while KO and SN critically revised the manuscript. All authors have read and agreed to the final manuscript.
Tables and figure
readability scores of ARB websites (N=24)
comparison of different DISCERN scores for different studies
category breakdown of readability scores of ARB websites (n = 24)
Figure 1: mean quality rating of websites using the DISCERN instrument
- Fahy E, Hardikar R, Fox A, Mackay S. Quality of patient health information on the Internet: reviewing a complex and evolving landscape. The Australasian Medical Journal. 2014;7(1):24. PubMed | Google Scholar
- Lacher D, Nelson E, Bylsma W, Spena R. Computer
use and needs of internists: a survey of members of the American College
of Physicians-American Society of Internal Medicine. Proceedings of the
AMIA Symposium; 2000: American Medical Informatics Association.
- Kiran DS, Bargale S, Pandya P, Bhatt K, Barad N, Shah N et al. Evaluation of Health on the Net seal label and DISCERN as content quality indicators for patients seeking information about thumb sucking habit. Journal of pharmacy & bioallied sciences. 2015;7(Suppl 2):S481. PubMed | Google Scholar
- Centre PR. Global
Attitudes & Trends. 2016. Accessed on February 17 2019
- Zickuhr K. Who's not online and why. 2013.
- McMullan M. Patients using the Internet to obtain health information: how this affects the patient–health professional relationship. Patient education and counseling. 2006;63(1-2):24-8. PubMed | Google Scholar
- Nicholas D, Huntington P, Gunter B, Withey R, Russell C. The British and their use of the web for health information and advice: a survey. Aslib Proceedings; 2003: MCB UP Ltd. Google Scholar
- Nasser S, Mullan J, Bajorek B. Assessing the quality, suitability and readability of internet-based health information about warfarin for patients. The Australasian Medical Journal. 2012;5(3):194. PubMed | Google Scholar
- Ong KL, Cheung BM, Man YB, Lau CP, Lam KS. Prevalence, awareness, treatment, and control of hypertension among United States adults 1999–2004. Hypertension. 2007;49(1):69-75. PubMed | Google Scholar
- Pew Research Centre. Chronic disease and the internet. 2010. Accessed on February 17 2019
- Kearney PM, Whelton M, Reynolds K, Muntner P, Whelton PK, He J. Global burden of hypertension: analysis of worldwide data. The lancet. 2005;365(9455):217-23. PubMed | Google Scholar
- Volpe M, Azizi M, Danser AJ, Nguyen G, Ruilope LM. Twisting arms to angiotensin receptor blockers/antagonists: the turn of cancer. European heart journal. 2010;32(1):19-22. PubMed | Google Scholar
- Corrao G, Zambon A, Parodi A, Poluzzi E, Baldi I, Merlino L et al. Discontinuation of and changes in drug therapy for hypertension among newly-treated patients: a population-based study in Italy. Journal of hypertension. 2008;26(4):819-24. PubMed | Google Scholar
- Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. Jama. 2001;285(20):2612-21. PubMed | Google Scholar
- van der Marel S, Duijvestein M, Hardwick JC, van den Brink GR, Veenendaal R, Hommes DW et al. Quality of web-based information on inflammatory bowel diseases. Inflammatory bowel diseases. 2009;15(12):1891-6. PubMed | Google Scholar
- Kaba R, Sooriakumaran P. The evolution of the doctor-patient relationship. International Journal of Surgery. 2007;5(1):57-65. PubMed | Google Scholar
- Price SL, Hersh WR. Filtering web pages for quality indicators: an empirical approach to finding high quality consumer health information on the world wide web. Proceedings of the AMIA Symposium; 1999: American Medical Informatics Association. 1999;911-915. Google Scholar
- Impicciatore P, Pandolfini C, Casella N, Bonati
M. Reliability of health information for the public
on the World Wide Web: systematic survey of advice
on managing fever in children at home.
BMJ. 1997;314(7098):1875. PubMed | Google
- Cline RJ, Haynes KM. Consumer health information seeking on the internet: the state of the art. Health education research. 2001;16(6):671-92. PubMed | Google Scholar
- Allain P, Schuck S, Beaufils C, Zekri O, Ganilsy M, Allain H. Drugs on the Internet. Hazards of public health. Presse medicale (Paris, France: 1983). 1998;27(3):117-21. PubMed | Google Scholar
- Hudson M, Humphries K, Tu JV, Behlouli H, Sheppard R, Pilote L. Angiotensin II receptor blockers for the treatment of heart failure: a class effect? Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy. 2007;27(4):526-34. Google Scholar
- Search Engine Watch. Nielsen netratings search engine ratings. 2016. Accessed on February 17 2019
- Eysenbach G, Köhler C. How do consumers search
for and appraise health information on the world wide
web? Qualitative study using focus groups, usability
tests, and in-depth interviews. BMJ.
2002;324(7337):573-7. PubMed | Google
- Moody E, Clemens K, Storsley L, Waterman A, Parikh C, Garg A et al. Improving on-line information for potential living kidney donors. Kidney international. 2007;71(10):1062-70. PubMed | Google Scholar
- Khazaal Y, Chatton A, Cochand S, Coquard O, Fernandez S, Khan R et al. Brief DISCERN, six questions for the evaluation of evidence-based content of health-related websites. Patient education and counseling. 2009;77(1):33-7. PubMed | Google Scholar
- Charnock D. The DISCERN handbook. Quality criteria
for consumer health information on treatment choices
Radcliffe: University of Oxford and The British Library.
- Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Education & Behavior. 2006;33(3):352-73. PubMed | Google Scholar
- McLaughlin G. SMOG: Simple measure of gobbledygook.
- Kincaid JP, Fishburne Jr RP, Rogers RL, Chissom BS. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975. Google Scholar
- Brenner BM, Cooper ME, De Zeeuw D, Keane WF, Mitch WE, Parving H-H et al. Effects of losartan on renal and cardiovascular outcomes in patients with type 2 diabetes and nephropathy. New England Journal of Medicine. 2001;345(12):861-9. PubMed | Google Scholar
- Mayo Clinic. Drugs and Supplements Valsartan (Oral Route). 2019. Accessed on February 17 2019
- Patient. Losartan - an angiotensin receptor blocker.
- Batchelor JM, Ohya Y. Use of the DISCERN instrument by patients and health professionals to assess information resources on treatments for asthma and atopic dermatitis. Allergology International. 2009;58(1):141-5. PubMed | Google Scholar
- Chong YM, Fraval A, Chandrananth J, Plunkett V, Tran P. Assessment of the quality of web-based information on bunions. Foot & ankle international. 2013;34(8):1134-9. PubMed | Google Scholar
- Saithna A, Ajayi O, Davis E. The quality of internet sites providing information relating to hip resurfacing. The Surgeon. 2008;6(2):85-7. PubMed | Google Scholar
- Tallgren M, Bäcklund M. Patient information about general anaesthesia on the internet. Anaesthesia. 2009;64(4):408-15. PubMed | Google Scholar
- Cicchetti DV. Guidelines, criteria, and rules of
thumb for evaluating normed and standardized assessment
instruments in psychology. Psychological assessment.
- Wilson M. Readability and patient education materials used for low-income populations. Clinical Nurse Specialist. 2009;23(1):33-40. PubMed | Google Scholar
- Estrada CA, Hryniewicz MM, Higgs VB, Collins C, Byrd JC. Anticoagulant patient information material is written at high readability levels. Stroke. 2000;31(12):2966-70. PubMed | Google Scholar
- Hutchinson N, Baird GL, Garg M. Examining the reading level of internet medical information for common internal medicine diagnoses. The American journal of medicine. 2016;129(6):637-9. PubMed | Google Scholar
- Cohen DL, Townsend RR. Update on pathophysiology and treatment of hypertension in the elderly. Current hypertension reports. 2011;13(5):330-7. PubMed | Google Scholar
- Institute for Healcare Improvement. Health
communications toolkits: improving readability of
patient education materials. 2019. Accessed on February 17 2019