The reliability of a quality appraisal tool for studies of diagnostic reliability (QAREL)

BMC Medical Research Methodology

Nikolai Bogduk³  Robin Turner⁴  Luke Rickards¹  Robert Moran²  Les Irwig⁴  Petra Macaskill⁴  Nicholas Lucas⁴ 

[1] Private Practice, Sydney, Australia;School of Health Science, UNITEC, Auckland, New Zealand;Department of Clinical Research, Newcastle Bone and Joint Institute, Royal Newcastle Centre, University of Newcastle, Newcastle, Australia;Screening and Test Evaluation Program, Sydney School of Public Health, University of Sydney, Sydney, Australia

关键词: Evidence-based medicine;    Systematic review;    Quality appraisal;    Reliability;   

Others  :  1091817
DOI  :  10.1186/1471-2288-13-111

 received in 2013-02-21, accepted in 2013-09-05,  发布年份 2013

【 摘 要 】

Background

The aim of this project was to investigate the reliability of a new 11-item quality appraisal tool for studies of diagnostic reliability (QAREL). The tool was tested on studies reporting the reliability of any physical examination procedure. The reliability of physical examination is a challenging area to study given the complex testing procedures, the range of tests, and lack of procedural standardisation.

Methods

Three reviewers used QAREL to independently rate 29 articles, comprising 30 studies, published during 2007. The articles were identified from a search of relevant databases using the following string: “Reproducibility of results (MeSH) OR reliability (t.w.) AND Physical examination (MeSH) OR physical examination (t.w.).” A total of 415 articles were retrieved and screened for inclusion. The reviewers undertook an independent trial assessment prior to data collection, followed by a general discussion about how to score each item. At no time did the reviewers discuss individual papers. Reliability was assessed for each item using multi-rater kappa (κ).

Results

Multi-rater reliability estimates ranged from κ = 0.27 to 0.92 across all items. Six items were recorded with good reliability (κ > 0.60), three with moderate reliability (κ = 0.41 - 0.60), and two with fair reliability (κ = 0.21 - 0.40). Raters found it difficult to agree about the spectrum of patients included in a study (Item 1) and the correct application and interpretation of the test (Item 10).

Conclusions

In this study, we found that QAREL was a reliable assessment tool for studies of diagnostic reliability when raters agreed upon criteria for the interpretation of each item. Nine out of 11 items had good or moderate reliability, and two items achieved fair reliability. The heterogeneity in the tests included in this study may have resulted in an underestimation of the reliability of these two items. We discuss these and other factors that could affect our results and make recommendations for the use of QAREL.

【 授权许可】

   
2013 Lucas et al.; licensee BioMed Central Ltd.

【 参考文献 】

• [1]Lucas NP, Macaskill PM, Irwig L, Bogduk N: The development of a quality appraisal tool for studies of diagnostic reliability (QAREL). J Clin Epidemiol 2010, 63:854-861.
• [2]Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HCW: The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res 2010, 19:539-549.
• [3]Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hrobjartsson A, et al.: Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol 2011, 64:96-106.
• [4]Gemmell H, Miller P: Interexaminer reliability of multidimensional examination regimens used for detecting spinal manipulable lesions: a systematic review. Clin Chiropr 2005, 8:199-204.
• [5]Hestboek L, Leboeuf-Yde C: Are chiropractic tests for the lumbo-pelvic spine reliable and valid? A systematic critical literature review. J Manipulative Physiol Ther 2000, 23:258-275.
• [6]Hollerwoger D: Methodological quality and outcomes of studies addressing manual cervical spine examinations: a review. Man Ther 2006, 11:93-98.
• [7]May S, Littlewook C, Bishop A: Reliability of procedures used in the physical examination of non-specific low back pain: a systematic review. Aust J Physiother 2006, 52:91-102.
• [8]Seffinger MA, Najm WI, Mishra SI, Adams A, Dickerson VM, Murphy LS, et al.: Reliability of spinal palpation for diagnosis of back and neck pain: a systematic review of the literature. Spine 2004, 29:E413-25.
• [9]Stochkendahl MJ, Christensen HW, Hartvigsen J, Vach W, Haas M, Hestbaek L, et al.: Manual examination of the spine: a systematic critical literature review of reproducibility. J Manipulative Physiol Ther 2006, 29:475-85. 485 e1-10
• [10]Van Trijffel E, Anderegg Q, Bossuyt PMM, Lucas C: Inter-examiner reliability of passive assessment of intervertebral motion in the cervical and lumbar spine: A systematic review. Man Ther 2005, 10:256-269.
• [11]Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al.: Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Ann Intern Med 2003, 138(1):40-44.
• [12]Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J: The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 2003, 3:25. BioMed Central Full Text
• [13]Bertilson B, Grunnesjo M, Johansson S-E, et al.: Pain drawing in the assessment of neurogenic pain and dysfunction in the neck/shoulder region: Inter-examiner reliability and concordance with clinical examination. Pain Med 2007, 8:134-146.
• [14]Bremander AB, Dahl LL, Roos EM: Validity and reliability of functional performace tests in meniscectomized patients with or without knee osteoarthritis. Scand J Med Sci Sports 2007, 17:120-127.
• [15]Brushoj C, Langberg H, Larsen K, et al.: Reliability of normative values of the foot line test: a technique to assess foot posture. J Ortho Sports Phys Ther 2007, 37:703-707.
• [16]Bybee RF, Dionne CP: Interrater agreement on assessment, diagnosis, and treatment for neck pain by trained physical therapist students. J Phys Ther Edu 2007, 21:39-47.
• [17]Cook C, Massa L, Harm-Emandes I, Segneri R, Adcock J, Kennedy C, Figuers C: Interrater reliability and diagnostic accuracy of pelvic girdle pain classification. J Manipulative Physiol Ther 2007, 30:252-258.
• [18]De Jong LD, Nieuwboer A, Aufdemkampe G: The hemiplegic arm: Interrater reliability and concurrent validity of passive range of motion measurements. Disability Rehab 2007, 29:1442-1448.
• [19]Dionne C, Bybee RF, Tomaka J: Correspondence of diagnosis to initial treatment for neck pain. Physiotherapy 2007, 93:62-68.
• [20]Gladman DD, Inman RD, Cook RJ, van der Heijde D, Landewe RMB, et al.: International spondyloarthritis interobserver reliability exercise. The INSPIRE study: I. Assessment of spinal measures. J Rheumatol 2007, 34:1733-1739.
• [21]Gladman DD, Inman RD, Cook RJ, Maksymowych WP, Braun J, et al.: International spondyloarthritis interobserver reliability exercise. The INSPIRE study: II. Assessment of peripheral joints, enthesitis, and dactylitis. J Rheumatol 2007, 34:1740-1745.
• [22]Hacker MR, Funk SM, Manco-Johnson MJ: The Colorado haemophilia paediatric joint physical examination scale: Normal values and interrater reliability. Haemophilia 2007, 13:71-78.
• [23]Hickey BW, Milosavljevic S, Bell ML, Milburn PD: Accuracy and reliability of observational motion analysis in identifying shoulder symptoms. Man Ther 2007, 12:263-270.
• [24]Hungerford BA, Gilleard W, Moran M, Emmerson C: Evaluation of the ability of physical therapists to palpate intrapelvic motion with the stork test on the support side. Phys Ther 2007, 87:879-887.
• [25]Kim Y-S, Kim J-M, Ha K-Y, Choy S, Joo M-W, et al.: The passive compression test: A new clinical test for superior labral tears of the shoulder. Am J Sports Med 2007, 35:1489-1494.
• [26]Kim HW, Ko YJ, Rhee WI, Lee JS, Lim JE, et al.: Interexaminer reliability and accuracy of posterior superior iliac spine and iliac crest palpation for spinal level estimations. J Manipulative Physiol Ther 2007, 30:386-389.
• [27]Kryger AI, Lassen CF, Andersen JH: The role of physical examination in studies of musculoskeletal disorders of the elbow. Occup Environ Med 2007, 64:776-781.
• [28]Lewis JS, Valentine RE: The pectoralis minor length test: A study of the intra-rater reliability and diagnostic accuracy in subjects with and without shoulder symptoms. BMC Musculoskelet Disord 2007, 8:64. BioMed Central Full Text
• [29]McCarthy CJ, Gittins M, Roberts C, Oldham JA: The reliability of the clinical tests and questions recommended in international guidelines for low back pain. Spine 2007, 32:921-926.
• [30]McEwan I, Herrington L, Thom J: The validity of clinical measures of patella position. Man Ther 2007, 12:226-230.
• [31]Myers JB, Oyama S, Wassinger CA, Ricci RD, Abt JP, et al.: Reliability, precision, accuracy, and validity of posterior shoulder tightness assessment in overhead athletes. Am J Sports Med 2007, 35:1922-1930.
• [32]Neumann PB, Grimmer-Somers KA, Gill VA, Grant RE: Rater reliability of pelvic floor muscle strength. Aust NZ Continence J 2007, 13:8-14.
• [33]Peeler J, Anderson JE: Reliability of the Thomas test for assessing range of motion about the hip. Phys Ther Sport 2007, 8:14-21.
• [34]Rainville J, Noto DJ, Jouve C, Jenis L: Assessment of forearm pronation strength in C6 and C7 radiculopathies. Spine 2007, 32:72-75.
• [35]Robinson HS, Brox JI, Robinson R, Bjelland E, Solem S, Telje T: The reliability of selected motion- and pain provocation tests for the sacroiliac joint. Man Ther 2007, 12:72-79.
• [36]Roussel NA, Nijs J, Truijen S, Smeuninx L, Stassijns G: Low back pain: Clinimetric properties of the trendelenburg test, active straight leg raise test, and breathing pattern during active straight leg raising. J Manipulaitve Physiol Ther 2007, 30:270-278.
• [37]Savic G, Bergstrom EMK, Frankel HL, Jamous MA, Jones PW: Inter-rater reliability of motor and sensory examinations performed according to American Spinal Injury Association standards. Spinal Cord 2007, 45:444-451.
• [38]Schneider M, Homonai R, Moreland B, Delitto A: Interexaminer reliability of the prone leg length analysis procedure. J Manipulative Physio Ther 2007, 30:514-521.
• [39]Sedaghat N, Latimer J, Maher C, Wisebey-Roth T: The reproducibility of a clinical grading system of motor control in patients with low back pain. J Manipulative Physiol Ther 2007, 30:501-508.
• [40]Visscher CM, Lobbezoo F, Naeije M: A reliability study of dynamic and static pain tests in temporomandibular disorder patients. J Orofac Pain 2007, 21:39-45.
• [41]Fleiss J: Statistical methods for rates and proportions. 3rd edition. Hoboken, N.J.: Wiley-Interscience; 2003.
• [42]Mokkink LB, Terwee CB, Gibbons E, Stratford PW, Alonso J, Patrick DL, Knol DL, Bouter LM, de Vet HCW: Inter-rater agreement and reliability of the COSMIN (Consensus-based Standards for the selection of health status Measurement Instruments) Checklist. BMC Med Res Methodol 2010, 10:8. BioMed Central Full Text