How do third-party evaluations of judgments affect decision quality on ill-structured problems? An interactive experiment

Third-party evaluations;Inspections;Reviews;Judgment;Public Company Accounting Oversight Board (PCAOB)

Garza, Brent Anthony

关键词: Third-party evaluations;    Inspections;    Reviews;    Judgment;    Public Company Accounting Oversight Board (PCAOB);   

Others  :  https://www.ideals.illinois.edu/bitstream/handle/2142/97703/GARZA-DISSERTATION-2017.pdf?sequence=1&isAllowed=y

美国|英语

来源: The Illinois Digital Environment for Access to Learning and Scholarship

【 摘 要 】

Using an interactive laboratory experiment, I start to investigate how third-party evaluations of judgments on ill-structured problems affect the decision quality of those being evaluated (“evaluatees”).First, an evaluatee completes an ill-structured estimation task.I deem the amount and type of costly procedures performed by the evaluatee to make that estimate their “judgment.”An evaluator then evaluates this judgment to determine whether the evaluatee should perform additional costly work to improve the estimate.Finally, the evaluatee may perform any additional work and adjust their estimate.I manipulate the evaluators’ level of focus on the quantity of work performed by evaluatees (more vs. less) and when evaluators document their assessment of evaluatees’ judgment (before vs. after conversing with evaluatees).I find that evaluators focused more on the quantity of work who document their initial assessment before conversing with evaluatees request greater amounts of the procedure they (i.e., evaluators) rate as more diagnostic but do not request greater amounts of the procedure they rate as less diagnostic.Evaluators’ behavior is consistent with theories of confirmation bias.Second, I find that evaluatees perform the additional work requested and, over time, learn to perform some of it before the evaluation.Third, this additional work generally improves evaluatees’ decision quality.I then examine the tradeoffs between performing procedures of differently-rated diagnosticities on decision quality.While results suggest that better decision quality occurs when evaluatees use a combination of the procedures, evaluators seem to over-request (and, thus, evaluatees over-perform) procedures that evaluators rate as more diagnostic.Therefore, while third-party judgment evaluations can improve decision quality on ill-structured problems, they may suboptimally do so.The theory and results of this study have implications for both accounting and non-accounting areas where judgments on ill-structured problems are evaluated by third parties.