Security researchers and practitioners lack techniques to quantitatively evaluate the strength of security systems against a determined attacker. Currently, evaluation is either qualitative, such as through security certification standards, or adhoc, such as through penetration testing and auditing. In this paper, we propose a framework that if applied to security systems, would produce quantita tive measures that can be used to compare and appraise security systems relative to each other. Our framework utilizes public challenges in conjunction with an inde pendent organization that mounts the challenges, regu lates their implementation and certifies the results in an attempt to provide normalized measures. Unlike various adhoc challenges that have been run in the past, we be lieve our framework can create a quantitative, challenge based security evaluation infrastructure that is fair, sus