1st Intenational IEEE Workshop on Information Assurance Middleware for Communications

Research Article

Automatic Calibration Using Receiver Operating Characteristics Curves

  • @INPROCEEDINGS{10.1109/COMSWA.2007.382484,
        author={ Prakash Kolan and Ram Vaithilingam and Ram Dantu},
        title={Automatic Calibration Using Receiver Operating Characteristics Curves},
        proceedings={1st Intenational IEEE Workshop on Information Assurance Middleware for Communications},
        keywords={Receiver Operating Characteristics curves  Threshold  Tolerance  behavior},
  • Prakash Kolan
    Ram Vaithilingam
    Ram Dantu
    Year: 2007
    Automatic Calibration Using Receiver Operating Characteristics Curves
    DOI: 10.1109/COMSWA.2007.382484
Prakash Kolan1,*, Ram Vaithilingam1,*, Ram Dantu1,*
  • 1: Department of Computer Science University of North Texas Denton, USA
*Contact email: prkO002@gcs.unt.edu, vO076@gunt.edu, rdantu@cs.unt.edu


Application-level filters, such as e-mail and VoIP spam filters, that analyze dynamic behavior changes are replacing static signature-recognition filters. These application-level filters learn behavior and use that knowledge to filter unwanted requests. Because behavior of a service request's participating entities changes rapidly, filters must adapt quickly by using end user's preferences about receiving that service request message. Many adaptive filters learn from the participating entities' behavior; however, none configure themselves automatically to an end user's changing tolerance levels. Also, filter administrators cannot manually change the threshold for each service request in real time. Traditional adaptive filters fail when administrators must optimize multiple filter thresholds manually and often. Thus, to improve a filter's learning, we must automate its threshold-update process. We propose an automatic threshold-calibration mechanism using Receiver Operating Characteristics (ROC) curves that updates the threshold based on an end user's feedback. To demonstrate the mechanism's real-time applicability, we integrated it in a Voice over IP (VoIP) spam filter that analyzes incoming Spam over IP Telephony (SPIT) calls. Using this mechanism, we observed good improvement in the VoIP spam filter's accuracy. Further, computing and updating the optimum threshold in realtime does not impede the filter's temporal performance because we update thresholds after each call's completion. Because we reach an optimum threshold for any initial setting, this mechanism works efficiently when we cannot predict end-user behavior. Furthermore, automatic calibration proves efficient when using multiple threshold values.