DIstributed SImulation & Online gaming Workshop

Research Article

The perils of using simulations to evaluate Massively Multiplayer Online Game performance

  • @INPROCEEDINGS{10.4108/ICST.SIMUTOOLS2010.8632,
        author={Alexandre  Denault and J\o{}rg  Kienzle},
        title={The perils of using simulations to evaluate Massively Multiplayer Online Game performance},
        proceedings={DIstributed SImulation \& Online gaming Workshop},
        publisher={ACM},
        proceedings_a={DISIO},
        year={2010},
        month={5},
        keywords={Simulation of Games Performance Measurements Design and Implementation of Massively Multiplayer Online Games},
        doi={10.4108/ICST.SIMUTOOLS2010.8632}
    }
    
  • Alexandre Denault
    Jörg Kienzle
    Year: 2010
    The perils of using simulations to evaluate Massively Multiplayer Online Game performance
    DISIO
    ICST
    DOI: 10.4108/ICST.SIMUTOOLS2010.8632
Alexandre Denault1,*, Jörg Kienzle1,*
  • 1: School of Computer Science, McGill University, Montreal, QC H3A 2A7, Canada
*Contact email: alexandre.denault@mail.mcgill.ca, Joerg.Kienzle@mcgill.ca

Abstract

To test and benchmark Massively Multiplayer Online Games (MMOGs) requires hundreds, if not thousands of human players. Given this impracticality, many researchers substitute experimentation with simulation. However, when investigating performance and scalability issues, simulated experiments often yield results that heavily depend on the experimental setup. This paper critically reflects on the use of simulation to conduct experiments in MMOGs. Using Mammoth, a MMOGs research framework, performance measurements such as CPU usage, memory usage and used network bandwidth are collected while running the same game scenario using five different simulation setups. The results are analyzed, and the discovered differences are discussed. The paper concludes that experiments which are aimed at measuring the performance of a MMOGs using simulations must be designed very carefully, especially if they are run on a single machine.