CS 8803
Experimental Performance Evaluation of Computer Systems and Networks
Spring 2005
Office: 218 GCATT
Office phone: 5-4205
Office hours: after class or by appointment
Email: dovrolis@cc.gatech.edu
Table of Contents
- Class meeting times: TR 3:05-4:25
- Classroom: Physics S106
- No class on Jan 13, Mar 15, Mar 17, and Mar 31 (the instructor
will be at conferences, out of town).
Computer systems and networks are often evaluated through
measurements, simulations, and emulations. This course will
cover a number of techniques that are quite useful for
experimental performance evaluation. These techniques are
from the areas of experimental design, statistics (both
parametric and non-parametric), data presentation,
workload characterization, random number generation, simulation,
queueing theory, and time series analysis/forecasting.
Additionally, the course will cover several case-studies of
experimental performance evaluation from the areas of
operating and distributed systems, computer architecture,
databases, and networks.
<
The Art of Computer Systems Performance Analysis, by Raj Jain.
John Wiley and Sons, 1991.
We will also study the following papers:
- V. Paxson,
Strategies for Sound Internet Measurement, Proc. ACM IMC, October 2004.
- A. Downey,
Evidence for long-tailed distributions in the Internet
Proc. ACM IMW, October 2001.
- M. Crovella and L. Lipsky,
Long-lasting transient conditions in simulations with heavy-tailed workloads,
Proc. Winter Simulation Conference 1997.
- W. Willinger, D. Alderson, L. Li,
A Pragmatic
Approach to Dealing with High-variability in Network Measurements ,
Proc. ACM IMC, October 2004.
The course will consist of two parts.
In the first part, the instructor will give a sequence of lectures
covering the course's material. The course will strongly depend
on the students' active participation in the class.
The second part will take place during the last 3-4 weeks of the semester,
and it will consist of student project presentations.
Prerequisites
Some solid knowledge of basic probability and statistics is
required. Note however that the course will be rather on the
practical side, meaning that the focus will be on the
applications of the presented techniques, rather than on
their mathematical underpinnings.
- Part I: Introduction to performance evaluation
- Techniques, metrics, and common mistakes
- Data presentation techniques
- Ratio games
- Part II: Statistical tools
- Summarizing measured data
- Basics of sampling theory
- Comparing systems using sampled data
- Basics of estimation theory and confidence intervals
- Tests of hypotheses and significance
- Parametric and non-parametric tests for goodness of fit
- Regression techniques
- Part III: Simulation
- Simulation basics and common mistakes
- Transient removal and termination criteria
- Random number generation
- Simulations with infinite-variance distributions
- Part IV: Experimental design and analysis
- Full factorial designs
- Full factorial designs with replications
- Fractional factorial designs
- One factor experiments
- Part V: Other topics
- Dealing with heavy-tailed distributions
- Histograms and density estimation
Each student that takes this course for credit will
have to do a project. Groups of 2-3 students are
acceptable, as long as their project requires a substantially
larger amount of work. The objective of the projects
is that students use what they learned in this course
in their own research domain.
The projects can take several different forms:
- You can use some of the techniques that
are covered in the course in your research work.
In that case, you can go back to a paper that you
have previously written (or that you currently work on)
and apply some performance evaluation techniques
in that work.
- You can work on a new problem,
advised by the instructor, that is related to
experimental performance evaluation within your
area of research interests.
- You can apply performance evaluation techniques
on a research study that was written earlier by others.
Note that in this case you may need to first acquire
the datasets used in that study from the authors.
All projects will be presented in class in the
last 2-3 weeks of the semester.
- Class participation: 50%
- Project and final presentation: 50%