Why Bother?

So, I've written a simulator to help evaluate scheduling algorithms. I will not go into the details of the code, because it is too complex at this time. But we will go through a few scheduling scenarios to learn some things.

Our Job Model

• A name.

• An interarrival probability distribution. This (and all subsequent probability distributions) is defined to be Uniform (UNI), Exponential (EXP) or Normal (NORM), with a given mean and minimum value. This is straightforward for Uniform distributions. For exponentials, one typically defines a distribution solely with a mean, and its minimum is zero. However, we allow for non-zero minima with the following equation: exp(mean,min) = exp(mean-min,0)+min.. For Normals, we assume that the standard deviation is 0.75*(avg-min).

  The interarrival probability distribution defines the period of time that passes between jobs of that type. There is another parameter whose value is either SERIAL or PERIODIC. If SERIAL, then a job of one type does not start until the interarrival period has passed after the last job of that type finished. If PERIODIC, then the interarrival time is measure from when the previous job started.

• A probability distribution for number of iterations. Jobs work with the following pseudo-code:

    for (i = 0; i < iterations; i++) {
       sleep(sleep_time);
       use_cpu(cpu_burst_time);
       do_io(io_burst_time);
    }
  

  The number of iterations is defined by a probability distribution.

• A probability distribution for the sleep time.

• A probability distribution for the cpu burst time.

• A probability distribution for the IO burst time. An IO device must also be specified.

Jobs are specified in a job file, in which each line specifies a job. Each line has 18 words:

1. Job Name.
2. Interarrival time distribution type (Uniform/Exponential/Normal).
3. Interarrival time distribution mean.
4. Interarrival time distribution minimum.
5. Interarrival type: Serial or Periodic.
6. Number of iterations distribution type.
7. Number of iterations distribution mean.
8. Number of iterations distribution minimum.
9. Sleep time distribution type.
10. Sleep time distribution mean. (Units: seconds)
11. Sleep time distribution minimum.
12. CPU-burst time distribution type.
13. CPU-burst time distribution mean. (Units: machine cycles)
14. CPU-burst time distribution minimum.
15. IO-burst time distribution type.
16. IO-burst time distribution mean. (Units: device units)
17. IO-burst time distribution minimum.
18. IO-burst device.

For example, the file job-cpu-1.txt defines one job, called CPU-B. These jobs are generated serially, one after another, with no interarrival time (i.e. as soon as one job finishes, the next begins). Each CPU-B job iterates 10 times, sleeping for zero seconds, then performing 180 million CPU operations, then doing 1 unit's worth of I/O to the console.

The file job-vi-1.txt also has one job, whose intent is to model interactive editing jobs such as vi. These are executed serially, and the job interarrival times are defined by an exponential whose mean is 30 minutes (1800 seconds), and whose minimum is 20 seconds. The jobs themselves iterate according to an exponential whose mean is 900, and whose minimum is 10, and each iteration sleeps for an average of 2 seconds, followed by an average of 500 cycle's worth of CPU time, and one unit of console I/O.

The file job-cpu-vi.txt mixes a CPU-bound job and one VI job. It should run well, since the VI job doesn't use much CPU.

The file job-cpu-vi-wav.txt has six different VI jobs, plus a WAV job, which represents a CD player (each iteration reads a second's worth of music from the CD-ROM drive, and then sleeps for a second to emulate playing the music).

Obviously, you may design your own job files.

The Machine

You describe a machine with a machine file. This file needs to have a SPEED (in millions of instructions per second), a context switch overhead (in instructions), an optional timer interrupt frequency (specified in instructions), and any number of devices. Devices are specified with names and unit speeds.

The file machine-1.txt defines a machine running at one million instructions per second, with a context switch overhead of 200 instructions, and four devices of varying speedss -- a disk, a cd-rom drive, a network card, and the console.

The file machine-2.txt defines a machine equivalent to machine-1.txt but with a timer that interrupts the CPU every 50000 instructions.

Note in both files, the units of DISK, CD-ROM and NETWORK are MB/s, while the console's units are bytes per second.

Other Simulator Parameters

To use the simulator, you have to link it with a scheduler (for example ss-fcfs.c is a non-preemptive first-come-first-serve scheduler), then you execute it with the following parameters:

1. OFF -- no output (although SIGQUIT will show the state of the machine at any time).
2. EVENTS -- show a line for each simulator event.
3. SYSTEM -- show the system state after each simulator event.
4. SAMPLE=delta -- After each delta seconds of simulated time, print out the state of the machine.
5. QUIT=qtime -- After qtime seconds of simulated time, print out the machine state and quit.

So, for example, if you run the simulator as follows:

UNIX> ss-fcfs job-cpu-1.txt machine-1.txt 100 100 EVENTS | more
       0.000000: [CPU-B:000]    Starting
       0.000000: [CPU-B:000] -> Ready
       0.000000: [CPU-B:000] -> Running
     180.000200: [CPU-B:000]    Running
     180.000200: [CPU-B:000] -> IO on CONSOLE (0.000100)
     180.000300: [CPU-B:000]    IO on CONSOLE (0.000100)
     180.000300: [CPU-B:000] -> Ready
     180.000300: [CPU-B:000] -> Running
     360.000300: [CPU-B:000]    Running
     360.000300: [CPU-B:000] -> IO on CONSOLE (0.000100)
     360.000400: [CPU-B:000]    IO on CONSOLE (0.000100)
     360.000400: [CPU-B:000] -> Ready
     360.000400: [CPU-B:000] -> Running
     540.000400: [CPU-B:000]    Running
     540.000400: [CPU-B:000] -> IO on CONSOLE (0.000100)
     540.000500: [CPU-B:000]    IO on CONSOLE (0.000100)
     540.000500: [CPU-B:000] -> Ready
     540.000500: [CPU-B:000] -> Running
...

Here's the result of running that one process using the QUIT output flag:

UNIX> ss-fcfs job-cpu-1.txt machine-1.txt 100 100 QUIT=1000000
SYSTEM STATE ---------------------------------------------------------

  Time: 1000000.000000

  Processes:
    555 [CPU-B:555] - Running

  Completed-Jobs:

       CPU-B COMP:    555 ELAP: 1800.001  CPU: 1800.000  RQ:    0.000   CS:    0.000
             SL:    0.000   IO:    0.001  IOW:    0.000 MRQ:    0.000 MIOW:    0.000

  CPU Utilization:   99.99%  Throughput: 0.00056  Turaround: 1800.00
  Avg-Wait-Time:      0.00   MAXRQ:        0.000  MAXIOW:      0.000
UNIX>

Test 0 -- An IO/Bound job: A VI process

ss-fcfs job-one-vi.txt machine-1.txt 100 100 QUIT=1000000

Are things as we'd expect? Well, yes: Each job averages 900 events with a sleep time of 2 seconds, so each job should average 1800 seconds (same as a CPU-bound job), followed by an average wait (interarrival) time of 1800 seconds. Therefore, roughly 555/2=277 VI jobs should finish in 1,000,000 seconds.

Test 1 -- A CPU-bound job and a VI job

• Since the VI jobs use very little CPU, we would expect that the performance of our CPU-bound jobs will change very little. In fact, after 1,000,000 seconds, I would expect 555 CPU-bound jobs to be completed, just as in the run above where there was no VI job.

• If the scheduler is working well, I'd expect the VI jobs to complete with little overhead, so roughly 277 jobs should finish.

Of course, when we run this job file with the non-preemptive FCFS scheduler, we don't get that behavior:

The CPU-bound jobs finish fine, but since they do not get preempted by the VI jobs, the VI jobs have to wait up to (roughly) 180 seconds before they get the CPU (to see that, look at the "Max RQ" value of VI-1). This has dire consequences -- the VI jobs have an average of over 150,000 seconds of wait time, and only four of them finish in 1,000,000 seconds.

A second scheduler is in ss-fcfsp. This is a first-come-first-served scheduler, but it is preemptive, which means that at every scheduling point, the running job is preempted and put at the end of a ready queue. With job-cpu-vi, this scheduler has great performance, because every time the VI process needs the CPU, it gets it, and when it it not running, the CPU process may run:

This is nice -- 262 VI jobs complete, and they spend a negligible amount of time on the ready queue. The CPU jobs do spend some time on the ready queue, when they are preempted so that the VI jobs can do their little amount of processing.

However, if we extrapolate this and run the simulation longer, we see a problem:

There are only 647 completed VI processes, and the 647th one is caught on the ready queue. What has happened? Well, this VI process was unfortunate enough to have its CPU burst (which must be fairly big -- a possibility with exponential distributions) get preempted by the CPU-bound process (this can only happen when the CPU process returns from its IO to the console). At this point, the CPU processes only give up the CPU for 100 cycles at a time, and the VI process exhibits horrendous performance. Note -- the "Max RQ Wait Time" for the machine is 180 seconds -- this is the VI process waiting for the CPU processes. Note also, that the "Job Stats" are only for completed processes, which is what the "Max RQ" for VI processes says 0.006 rather than 180.000.

To illustrate this problem more clearly, look at job-cpu-2.txt. This file has two identical CPU-bound jobs. When we run it, we see the following:

What is happening is that the CPU-B job never gets executed, except when CPU-A stops for I/O (and since the I/O time is quicker than the context-switch time, CPU-B still does no work). This is because CPU-A gets the CPU whenever it tries to schedule it. To see this perhaps more clearly, look at the first few events when this job is executed:

       0.000000: [CPU-B:000]    Starting
       0.000000: [CPU-B:000] -> Ready
       0.000000: [CPU-B:000] -> Running
       0.000000: [CPU-A:000]    Starting
       0.000000: [CPU-A:000] -> Ready
       0.000000: [CPU-B:000] -> Ready
       0.000000: [CPU-A:000] -> Running
     180.000200: [CPU-A:000]    Running
     180.000200: [CPU-A:000] -> IO on CONSOLE (0.000100)
     180.000200: [CPU-B:000] -> Running
     180.000300: [CPU-A:000]    IO on CONSOLE (0.000100)
     180.000300: [CPU-A:000] -> Ready
     180.000300: [CPU-B:000] -> Ready
     180.000300: [CPU-A:000] -> Running
     360.000500: [CPU-A:000]    Running
     360.000500: [CPU-A:000] -> IO on CONSOLE (0.000100)
     360.000500: [CPU-B:000] -> Running
     360.000600: [CPU-A:000]    IO on CONSOLE (0.000100)
     360.000600: [CPU-A:000] -> Ready
     360.000600: [CPU-B:000] -> Ready
     360.000600: [CPU-A:000] -> Running
     540.000800: [CPU-A:000]    Running
     540.000800: [CPU-A:000] -> IO on CONSOLE (0.000100)
     etc.

Test #2 -- A Round-Robin Scheduler

(You will note that this takes a while to run -- about 30 seconds on my LINUX box). Also, you'll see that the CPU utilization and throughput are slightly lower, since more time is spent context switching.

Now, if we run job-cpu-vi.txt, we'll see that the VI job has a much more acceptable max wait time:

And, if we run six VI jobs and a WAV player along with the CPU-bound job, they all run pretty well:

However, suppose we run job-10-cpu-vi-wav.txt, which has ten identical CPU-bound jobs, plus two VI jobs and a WAV player:

While everything runs as you think it should, the MAX ready queue time of the VI and WAV jobs is very bad. Why? Because whenever one of these I/O-bound jobs needs scheduling, it starts off at the end of the ready queue, and has to wait for the 10 CPU-bound jobs to finish their time slices. Clearly, this motivates the need for a smarter scheduling algorithm.

Test 3 -- The "Convoy" effect

Now, add a CPU-bound job to the mix (job-convoy-2.txt). Since the scheduler is the non-preemptive FCFS scheduler, the WAV jobs all wait for the CPU-bound job, and when that job gives up the CPU, all three WAV jobs end up competing for the IO device:

Interesting, no? We see the "convoy" effect here, because the long CPU-bound job causes the IO-bound processes to wait not only on the ready queue, but also on the CD-ROM queue. You see this with the high IO-Wait times for the three WAV jobs.

The time-slicing scheduler doesn't exhibit this effect:

As you can see, the IO-Wait times are now negligible once again.

As an interesting mental exercise, look at the following job file: job-convoy-3.txt. This has 20 CPU-bound jobs with the 3 WAV jobs. Would you expect this to exhibit the convoy effect or not? Think about it.

My feeling was that this would exhibit the convoy effect, since each WAV job now has to wait for 20 CPU-bound time slices to use the IO device. Look at the output, though:

I see no convoy effect. Why?

Well, the answer is that although each WAV job will have to wait for a bunch of CPU-bound jobs, since the WAV jobs sleep for random periods of time, they interleave with the CPU-bound jobs randomly, and it is unlikely that two WAV jobs get placed on the ready queue adjacently. Thus, no convoy effect. Interesting, no?

Test 4 -- Reducing the latency of I/O-bound jobs

Here is how it performs on job-10-cpu-vi-wav.txt:

Well, fortunately, this solves the VI problem -- the max RQ times of these jobs have dropped to nearly nothing, since they always get scheduled in preference to the CPU-bound jobs.

However, there is a problem -- the CPU-C process is getting starved. Why? Well, think about what happens when there is no IO-Bound process. At first, all CPU processes have the same cpu-burst time. One of them is selected, and then it will get to run to completion before all the others. This is because once it does some work, its CPU burst time is decremented, and is thus less than the others. When it is done, another of the CPU-bound processes is selected at random, and then it is executed to completion. Which CPU-bound process is selected is up to the vagaries of the red-black tree library, and I would put money that CPU-C is at the end of the tree; no process gets put after it.

Is this the right thing? While you're thinking about that, see how SJF performs on job-10-cpu-one-shorter.txt, which is the same as job-10-cpu-vi-wav.txt, except the first CPU-bound process has 179-second CPU bursts instead of 180-second CPU bursts:

Again, the VI and WAV jobs work perfectly. However, since CPU-A's burst time is less than all the others, it gets scheduled in preference to the others, and the others get starved.

You'll note that all the others (with the exception of CPU-C) do get scheduled when CPU-A does its I/O, but that does not affect them, because the I/O time of CPU-A is less than the context-switch overhead. CPU-C is once again caught at the end of the red-black tree, and never gets scheduled.

Now, is it right that one job runs while all the others starve? In terms of throughput, yes. You'll have to think that through for yourself. However, in terms of fairness, no. So I tweaked ss-sjf so that if a process's CPU burst is more than the quantum, then it goes into a queue that gets serviced in a FCFS manner. Here's how it runs on job-10-cpu-one-shorter.txt:

That looks a lot better, doesn't it?

Test 5 -- We are not omniscient.

P starts with an initial value (we set it to zero). Alpha is a number between 0 and 1. If 0, then our prediction is always the initial value of P. If 1, then our prediction is always the last cpu burst. Otherwise, it's some combination of the initial value and the most recent CPU bursts. Obviously, the closer that alpha is to one, the more it weighs the most recent bursts. If you look at the equation, it is an exponentially decaying equation -- meaning that the most recent CPU bursts are always weighed more than the less recent ones.

I've hacked this up in ss-exp, where alpha is compiled in. Here are a few values of alpha:

As you can see, since the prediction is always 0 (the initial value), all processes will have the same prediction, and the process that gets scheduled is the one lucky enough to be at the front of the red-black tree. The unfortunate process (CPU-C) gets starved.

Here are the results when alpha is 1.0, and we only look at the last CPU burst:

This is much better, since the IO-Bound processes get the CPU in preference to the CPU-bound processes. However, the CPU-bound processes appear to be slave to the luck of the red-black tree. Some of them (CPU-H and CPU-I) are getting scheduled more than the others. I haven't done any tracing to see why this is the case.

Here we set alpha to 0.5. The performance looks pretty much identical to when alpha is 1, with one main difference -- the VI/WAV processes have smaller Max wait times. Think about why? (The answer is what happens when the CPU-Bound process does IO, and looks like a better process to schedule, perhaps, than a VI/WAV process).

Check out the wild asymmetry with the CPU-bound processes. Think about why that would be the case.

Test 6 -- Predictions with a threshhold

Test 7 -- Using a Multilevel Feedback Queue

First, ss-mlfq-1 implements a three-level queue. The first level is where processes go that have either just started running (due to starting, or due to sleeping/IO) or gave up the CPU voluntarily due to a non-timer event. The second level is for processes that have been evicted from the CPU due to one timer interrupt. The third level is for processes that have been evicted due to multiple timer interrupts. These are the CPU-bound jobs. The scheduler processes each queue in a round-robin fashion, but it only gives the CPU to a job in the second-level queue if there are none in the first-level queue. Similarly, it only gives the CPU to a job in the third-level queue if there are none in the other queues.

Here is the result of running ss-mlfq-1:

Note, this doesn't solve the context-switch problem, but it does work nicely, just like our predictive scheduler.

Now, ss-mlfq-2 is just like ss-mlfq-1, except that it only evicts a CPU-bound job from the CPU if there is a job on another queue, or if 10 time quanta have passed since it was scheduled. In that way, the context-switch overhead of CPU-bound jobs should be reduced:

Note two things here -- first, the context switch overhead has indeed been reduced. Second, as a result, the CPU utilization has gone up, the job throughput has gone up, and the turnaround time has gone down. In fact, the only metric that has gone the "wrong way" is the Max RQ time, which has increased dramatically. However, since these CPU-bound jobs should not care about response time, this is not really important.

As a final aside, if I change the quantum for CPU-bound jobs to 100 times the regular quantum, there is improvement, but not ten-fold:

There should be one final fix, in my opinion. I think that jobs on the first queue should be ordered by prediction, so that when the CPU bounds revert to the top queue, they are scheduled with lower priority than the I/O bound jobs. That will lower the Max RQ time, most likely.