CS360 Lecture notes -- Thread #2


In this lecture, we cover race conditions and mutexes.


Race conditions

A race condition is when a piece of code, with multiple units of execution (such as threads), has a piece of data which can be read and written in multiple, unspecified, orders. Race conditions are typically bad. I say "typically," because sometimes they won't matter. However, usually a race condition in your code is going to be a bug eventually, so it is a good idea to know how to recognize race conditions, and how to use synchronization data structures (such as mutexes and condition variables) to avoid them, while still allowing a good degree of parallelism.

Take a look at race1.c. Its explanation is inline with the code:

/* This program forks off two threads which share an integer, 
   on which there is a race condition. */

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

/* This is information shared by the two threads. */

typedef struct {
  int i;
  int die;
} Shared_Info;

/* This is information which will be unique to each thread (SI is shared) */

typedef struct {
  int id;
  Shared_Info *SI;
} Info;

/* Here's the thread code -- pretty simple. */

void *thread(void *x)
{
  Info *I;
 
  I = (Info *) x;

  while (!I->SI->die) I->SI->i = I->id;
  return NULL;
}

/* The main code sets up the shared and unique info, then forks off two threads.
   It then sleeps for two seconds and prints out the shared variable, si.i.
   Finally, it calls pthread_join() to wait for the two threads to die, and
   prints out the shared variable again. */

main(int argc, char **argv)
{
  pthread_t tids[2];
  Shared_Info si;
  Info I[2];
  void *retval;

  /* Set up the data to send to the threads. */

  I[0].id = 0;
  I[0].SI = &si;

  I[1].id = 1;
  I[1].SI = &si;

  si.die = 0;
  
  /* Create the two threads and sleep */

  pthread_create(tids, NULL, thread, I);
  pthread_create(tids+1, NULL, thread, I+1);
  sleep(2);

  /* Tell the threads to die, then print the shared info. */

  si.die = 1;
  printf("%d\n", si.i);

  /* Wait for the threads to die and print out the shared info again. */

  pthread_join(tids[0], &retval);
  pthread_join(tids[1], &retval);
  printf("%d\n", si.i);
  exit(0);
}

Ok -- this program forks off two threads. Each thread has its own Info struct, which contains an id unique to the thread -- either 0 or 1. The Info struct also has a pointer to a Shared_Info struct, which is shared between the two threads. The Shared_Info struct has two variables -- i, which each thread is going to repeatedly overwrite with its id, and die, which each thread checks, and when it is one, the threads exit.

Ask yourself the following question: Where are the Info and Shared_Info structs stored? Heap or stack? If stack, whose stack?

The answer is that they are stored on the main thread's stack. There is no restriction on where threads can access memory -- a pointer is a pointer, and if it points to another thread's stack, so be it!

It should be pretty clear that this program has a race condition. The two threads are wontonly overwriting I->SI->i without any synchronization. If I asked you what the output of this program will be, you have to say that you don't really know. It can be one of four things:

UNIX> race1
0
1
UNIX> race1
1
0
UNIX> race1
0
0
UNIX> race1
1
1
UNIX>
The shell script r1.sh runs it 100 times, putting the output of each run on a single line. After taking the 200 seconds to run the shell script, you can see that all outputs have occurred:
UNIX> sh r1.sh > r1-output.txt
UNIX> grep '0 0' r1-output.txt | wc
      26      52     104
UNIX> grep '0 1' r1-output.txt | wc
      55     110     220
UNIX> grep '1 0' r1-output.txt | wc
      10      20      40
UNIX> grep '1 1' r1-output.txt | wc
       9      18      36
UNIX> 

This is most definitely a race condition. Is it a bad one? Not really, because this program doesn't do anything but demonstrate a race condition. Let's look at a more complex race condition:


Race2.c: A more complex race condition.

Look at race2.c. This is another pretty simple program. The command line arguments call for the user to specify the number of threads, a string size and a number of iterations. Then the program does the following. It allocates an array of stringsize characters. Then it forks off nthreads threads, passing each thread its id, the number of iterations, and the character array. Each thread loops for the specified number of iterations. At each iteration, it fills in the character array with one character -- thread 0 uses 'A', thread 1 uses 'B' and so on. At the end of an iteration, the thread prints out the character array. So, if we call it with the arguments 4, 4, 1, we'd expect the following output:
UNIX> race2 4 4 1
Thread 0: AAA
Thread 1: BBB
Thread 2: CCC
Thread 3: DDD
Similarly, the following make sense:
UNIX> race2 4 4 2
Thread 0: AAA
Thread 0: AAA
Thread 1: BBB
Thread 1: BBB
Thread 2: CCC
Thread 2: CCC
Thread 3: DDD
Thread 3: DDD
UNIX> race2 4 30 2
Thread 0: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Thread 0: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Thread 1: BBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Thread 1: BBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Thread 2: CCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Thread 2: CCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Thread 3: DDDDDDDDDDDDDDDDDDDDDDDDDDDDD
Thread 3: DDDDDDDDDDDDDDDDDDDDDDDDDDDDD
UNIX> 
Unfortunately, that output is not guaranteed. The reason is that threads can be preempted anywhere. In particular, they may be preempted in the middle of the for() loop, or in the middle of the printf() statement. This can lead to strange output. For example, try the following:
UNIX> race2 2 70 200000 | grep 'AB'
This searches for output lines where the character 'A' is followed by a B. When I ran this, I got:
UNIX> race2 2 70 200000 | grep 'AB'
Thread 0: AAAAAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Thread 1: AAAABBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
UNIX> 
This shows two instances where thread 0 was interrupted by thread 1, which had been interrupted in the middle of its for loop. When thread 1 resumed, it overwrote the string with B's.

So, this program too has a race condition, this time with the shared array s. The output above is particularly confusing, which is often what happens with race conditions.

When you program with threads, you must pay attention to shared memory. If more than one thread can modify the shared memory, then you often need to protect the memory so that wierd things do not happen to the memory.

In our race2 program, we can "fix" the race condition by enforcing that no thread can be interrupted by another thread when it is modifying and printing s. This can be done with a mutex, sometimes called a ``lock'' or sometimes a ``binary semaphore.'' There are three procedures for dealing with mutexes in pthreads:

pthread_mutex_init(pthread_mutex_t *mutex, NULL);
pthread_mutex_lock(pthread_mutex_t *mutex);
pthread_mutex_unlock(pthread_mutex_t *mutex);
You create a mutex with pthread_mutex_init(). You have to have allocated memory for it ahead of time (i.e. pthread_mutex_init() does not call malloc(). Then any thread may lock or unlock the mutex. When a thread locks the mutex, no other thread may lock it. If a thread calls pthread_mutex_lock() while the mutex is locked, then the thread will block until the mutex is unlocked. Only one thread may lock the mutex at a time.

I want to point out here, that pthread_mutex_lock() does not actively "lock" other threads. Instead, it locks a data structure, which can be shared among the threads. The locking and unlocking of the data structure makes synchronization guarantees, which are very important to avoiding race conditions. However, I don't want you to get into the habit of thinking that pthread_mutex_lock() actively blocks other threads, or "locks them out." It doesn't -- it locks a data structure, and when other threads try to lock the same data structure, they block. Please reread this paragraph.

So, we "fix" the program with race3.c. You'll notice that a thread locks the mutex just before modifying s and it unlocks the mutex just after printing s. This fixes the program so that the output makes sense:

UNIX> race3 4 4 1
Thread 0: AAA
Thread 1: BBB
Thread 2: CCC
Thread 3: DDD
UNIX> race3 4 4 2
Thread 0: AAA
Thread 0: AAA
Thread 2: CCC
Thread 2: CCC
Thread 1: BBB
Thread 1: BBB
Thread 3: DDD
Thread 3: DDD
UNIX> race3 4 70 1
Thread 0: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Thread 1: BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Thread 2: CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Thread 3: DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
UNIX> race3 2 70 100000 | grep AB     This call will never have any output because of the mutex.
UNIX> 

Terse advice on mutexes

One of the challenges in dealing with synchronization primitives is to get what you want without constricting the threads system too much. For example, in your jtalk_server program, you will have to have a data structure that holds all of the current connections. When someone attaches to the socket, you add the connection to that data structure. When someone quits his/her jtalk session, then you delete the connection from the data structure. And when someone sends a line to the server, you will traverse the data structure, and send the line to all the connections. You will need to protect the data structure with a mutex. For example, you do not want to be traversing the data structure and deleting a connection at the same time. One thing you want to think about is how to protect the data structure, but at the same time not cause too many threads to block on the mutex if they really don't have to. We'll talk more about this later.

Preemption

Unfortunately, we no longer use Solaris-based systems, and POSIX threads on Linux are preemptive. So, the text below no longer applies, and is not required reading for CS360. However, I leave it here because it is still interesting, especially for those of you studying user/system level threads in your Operating Systems class.

The previous lecture notes talked a bit about preemption. Here is some more specific information about preemption in Solaris. It is kind of confusing, but once you understand all the details, you'll see that the Solaris thread system is very well designed.

There are two kinds of threads in Solaris: user-level threads, and system-level threads. The distinction between the two is kind of confusing, but I'll try to enlighten you. User-level threads exist solely in the running process -- they have no operating system support. That means that if a program has many user-level threads, it looks the same to the operating system as a ``normal'' Unix program with just one thread. In Solaris, user-level threads are non-preemptive. In other words, when a thread is running, it will not be interrupted by another user-level thread unless it voluntarily blocks, through a call such as pthread_exit() or pthread_join().

When one thread stops executing and another starts, we call that a thread context switch. To restate the above then, user level threads only context switch when they voluntarily block. If you think about it, you can implement thread context switching with setjmp()/longjmp(). What this means is that you don't need the operating system in order to do thread context switching. This in turn means that context switching between user-level threads can be very fast, since there are no system calls involved.

So what is a system-level thread? It is a unit of execution as seen by the operating system. Standard non-threaded Unix programs are each managed by a separate system-level thread. The operating system performs time-slicing by periodically interrupting the system-level thread that is currently running, saving its state, and running a different system-level thread. This is how you can have multiple programs running simultaneously. Such an action is also called context switching.

When you call pthread_create(), you create a new user-level thread that is managed by the same system-level thread as the calling thread. These two threads will execute non-preemptively in relation to each other. In fact, whenever a collection of user-level threads is serviced by the same system-level thread, they all execute non-preemptively in relation to each other. All of the programs in the previous threads lecture work in this way.

Let's look at a few more. First, look at preempt1.c. This is a program that forks off two threads, each of which runs an infinite loop. When you run it:

UNIX> preempt1
thread 0.  i =          0
thread 0.  i =          1
thread 0.  i =          2
thread 0.  i =          3
thread 0.  i =          4
thread 0.  i =          5
...
You'll see that only thread 0 runs. (If you can't kill this with control-c, go into another window and kill the process with the kill command). The reason that thread 1 never runs is that thread 0 never voluntarily gives up the CPU. This is called starvation.

Now, you can bind different user-level threads to different system-level threads. This means that if one user-level thread is running, then at some point the operating system will interrupt it and run another user-level thread. This is because the two user-level threads are bound to different system level threads.

One way to bind a user-level thread to a different system level thread is to call pthread_create() in a different way. Look at preempt2.c. You'll see that you give an ``attribute'' to pthread_create() that says ``create this thread with a different system-level thread.'' Now when you run it, you'll see that the two threads interleave -- every now and then, the running thread is preempted, and the other thread gets to run:

UNIX> preempt2
thread 0.  i =          0
thread 1.  i =          0
thread 0.  i =          1
thread 1.  i =          1
thread 1.  i =          2
thread 0.  i =          2
thread 0.  i =          3
thread 1.  i =          3
thread 0.  i =          4
thread 1.  i =          4
thread 0.  i =          5
thread 1.  i =          5
Now, here's the tricky part of Solaris. If a thread makes a blocking system call, then if there are other user-level threads bound to the same system-level thread, a new system-level thread is created and the blocking thread is bound to it. What this does is let the other user-level threads run while the thread is blocked.

This is a fundamental difference of Solaris and older operating systems such as SunOS (the precursor to Solaris). SunOS allows only one system-level thread per process. Therefore, if a user-level thread makes a blocking system call in SunOS, all threads block until the system call completes. This is a drag. The design in Solaris is very nice.

So, look at preempt3.c. First, you should see that the threads are created as user-level threads bound to the same system-level thread. Next, you'll see that the thread 0 first reads a character from standard input before beginning its loop. This is a blocking system call. Therefore, it results this threads being bound to a separate system threads from the main thread and thread 1. Therefore, while it blocks, thread 1 can run. Go ahead and run it:

UNIX> preempt3
Thread 0: stopping to read
thread 1.  i =          0
thread 1.  i =          1
thread 1.  i =          2
thread 1.  i =          3
..
So, thread 0 is blocked, and thread 1 is running. They are thus bound to separate system threads. Now, type RETURN, and thread 0 will start up again, and you'll see that they interleave as in preempt2:
...
thread 1.  i =          3
                                ( RETURN was typed here )
Thread 0: Starting up again
thread 0.  i =          0
thread 1.  i =          4
thread 0.  i =          1
thread 1.  i =          5
thread 0.  i =          2
thread 1.  i =          6
thread 0.  i =          3
...
That's user/system level threads and preemption in a nutshell. Go over these examples again if you are confused.