01 Mar, 2023

Race Conditions in Multi-Threaded Applications

Vulnerability Assessment as a Service (VAaaS)

Tests systems and applications for vulnerabilities to address weaknesses.

What is multi-threaded application?

A multi-threaded application is a computer program that uses multiple threads to perform multiple tasks or operations concurrently. A thread is a lightweight process that can be executed independently of other threads within the same process. By using multiple threads, a program can improve its performance, responsiveness, and scalability.

In a multi-threaded application, each thread runs a separate sequence of instructions, which can execute concurrently with other threads in the same program. This enables a program to perform multiple tasks or operations at the same time, such as processing user input while also performing background tasks like file I/O (input/output) or network communication.

Multi-threading can be used in a variety of applications, including web servers, database management systems, and multimedia applications. A multi-threaded application can provide a more responsive user interface, as background tasks can run concurrently without blocking the user interface thread.

What is race conditions vulnerability?

In cybersecurity, a race condition occurs when a security vulnerability arises from the timing and ordering of events in a system. Race conditions can enable attackers to gain unauthorized access to resources or information, bypass security controls, or escalate privileges.

For example, in a multi-user system, a race condition could occur if two users attempt to access the same resource simultaneously. If the system does not properly enforce access controls or resource locking mechanisms, one user may be able to gain access to the resource before the other, potentially compromising its confidentiality, integrity, or availability.

Another example of a race condition in cybersecurity is a time-of-check to time-of-use (TOCTTOU) vulnerability. This occurs when a program checks a condition or resource at one point in time, but then uses or relies on that condition or resource at a later point in time. If an attacker can modify or manipulate the condition or resource in the intervening time period, they may be able to exploit the vulnerability and gain unauthorized access or execute arbitrary code.

To prevent race conditions in cybersecurity, developers and system administrators should implement secure coding practices, such as proper use of synchronization mechanisms and access controls, as well as thorough testing and auditing of their systems to identify and address vulnerabilities. It’s also important to keep software and systems up-to-date with security patches and updates to mitigate known vulnerabilities.

What is thread?

In computing, a thread is the smallest unit of execution that a program can schedule for execution. A thread is a sequence of instructions that can be executed independently of other threads within the same program. Multiple threads can run concurrently within the same process, allowing a program to perform multiple tasks or operations simultaneously.

By running multiple threads concurrently, an application can perform multiple tasks simultaneously, which can lead to faster and more responsive performance. By utilizing multiple threads, an application can more easily scale to handle larger workloads and increased user demand. Using threads can help to break up complex tasks into smaller, more manageable pieces of code, making it easier to write, test, and maintain large and complex applications.

Threads can communicate and share resources with other threads in the same process, such as variables, memory, or files. However, this also introduces new challenges, such as race conditions, deadlocks, and synchronization issues, which can cause unpredictable and undesired behavior if not handled properly.

To develop a program that uses threads, developers must carefully design their program to ensure that shared resources are accessed in a safe and synchronized manner. This requires the use of synchronization techniques such as locks, semaphores, or atomic operations to ensure that only one thread can access a shared resource at a time. Additionally, developers must test and debug their program to identify and fix any race conditions or synchronization issues that may arise.

What is race conditions in multi-threaded application

Race conditions in multi-threaded applications occur when two or more threads access a shared resource simultaneously and the final outcome of the program depends on the order in which the threads execute. This can result in unpredictable and undesired behavior, including data corruption, program crashes, or security vulnerabilities.

In a multi-threaded application, threads often share access to resources such as memory, files, or network connections. If two or more threads attempt to modify the same resource simultaneously, a race condition can occur. For example, if one thread is reading from a file while another thread is attempting to delete it, the final outcome depends on which thread executes first.

Race conditions can be difficult to detect and reproduce, as they often occur sporadically and depend on factors such as the system load and the timing of thread execution. They can also be difficult to fix, as they require careful synchronization and coordination between threads to ensure that shared resources are accessed in a safe and predictable manner.

To avoid race conditions in multi-threaded applications, developers should use synchronization techniques such as locks, semaphores, or atomic operations to ensure that only one thread can access a shared resource at a time. They should also carefully manage the order in which threads execute to ensure that shared resources are accessed in a consistent and predictable manner. Additionally, thorough testing and debugging can help to identify and address race conditions before they can cause serious problems.

How users can detect race conditions in multi-threaded application

Race conditions in multi-threaded applications can manifest in various ways, but some common application behaviors that may indicate a race condition include:

Incorrect or unexpected output. If the output of the application is incorrect or unexpected, it may indicate that multiple threads are accessing the same resource concurrently and producing inconsistent results.

Crashes or freezes. If the application crashes or freezes unexpectedly, it may be a sign that a race condition is causing the program to hang or become unresponsive.

Inconsistent behavior. If the application’s behavior is inconsistent, it may indicate that different threads are accessing shared resources in different ways, leading to unexpected results.

Performance issues. If the application’s performance degrades under high load or concurrency, it may indicate that multiple threads are contending for the same resources and causing delays or bottlenecks.

Deadlocks. A deadlock occurs when two or more threads are blocked waiting for each other to release a resource, resulting in a situation where none of the threads can make progress. Deadlocks can be a sign of a race condition if they occur when multiple threads are accessing shared resources.

How pentesters or developers can detect race conditions in multi-threaded application

Code review. One way to detect race conditions is to perform a thorough code review of the application’s source code. Developers can look for areas of the code where shared resources, such as variables and data structures, are accessed by multiple threads without proper synchronization.

Example 1 

When multiple threads access the same shared resource without proper synchronization, a race condition can occur. For example, if two threads try to increment the value of a shared variable at the same time, they may both read the same value and then write back the same incremented value, resulting in the variable being incremented only once instead of twice.

				
					import threading

# A shared variable
counter = 0

def increment():
    global counter
    for i in range(100000):
        counter += 1

# Create two threads that both call the increment function
thread1 = threading.Thread(target=increment)
thread2 = threading.Thread(target=increment)

# Start the threads
thread1.start()
thread2.start()

# Wait for the threads to finish
thread1.join()
thread2.join()

# The expected result is 200000, but due to the race condition, the 
# actual result may be lower
print("Counter value:", counter)
				
			

In this example, we have a shared variable called counter that is incremented by each thread in a loop. The increment function increments the counter variable by 1 for 100000 times. However, since we have two threads running the increment function simultaneously, they may read and write to the counter variable at the same time, causing a race condition. As a result, the counter variable may not be incremented as expected, and the final value may be lower than the expected value of 200000.

Example 2

If multiple threads write to the same file without proper synchronization, a race condition can occur where the writes get interleaved and the output becomes garbled.

				
					import threading

def write_to_file(file_name, text):
    with open(file_name, 'a') as f:
        f.write(text)

threads = []

# Create 5 threads that write to the same file
for i in range(5):
    t = threading.Thread(target=write_to_file, args=('output.txt', f'Thread {i}\n'))
    threads.append(t)
    t.start()

# Wait for all threads to finish
for t in threads:
    t.join()
				
			
				
					Thread 1
Thread 2
ThThread 3
read 4
d 0
				
			

In this code, multiple threads are created to write text to the same file output.txt. However, since there is no synchronization between the threads, the text from each thread can become interleaved and the output can become garbled. For example, the output in output.txt could look something like above. 

Example 3

When multiple threads update the same data structure without proper synchronization, a race condition can occur. For example, if two threads try to add an item to the same list at the same time, they may both read the same length of the list, insert their item, and then write back the new length of the list, resulting in one of the items being overwritten.

				
					import threading

# A shared list
my_list = []

def add_item():
    global my_list
    # Read the current length of the list
    current_length = len(my_list)
    # Add an item to the list
    my_list.append(threading.current_thread().name)
    # Write back the new length of the list
    new_length = len(my_list)
    print(f"{threading.current_thread().name} added an item to the list. List length: {new_length}")
    # Check for race condition
    if new_length != current_length + 1:
        print(f"Race condition detected by {threading.current_thread().name}! List length should be {current_length + 1}, but is {new_length}.")

# Create multiple threads that add an item to the list
t1 = threading.Thread(target=add_item)
t2 = threading.Thread(target=add_item)
t1.start()
t2.start()
t1.join()
t2.join()

				
			

In this example, two threads are trying to add an item to the same list. The add_item() function reads the current length of the list, adds an item to the list, and then writes back the new length of the list. However, because there is no synchronization between the threads, a race condition can occur where both threads read the same length of the list, insert their item, and then write back the same new length of the list, resulting in one of the items being overwritten.

When you run this code, you may see output like the following:

				
					Thread-1 added an item to the list. List length: 1
Thread-2 added an item to the list. List length: 1
Race condition detected by Thread-1! List length should be 2, but is 1.

				
			

In this case, the race condition occurred because both threads read the same length of the list and then wrote back the same new length of the list, resulting in only one item being added instead of two. The if statement in the add_item() function detects the race condition and prints a message to the console.

Example 4 

In this example, there are two threads and two locks. Each thread acquires one lock and then tries to acquire the other lock. If one thread acquires lock1 and the other thread acquires lock2, then a deadlock occurs because each thread is waiting for the other thread to release the lock it needs.

				
					import threading

# Create two locks
lock1 = threading.Lock()
lock2 = threading.Lock()

def thread1():
    lock1.acquire()
    lock2.acquire()
    # Do something
    lock1.release()
    lock2.release()

def thread2():
    lock2.acquire()
    lock1.acquire()
    # Do something else
    lock2.release()
    lock1.release()

# Create two threads and start them
t1 = threading.Thread(target=thread1)
t2 = threading.Thread(target=thread2)
t1.start()
t2.start()
t1.join()
t2.join()

				
			

Testing. Pentesters can perform comprehensive testing of the application using tools such as stress testing and load testing to simulate high traffic and concurrent access scenarios. This can help to identify potential race conditions and synchronization issues.

Debugging. If a race condition is suspected, devs can use a debugger to step through the code and track the state of variables and other resources as they are accessed by different threads. This can help to identify the root cause of the race condition.

Logging. Devs can add logging statements to the code to track the sequence of events and identify any inconsistencies or unexpected behavior that may indicate a race condition.

TOP CWEs for race conditions in multi-threaded application

CWE-362: Concurrent Execution using Shared Resource with Improper Synchronization (‘Race Condition’) This CWE occurs when multiple threads execute concurrently, access a shared resource without proper synchronization, and result in an unexpected interleaving of the resource. This can lead to inconsistent data, incorrect results, and crashes.

CWE-835: Loop with Unreachable Exit Condition (‘Infinite Loop’) This CWE occurs when a loop is executed in a multi-threaded environment, and one thread modifies the loop variable while another thread is waiting for the loop to terminate. This can lead to an infinite loop, and the program becomes unresponsive.

CWE-831: Improper Synchronization This CWE occurs when multiple threads access a shared resource without proper synchronization, leading to a race condition. This can result in inconsistent data, incorrect results, and crashes.

CWE-833: Deadlock This CWE occurs when two or more threads are blocked, waiting for each other to release a resource that they need to proceed. This can result in a deadlock, where the threads are unable to proceed and the program becomes unresponsive.

CWE-834: Improperly Controlled Modification of Atomic Operation This CWE occurs when multiple threads modify an atomic operation, such as an integer or a boolean, without proper synchronization. This can lead to inconsistent data and incorrect results.

TOP CVEs for race conditions in multi-threaded application

CVE-2016-2324 – Apache Tomcat Security Manager Bypass via Thread Context Classloader This CVE affects Apache Tomcat and is related to a race condition in the Security Manager implementation. It can be exploited to bypass the Security Manager and gain unauthorized access to resources.

CVE-2018-16640 – WordPress 4.9.8 Privilege Escalation This CVE affects WordPress and is related to a race condition in the update mechanism. It can be exploited to escalate privileges and execute arbitrary code on the affected system.

CVE-2019-11477 – Linux Kernel SACK Panic This CVE affects the Linux kernel and is related to a race condition in the TCP implementation. It can be exploited to cause a denial-of-service (DoS) attack by crashing the system.

CVE-2019-11478 – Linux Kernel SACK Slowness This CVE also affects the Linux kernel and is related to a race condition in the TCP implementation. It can be exploited to cause a DoS attack by slowing down the system.

CVE-2020-25223 – Apache Struts2 Race Condition Vulnerability This CVE affects Apache Struts2 and is related to a race condition in the TokenInterceptor implementation. It can be exploited to bypass authentication and gain unauthorized access to resources.

CVE-2021-32027 – Apache Tomcat jsp:forward Vulnerability This CVE affects Apache Tomcat and is related to a race condition in the jsp:forward implementation. It can be exploited to bypass security constraints and gain unauthorized access to resources.

CVE-2021-31800 – Microsoft Exchange Server Remote Code Execution Vulnerability This CVE affects Microsoft Exchange Server and is related to a race condition in the Exchange Control Panel (ECP) application. It can be exploited to execute arbitrary code on the affected system.

Real world examples 

In January 2021, a bug was reported in Apple’s macOS Big Sur that could cause a race condition when multiple processes were attempting to access the same file at the same time. This could result in data corruption and lead to system crashes.

In March 2021, a vulnerability was reported in Microsoft Exchange Server that could allow attackers to execute arbitrary code on the affected system by exploiting a race condition in the application.

In April 2021, a vulnerability was reported in Apache Cassandra that could allow attackers to execute arbitrary code on the affected system by exploiting a race condition in the application’s handling of TLS/SSL connections

In May 2020, Microsoft Windows faced a vulnerability related to a race condition in the Windows Graphics Device Interface (GDI). Attackers could exploit this vulnerability to escalate privileges and execute arbitrary code on the affected system.

In July 2020, Apple macOS faced a vulnerability related to a race condition in the kernel code. Attackers could exploit this vulnerability to gain elevated privileges and execute arbitrary code on the affected system.

In August 2020, the Android operating system faced a vulnerability related to a race condition in the binder driver. Attackers could exploit this vulnerability to gain elevated privileges and execute arbitrary code on the affected system.

How to protect application from race conditions in multi-threading

Synchronization: Use synchronization mechanisms such as locks, semaphores, and monitors to ensure that only one thread can access shared resources at a time. Proper synchronization will prevent multiple threads from accessing or modifying the same resource simultaneously.

Atomic operations: Use atomic operations that guarantee that an operation will be performed in a single, indivisible step. Atomic operations are especially useful for simple operations on shared variables, such as incrementing or decrementing a counter.

Thread safety: Make sure that the code is designed to be thread-safe from the start. Thread safety can be achieved by avoiding global variables, using immutable data structures, and separating state from behavior.

Code reviews: Conduct code reviews to ensure that the application is designed with concurrency in mind. Code reviews can identify potential race conditions and ensure that the code is written in a way that is easy to understand and maintain.

Conclusion

In conclusion, race conditions in multi-threaded applications can be a significant source of security vulnerabilities and other issues that can impact the performance and stability of the application. These issues can arise when multiple threads attempt to access shared resources without proper synchronization or when the application is not designed with concurrency in mind.

To address race conditions in multi-threaded applications, developers must implement synchronization mechanisms, use atomic operations, design the code to be thread-safe from the start, perform thorough testing, and conduct code reviews. Additionally, developers must stay up-to-date with the latest security vulnerabilities and patches to ensure that their applications are protected against new and emerging threats.

Overall, while race conditions can be challenging to detect and mitigate, they can be prevented with proper design, testing, and implementation of synchronization techniques, leading to more secure and reliable multi-threaded applications.

Other Services

Ready to secure?

Let's get in touch