Enforce Alignment to Avoid False Sharing

I have been working on a C++ TCP server that utilizes Windows IO Completion Ports. So far, the toughest challenge has been maintaining the scalability of the server. Among all the concurrency problems, the one I absolutely try to avoid is false sharing, where one CPU modifies a piece of data that invalidates another CPU’s cache line.

The symptom of false sharing is extremely difficult to detect. As preventive measure, I grouped all shared data carefully into a single object so I can easily visualize the potential contentions. And I add padding accordingly if I think contention exists.

Then I came across a chapter in Windows Via C/C++, it provided a cleaner solution.

Just align them to different cache line

My TCP server follows the proactor pattern, so I have a I/O thread pool to handle send and receive requests and dispatch events. Naturally, the threads have some piece of data that they share in read, write or both.

Here’s just a dummy example.

class CSharedData
{
public:
	CSharedData() : data1(0), data2(0), data3(0) {}
	unsigned int data1; // read write
	unsigned int data2; // read write
	unsigned int data3; // read write
};

Since my processor’s cache line is 64 bytes, the data structure above is definitely going to cause contention,Ā  say data1 is updated by one thread, and data2 is read by another. To solve this, just simply force every read write data member to be in different cache line through __declspec(align(#)).

class __declspec(align(64)) CSharedData
{
public:
	CSharedData() : data1(0), data2(0), data3(0) {}
	__declspec(align(64))
		unsigned int data1;
	__declspec(align(64))
		unsigned int data2;
	__declspec(align(64))
		unsigned int data3;
};

Thoughts

With __declspec(align(#)), you can even specify the alignment of the data structure itself. This is very useful for putting shared objects in containers like std::vector. See Okef’s std::vector of Aligned Elements for why this is a bad idea.

It would be nice if the alignment can be changed at runtime base on processor spec. I know it doesn’t make sense technically, but it is on my wishlist. šŸ™‚

6 thoughts on “Enforce Alignment to Avoid False Sharing

  1. Ofek Shilon says:

    About runtime alignment – is your data is instantiated on the heap only? If so, maybe it’s possible to derive different-alignment versions from a common base, and choose which to instantiate based on CPU attributes at runtime? (haven’t tested this, just a thought)

  2. Jon says:

    I’d be interested to see performance data on what happens with and without the padding.

    @Ofek: wouldn’t that imply a (small) virtual function call overhead?

    • “Iā€™d be interested to see performance data on what happens with and without the padding.”

      Sure. I actually wrote a small test to verify that the align(#) keyword is doing something.

      I spin up three threads to increment some numbers. On my quad-core CPU (I5-750), the unaligned version takes three times longer than the aligned version. Best part, you can’t tell from task manager because the CPU monitor counts cache misses as busy.

      Here is the code. http://codepad.org/jZZSdoxJ

Leave a Reply

Your email address will not be published. Required fields are marked *