27 March 2008 - 21:21Billions and Billions Served
Our friend Gavin at Suretec brought this ComputerWorld article to our attention, describing a white paper Oracle just published about benchmarking their OID server with a 2 billion entry database. I guess they seem to think this was a world's first or something, topping Novell's boasts of a 1 billion entry database a few years back. I hate to break it to the OID guys, but in fact I benchmarked OpenLDAP with a 5 billion entry database here in Symas' labs a month or two ago. "What took you so long?"
I'm not even sure why Oracle thinks it's a big deal. I just threw the test together on a whim, it wasn't even much of an effort. I mentioned it in passing on the UMich LDAP mailing list while explaining to a Microsoft engineer exactly how broken ActiveDirectory actually is. Granted, the server I used maxed out at 60,000 searches/second, but it only had 8 Opteron cores (at only 1.9GHz each, not the speediest), 16GB of RAM, and a single Gigabit ethernet in use. Oracle reported a peak of around 101,800 searches/second on an SGI Altix with 64 Itanium2 cores, 256GB of RAM, and 8 dedicated Gigabit ethernets. Frankly, given that class of hardware resources, the OID numbers are pathetic. In fact we benchmarked OpenLDAP on SGI Altix back in 2005 and 2006, and got far better latencies even then, using early versions of OpenLDAP 2.3, than Oracle is reporting now. We're talking sub-millisecond search latencies, compared to their several milliseconds. I suspect that with OpenLDAP 2.4 today the performance delta would be even greater.
In reading thru their white paper, I can certainly agree with their assessment of the business case. If you're a telco and you're targeting the market in Asia, you need infrastructure that can handle a billion or two accounts, no question. Of course, that's still just thinking small. The world population today is around 6.6 billion people. Right now, today, you can load a single OpenLDAP server with an entry for every person on earth, and get useful work done with it. I suppose only an Orwellian Big Brother would ever need such a directory, and I don't really want to think about that here. But as a software engineer, it's nice to know that my code can handle that.
Some more careful reading of their paper raises some serious questions about the meaningfulness of their tests though. For instance, they only ran their search rate test for 30 minutes. With an average rate of only 101,800 searches/second, in 30 minutes they only hit at most 183 million entries in their DB. Less than 10%. For a test result to have any actual validity, you want to know the sustained search rate over a long enough period of time to know that most of the database was touched, otherwise you may only be seeing a best-case burst rate that never occurs in the real world, after the various software caches have been saturated. E.g., for this search test to be valid they should have run it for at least 10-11 times longer, to give a chance to touch a meaningful portion of the database. (Even then, given randomly generated search targets, it's pretty likely that you won't hit 100% of the entries, but it would at least prove that you can sustain this throughput without hitting some unexpected resource exhaustion...)
One of the other bragging points in their test seems to be that they handled up to 16,000 concurrent clients without degradation. On the face of it that's a pretty impressive feat, but it just doesn't make sense. They're using 8 dual-socket quad-core machines for load generation, so a total of 64 Xeon cores. The machines are decent enough, but you have to remember that these tests are conducted with SLAMD, which is all Java based, and there's a significant resource footprint for those load generator threads. In my own tests with SLAMD the load generators tend to max out at around 15-16 threads per CPU core. By the time you get to about 40 threads per CPU you're spending the majority of CPU cycles in scheduling overhead, and hardly any cycles doing any useful load generation. Here they're talking about 250 threads per core, which is basically pointless. They may have had 16,000 connections open at once, but with an average latency of around 160ms, each connection was only getting about 6 requests/second served. If you're a telco getting response rates like that, most of your customers will revolt and switch to another carrier with snappier response. (To put that in perspective - human factors research tells us that any response within 50ms is perceived as immediate. Anything above 100ms is perceived as a noticeable delay. Take a telco database with 160ms response time, and add on the latencies of the inevitably large number of software layers above that, and you have a recipe for extremely unhappy customers.) These raw numbers look impressive at first, but when you do the math, you see there's not much here to be proud of.
I'll note that in the benchmark results we've posted in previous articles here, all of our tests were run with sufficient duration to cycle through all of the entries in the DB several times over. Granted, those were much smaller directories too, so cycling OpenLDAP thru the DB a few times only took a matter of minutes. But in that same amount of time most of the other servers we tested couldn't even cycle through once. We don't get the opportunity to test on a big SGI Altix very often, but I'll be happy to rerun the tests Oracle described with OpenLDAP the next chance we get.
I guess Larry Ellison shouldn't feel too bad, since Oracle also owns BerkeleyDB. They can honestly claim that the fastest and most scalable directory software in the world is powered by Oracle technology. It just isn't Oracle Internet Directory. It wasn't in 2005, and it still isn't today. Only OpenLDAP.
No comments:
No trackbacks: