A few weeks ago I completed an upgrade from OpenSER 1.2 to Kamailio1.5. Overall it's been working quite well and actually was not nearly as hard as I feared it might be. Now that I've got this new version with a ton of new bells and whistles, I thought I'd try out a few of them. One of the most exciting is shared memory hashtables.

First a quick primer on hashtables. Currently whenever the proxy needs a value from the database it simply queries the database and gets the value. Straightforward, right? But as the system grows, the database can be greatly put under strain. In my Kamailio cluster the single biggest CPU user is in fact the MySQL database by a large margin. And since the database changes pretty infrequently it would be nice if we could cache those values.

The problem is that Kamailio is a multi-process daemon, so what's cached in one isn't going to be cached in another. Enter shared memory hashtables. The same cache is shared among all the processes. Usage is pretty simple. Here's an example:

# IP based authentication
if ($sht(ht1800=>$var(fU)::ipauth) == null){
	if (is_user_in("From", "ipauth")){
		$sht(ht1800=>$var(fU)::ipauth) = "yes";
	}
	else{
		$sht(ht1800=>$var(fU)::ipauth) = "no";
	}
	$var(ipauth) = $sht(ht1800=>$var(fU)::ipauth);
}
else{
	$var(ipauth) = $sht(ht1800=>$var(fU)::ipauth);
}

if ($var(ipauth) == "yes"){
	# do whatever...
}

Don't worry too much about the nitty-gritty. First we check to see if the value is cached (($sht(ht1800=>$var(fU)::ipauth) == null)). If it's set, we just use the cached value and continue. If not, we have to ask the database (is_user_in("From", "ipauth")) and then save the result in the cache. It actually is that easy.

But the $64,000 question, has it helped? Luckily I thought ahead and got some data before I made the change. I grabbed the query log for a day before and after the change. I made sure to select two days where the call usage was the similar (1% difference). Overall there was a 20% reduction in SQL queries.

I only put in the one change as a proof of concept. Since it sure seems to have panned out I'll continue wrapping all my database calls with hashtable caches. One thing I still need to work out is how to invalidate the cache when I make a change through my provisioning system. And once I'm able to upgrade to Kamailio 3.0 I'll switch over to memcached, which has all the benefits of a shared memory hashtable but it's also clustered. Ooh, I'm so excited!

The other day my boss and I managed to create a call routing loop between two of our phone systems. It was kinda fun, in one of those painful ways. Here's how it went down.

Phone system A has extensions in the range of 2XXX. Phone system B has extensions in the rage of 1XXX. The PBXes have an IAX2 trunk between them to allow direct dialing. The phones themselves have a dialplan which recognizes 1XXX and 2XXX as patterns, which allows for faster dialing. Users must dial a 9 to reach an outside line.

There was just a slight problem. Due to a misconfiguration on my part, B was set up to route any unknown 1XXX extensions back over to A. For a real extension, say 1000, the rule wouldn't match because B would know to send that call to the local user. But let's say that somebody's boss forgot to dial a 9 when dialing a long distance number. So the phone sees the first 4 digits, 1 + area code, and assumes it's an extension. A sends the call to B. Well, that extension didn't exist so B sent the call back to A. That would be the point when the fun began.

I would have thought that the PBXes would create a whirlwind of calls until the original call eventually timed out and broke the chain, thus bringing all the calls to a halt. But somehow that isn't what happened. The system stabilized at 400 or so active channels. Sat that way for a few hours I believe. What finally did stop it was my removing the rule to allow 1XXX over the IAX2 trunk from B to A. Within just a few seconds all the calls came to a grinding halt (in a good way).

I've decided then that having local extensions that start with a 1 is a less than desirable thing. It's not the end of the world, but avoiding it can relieve some potential headaches so I intend to eschew it where possible.