My first day at the Utah Open Source Conference is just about done. I really need to get to bed so I don't fall asleep during my presentation tomorrow. That would be embarrassing. I only made it down at 6:30pm for the dinner and keynotes, which I'm glad I attended. Mac's talk was great and so was Paul's, although it did seem to drag on. Maybe that's because I needed to go to the bathroom. I also had a chance to visit with a few Linux newbies at my table, which is always fun. But the highlight of my day had to be meeting Harleypig. Thankfully, no speedos were involved.

I just solved a weird MySQL replication problem and thought I would share with you all. First, the background. I have a master and slave running with one-way replication. The slave just sits by waiting for his time to shine, but otherwise doesn't do anything. Well, last week the master had a problem with the filesystem. I can't recall exactly what it was, out of space or something. It confused the heck out of the master and so it basically shut down. One of my co-workers fixed the problem and got the master running again, but the slave was in a pickle. Here is the error it was showing:

Relay_Master_Log_File: mysql-bin.031
     Slave_IO_Running: Yes
    Slave_SQL_Running: No
           Last_errno: 0
           Last_error: Query 'DELETE FROM foo WHERE bar = 1' caused different errors on master and slave.
                       Error on master: 'Got error %d from table handler' (1030), Error on slave:
                       'no error' (0). Default database: 'baz'
  Exec_master_log_pos: 118871

Because it thought the command failed on the master, it refused to continue. I can't say it's an altogether bad plan since data integrity is generally the main theme of a database (yes, cue the jokes about my using MySQL in the first place). The question became, how do I get the slave to start up again. "SLAVE STOP; SLAVE START" didn't have any effect.

The trick was suggested to me by a post at mysql.com which pointed out a tool new to me, mysqlbinlog. See I figured the simplest thing would be to restart replication at the step just after the "failed" transaction, since I knew that transaction had actually succeeded. But I have no idea how the binlog counters work, so I couldn't just make up numbers. It's some kind of binary offset. Well, mysqlbinlog will show it to you.

# mysqlbinlog mysql-bin.031 -j 118871 |less

Which of course showed me this:

# at 118955
#080605 18:59:09 server id 1  log_pos 118955    Query   thread_id=3218  exec_time=0     error_code=0

So on my slave I restarted replication at offset 118955 and like magic, the slave ripped through the binlogs and caught up in practically no time at all.

I guess I could format this post in a "Cheers and Jeers" style, but that seems a little clichéd. See I was replacing the ink cartridge in my HP J5780 printer (which is a fine unit, I might add). So on the one hand I was miffed that the "starter" ink cartridge was so tiny. But to my surprise I discovered that the replacement cartridge came with a prepaid return envelope for sending the empty cartridge back to HP.

Recycling cartridges isn't anything new. Office stores have been doing that for a long time. They refill them with ink and sell them to you again (at outrageous prices). But what is impressive is that HP took a step to make it so darn easy. I'll shamefully admit that I've never recycled my cartridges before because I just never remember. I rarely go to office stores and even then I never remember to take the empties. So they just end up in the trash, which isn't good. This obviously is better.

The one question I have though is whether shipping this cartridge to Nashville, Tennessee is better for the environment than either A) sending it to the local landfill or B) dropping it off at the local Staples and sending a whole bunch of cartridges to Nashville. I imagine B is more efficient, but I'm not sure about A. But since I lack the resources and/or willpower to research it, I'll just send off this old cartridge in this handy dandy green envelope and assume that I'm helping the planet. Helps me sleep better at night, you know.

Hans and I were discussing QoS the other day, specifically regarding using Wondershaper from the LARTC. I had managed to mess mine up and I subsequently noticed a horrible turn for the worse in my VoIP calls. Wondershaper has to be adapted for use by OpenWRT and in the process I misspelled sch_ingress.o as sch_insmod.o. Too much insmodding that day, I think. The net effect was that download speeds were not shaped at all.

Once I got it corrected, I decided to do a few tests just to confirm that using Wondershaper actually made a difference. I'll cut to the chase for the lazy: it did. I made 45 second calls to music on hold from my softphone, Twinkle. In the background I had Wireshark running. I used the RTP analyzer in Wireshark to look at the statistics after all was said and done. I used both versions of Wondershaper, the CBQ and HTB. I had a single download running the whole time eating up all spare bandwidth.

With no shaping: 4.4% loss (95 packets), 60ms jitter
With CBQ Wondershaper: 0.2% loss (5 packets), 35ms jitter
With HTB Wondershaper: 0.3% loss (6 packets), 28ms jitter

So my unscientific conclusion is that both versions of Wondershaper work about the same and they both make a huge difference. I could easily hear the packet loss on the first call, but not so much on the other two calls.

I built a Twinkle 1.1 Ubuntu package the other day. I wanted to try out the new buddy lists feature and the currently available version is only 1.0.1. This package is built on Gutsy Gibbon. I make no warranties of its successful functioning on your machine. But it does work just fine for me.

twinkle_1.1-0_i386.deb

Another great opinion on Net Neutrality which closely (if not exactly) mirrors my own. For those too lazy to go and read for themselves, here's a quick snippet.

We need policy to help cut a path for more competition, rather than protecting incumbents -- a Bandwidth Competition Act of 2008, not bogus net neutrality. All takers should be allowed access to poles or underground conduits. This is where neutrality should be enforced, instead of being a choke point.

As I've long said, a government bureaucracy isn't going to solve the problem. It's going to create less incentive for Internet companies (like mine, full disclosure) to even toss their hat in the ring. Try forming your own telephone system and you'll know what I mean. The rules are ridiculously complicated and it takes an army of lawyers to sort through them. Please please please don't turn the Internet into the phone system.

My bud Hans and I tonight tested out encrypted VoIP with ZRTP. I noticed a while back that Twinkle supports it and have wanted to test it out, but none of my desk phones support ZRTP.

It was fun. When the call terminated, Twinkle displayed a cute message about verifying the SAS (short authentication string). It was 4 character (hprj, if you're curious) that represented our encryption key. It's the way ZRTP verifies that a man-in-the-middle attack is not underway. There was a padlock icon which we both clicked to verify that the SAS was correct. I'm not sure what if anything happened because of that, except that we both verified that our SIP phones have not been tapped by the feds.

In the SDP, ZRTP is advertised with "a=zrtp". It's not a separate protocol per se. The actual codec was selected through the normal means (we used speex/16000). Looking at the RTP data, I see a whole bunch of "AES256", "SHA256" and "DH4096". Presumably that's part of the ZRTP negotiation. I didn't delve further. What I see though is that the encrypted data is simply represented as Speex RTP, but the actual data has been scrambled so it would be meaningless to a passerby.

Based on this testing, I predict good things for ZRTP. It was quite painless to use as a caller. As long as it's enabled by default in the phone, there's really nothing else that a user has to do to use it. The SAS is short and you only have to verify it if you care. Phil Zimmerman says that you don't even have to verify the SAS every time. Just once in a while is good enough. And obviously anytime you're conducting private business (which is not the same thing as illegal business). The simple fact that ZRTP is used every time means that you can't tell whether a call is valuable or not just based on it being encrypted.

The one possible failure of ZRTP is that it doesn't hide any of the signalling data, so a spy would be able to see who you were calling. That problem would be quite hard to solve. I'm not sure of the benefit either as the cost to mask that information is much higher. You pretty much have to know all the routing information ahead of time. Even then, an eavesdropper could still see the two IP addresses involved, which will give away some amount of information. So for now, ZRTP is a good solution.

I ran across a site today named Leads Online. It's a tool for law enforcement to collaborate with pawn shops and the like. Essentially the pawn shop uploads all of its transactions at the end of the day and then if a law enforcement agency is looking for some stolen stuff, they search through the database. Leads Online tells how it's such a convenient and easy service, and helps good guys catch the bad guys.

My first thought was substantially different. I don't really relish in the thought of law enforcement having carte blanch access to business records, possibly without warrants. There's some good potential for abuse there. I recently read through all the Idaho Falls city ordinances (yes, it was actually interesting) and I'm aware that pawn shops are required to keep records of purchases for 2 weeks (IIRC), just in case a question of ownership arises. But there is not a requirement for them to proactively send those details to the police department. That seems to be a fair trade.

One immediate concern I have is that being a private company, they are not subject to laws that courts and police would be. They don't have to answer Freedom of Information Act responses. They aren't subject to public oversight. I don't think we should be outsourcing our key public infrastructure to private industry whose primary motivation is the almighty buck.

Even more than that, what really scares me isn't the intended uses but the ways the system could be expanded. There are all sorts of ways the database could be reused for marketing purposes, thus invading the privacy of sellers and buyers alike. There are pathetic privacy laws in the country, so it's hard to believe that anyone would have any recourse.

And what if law enforcement decided they wanted to browse through the database? The obvious abuse would be looking for religious material. Yeah, it seems pretty unlikely but the point isn't that we trust good people. I know a few law enforcement people and I would have no qualms with them using something like this. The point is that just as there are unscrupulous citizens, there are unscrupulous police. We need to make sure the system fails gracefully and I don't believe this one does.

I had the dubious honor the last couple days of breaking up an argument between a Linksys PAP2T and a Linksys WRT54G. You might think they would work well together, coming from the same family and all. Well not really.

The problem was caused by the WRT54G (version 5, btw) which refused to route SIP traffic back to the PAP2T. It was allowing HTTP and DNS traffic just fine, but it balked at SIP for some reason. The symptoms were that the PAP2T kept sending REGISTER requests without a WWW-Authenticate header. For a long time I thought it was a setting on the PAP2T, but instead it was because the PAP2T never got the WWW-Authenticate challenge, so it obviously couldn't respond. A dead giveaway should have been that the PAP2T would send 5 REGISTER requests in a row. It must have figured there was network congestion or something, so it was retransmitting.

The solution was to upgrade the WRT54G from version 1.00.0 to 1.02.2. Some bug in the 1.00.0 firmware caused the issue and now it's resolved. I haven't looked at the release notes to see what it was, and frankly I probably won't. I've never really liked the version 5 WRT54Gs and this just helps solidify it.

Generally speaking I find most of what Mark Cuban has to say fairly good, but today I have to disagree. His latest rant against P2P I think starts off with faulty premises, thus the conclusions don't hold.

As a consumer, I want my internet experience to be as fast as possible. The last thing I want slowing my internet service down are P2P freeloaders. Thats right, P2P content distributors are nothing more than freeloaders. The only person/organization that benefits from P2P usage are those that are trying to distribute content and want to distribute it on someone else's bandwidth dime.

Have we conclusively shown that P2P slows down the Internet? I'd like to see the proof of that. It's apparent that what he's mainly complaining about here is the amount of upload traffic. If somebody downloaded the same content via non-P2P, aka the non-freeloading way, they would eat up the same amount of download. So the only difference is the extra upload with P2P. Is that really a problem though? I can tell you, my network has upload to spare. I would estimate that our upload traffic is about half of what the download is at.

Does anyone really think its free ? That all the bandwidth consumed with content being distributed by P2P isn't being paid for by someone ? That bandwidth is being paid for by consumers.

When I buy a DS3, I get 45mbps of bandwidth in each direction. I have to have enough to support the download demands of my customers and I just get the upload to boot. It's just sitting there. So yeah, the consumer is paying for it but there's no way to recover that cost. Even if every consumer stopped uploading, they would still pay the same amount. Why not put it to some good use?

And there is a good use. If that bandwidth isn't used by the P2P content distributors, they end up forking out additional money on their end for the extra bandwidth. They then have to pass the cost on to consumers. Why should consumers pay twice?

Consumers who pay for personal, not commercial applications. When consumers provide their bandwidth to assist commercial applications, they are subsidizing those commercial applications which if it isn't already, should be against an ISPs terms of service.

I'm a little surprised honestly, to see Cuban talk about consumers as people who just eat and eat whatever the big media companies feel generous enough to feed them. What about the generation of the long tail? Consumer created content and all that jazz? I realize that most of what's going on with P2P is unauthorized distribution of other peoples' copyrighted works, but baby, bathwater. You know the saying.

And finally, to round out my argument I will point out that blocking P2P isn't an easy task. P2P software designers have made is hard on purpose. That means ISPs have to shell out for big hardware to do the task. Who ends up eating those costs? I would have to redesign my network in order to even accommodate the devices, which would add even more.

I won't even get into the whole neutrality debate. That's a can of worms for another day.