Today is the great SOPA/PIPA blackout. These are two proposed bills in Congress which would create a censorship regime for the Internet, much like China, Iran, and Syria have. And we know how great those governments are, so why wouldn't we follow in their footsteps? Wikipedia and Google are the most notable, but thousands of other sites went offline in protest. We at zmonkey.org fully support these actions. Draconian Internet censorship laws are not the solution to anything, especially not when the primary problem is an outmoded business model in Hollywood. If you value your freedom on the Internet, join me in telling your representatives how horrible this proposal is.

I started using Jigl for my private photo gallery, and quite like it. In the process of generating the HTML, I received the following error:

convert: missing an image filename `0601091401.jpg' @ convert.c/ConvertImageCommand/2775

The problem turned out to be a zero byte file named 0601091401.jpg, but ImageMagick puked on it because it wasn't actually a regular file; rather, it was a symlink to that empty file. Apparently ImageMagick doesn't respond kindly to nonsense like that. The solution was just to remove the source file and the offending symlink.

South Korea is reportedly preparing to block TCP port 25 and force all mail traffic through "official mail servers", whatever that means. The BBC article isn't much on technical details. I'm still on the fence about blocking port 25. Fundamentally, I'm opposed to blocking ports outright. If somebody is shown to abuse a service, I'm fine with restrictions. I'm not excited about the prospect of requiring all mail to go through a central server, whether it's mine or someone else's. Consolidation like that tends to make small problems into big ones, such as when one user does slip through and send out spam, now everybody else on that server is put on a blacklist. I can tell you, that's no fun to clean up.

I'm also not convinced that blocking port 25 eliminates spam. It closes down one loop hole, yes, but spammers just need to find a single vulnerability. The common one I see now is to steal mail passwords and use the victim's legitimate mail servers as a relay. Using a good password and not sharing it between sites is good protection against that, but not everyone has gotten that memo. Destination end filtering, as imperfect as it is, will be around to stay for a long time.

Say you wanted to create a RAID1 device but didn't have all your devices ready. Here is how you could create a degraded RAID1 array and then add the second device at a later time. For these examples /dev/sda1 is the first device which will become our RAID and /dev/sdb1 will be added later.

First step is easy, create the RAID array. Our array will be /dev/md0. The "-n 1" option tells it that just a single device will be used and "-f" is required to use such a non-standard option.

$ mdadm --create /dev/md0 -l raid1 -f -n 1 /dev/sda1

You can confirm that your device is working through /proc/mdstat:

$ cat /proc/mdstat
md0 : active (auto-read-only) raid1 sda1[0]
4881700 blocks super 1.2 [1/1] [U]

Now we need to add this device to /etc/mdadm/mdadm.conf, otherwise the system won't see it automatically after the next boot. First we have to find out the UUID that was assigned to the array:

$ mdadm --detail /dev/md0 |grep UUID
UUID : 7fc1afb1:e68bef6b:8972b79f:1bd36cd5

Add a line like this to your mdadm.conf:

ARRAY /dev/md0 metadata=1.2 uuid=7fc1afb1:e68bef6b:8972b79f:1bd36cd5

That's all it takes to create the degraded array. Now comes the fun when you want to expand it. First step is to grow the array to have 2 devices instead of just one with this command:

$ mdadm --grow /dev/md0 -n 2

You can confirm it's there with mdadm --detail:

$ mdadm --detail /dev/md0
/dev/md0:
...
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Number Major Minor RaidDevice State
0 8 19 0 active sync /dev/sda1
1 0 0 1 removed

All that's left is to add the new drive to the array where Linux will automatically put it into place where the missing disk is.

# mdadm --manage /dev/md1 --add /dev/sdb1
mdadm: added /dev/sdb1

# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
29221139 blocks super 1.2 [2/1] [U_]
[>....................] recovery = 1.7% (506496/29221139) finish=8.5min speed=56277K/sec

I couldn't tell you how long these features have been in Asterisk but I decided to give IPv6 and SIP/TCP a try in the latest 1.8 release. Basically it's a simple process. There are no special options required in the build, not that I could find anyway.

Once it's installed, there are two configuration options that need to be set. By default Asterisk will only listen on IPv4 and UDP so add these options in sip.conf:

  tcpenable=yes
  bindaddr=::

Obviously those two can be used independently if you want to try just IPv6 or just TCP, but where's the fun in doing one thing at a time?

So far everything is working just like it used to. I only have one phone that supports IPv6 or TCP, a softphone for Android named Sipdroid. Hopefully more devices will add support in the future.

Looks like World IPv6 Day was a good success. I saw a lot of traffic on NANOG announcing various issues people had seen, and many people declaring success. My own little fiefdom experienced relatively few problems. My Linux laptop had no issues with Google, Facebook and the other sites I could find. I reset my traffic counters this morning so I could watch it:

• IPv4 inbound: 476MB, IPv4 outbound: 77M
• IPv6 inbound: 5320K, IPv4 outbound: 1744K

On the work side of things, unfortunately we only have a few IPv6 test subjects so we weren't able to verify our own network. Given that it works fine in all our other tests I'm sure it's working correctly. What I wonder is if anybody had issues due to broken tunnels; none have been brought to my attention anyway.

If you've ever thought to yourself, "self, I'm going to borrow these public IP addresses which aren't mine and use them for my internal, natted network", well then let me set you straight. Don't do it. Disaster will strike at some point and the consequences will not be good. But if you do it anyway, don't expect me to feel sorry for you either.

Also, NAT sucks.

Despite what some simple Googling might imply, it's entirely possible to use a /32 as a netmask for an IP address on Linux. The important detail is that it can't be the primary IP address. That primary can be an RFC1918 address (i.e. nonrouteable on the Internet) but your default gateway needs to be able to route to you via something other than the /32.

Here's the setup. Assume a Cisco router on 192.0.2.1/24 and a Linux server on 192.0.2.2/24. The /32 we'll use is 192.0.3.1/32.

On the server:

ip addr add dev eth0 192.0.2.2/24
ip addr add dev eth0 192.0.3.1/32

On the router:

interface FastEthernet0/1
ip address 192.0.2.1 255.255.255.0
ip route 192.0.3.1 255.255.255.255 192.0.2.2

At this point, you should be able to ping both 192.0.2.2 and 192.0.3.1 from the router.

The other step you may need to take is to add some routing on the Linux server to source from the 192.0.3.1 address. Normally traffic that is received on that IP will reply with that IP, so that's fine, but for new traffic if you want the source to be the /32, you'll need a special route:

ip route add 10.0.0.0/8 via 192.0.2.1 src 192.0.3.1

See, I told you it was easy.

Like any good tool, Rsync can be easily used to destroy your life. In my case it was a typo on the destination directory and use of the "--delete" option. Just like that, my SD card was emptied. Luckily I didn't have too many files on there, as I only recently acquired it, but there I was thinking "boy I wish I had backups of that". The good news is, I found PhotoRec.

Don't let the name mislead you; PhotoRec is capable of scanning a number of modern filesystems and look for signatures of common file types such as images, music, and documents. It doesn't require an intact filesystem but it certainly helps. I won't go through the ins and outs of the process as the documentation does the job quite nicely.

Photorec can't do everything, of course. It did find what appears to be all of my files, but the names are left up to me. That's a problem I can live with. Having more than once scanned through a dd image of a disk to scrape out data, I have been thoroughly impressed.

The last day of the conference started with a fabulous presentation by Joshua Drake from Command Prompt. From my former days as a Postgres admin, I recall he was quite the presence on the Postgres mailing list so I was quite excited. He went from bottom to top (could have been top to bottom; hard to tell) on how to make Postgres fast. Among the key concepts were that RAID5 is always to be avoided (use RAID10), put your transaction log on a separate spindle (RAID1 please), and take time to learn EXPLAIN ANALYZE. The details take much longer to learn but if I ever get the chance to run a Postgres database again, these will come in very handy.

Miscellaneous Notes:

• hard drives are the slowest
• raid1 great for pg_xlog
• split transaction log
• raid10 for data, separate raid1 for transaction log
• transcation log on ext2 because it *is* a journal
• don't use raid5
• always always always use a battery backup unit (bbu)
• many controllers will disable cache if no bbu
• max out ram. 98% of data sets fit in 4GB of ram.
• sata is great. get a bbu and get twice as many spindles.
• pg is process-based. more cores, better performance.
• autovacuum is not optional. do it.
• default autovacuum works 99% of the time
• shared_buffers - pre-allocated working cache of tuples
• rule of thumb is 20% of available ram
• check kernel.shmmax
• work_mem - amount a sort, etc can use before it spills to disk
• set work_mem to a larger value if you know a query needs it
• explain analyze shows the sort method and the amount of mem
• maintenance_work_mem - maintenance tasks - analyze, vacuum, create index, reindex
• set maint as high as possible. ceiling, not allocation.
• effective_cache_size - very misunderstood. hint.
• % of cached + shared_buffers = effective_cache_size
• generally 40% to 70%
• log_checkpoints - off by default. correlate between checkpoints and spikes in %IOWait from sar
• checkpoint_timeout - force timeout. 15-20 minutes. default 5.
• checkpoint_completion_target - do not change this
• checkpoint_segments - default 3. set to 10. how many transactions logs before a checkpoint is forced.
• use checkpoint_warning to see if you need more
• each segment is 16MB
• recovery means going through each segment
• wal_sync_method - leave default
• synchronous_commit - wait for wal to be written before client completes
• turn off for faster commits. generally turn off.
• default_statistics_target - arbitrary value to determine stats collected by vacuum
• analyze will take longer, but plan is better
• can be set per-column
• know to increase it with explain analyze. if plan == actual, good.
• sqlite - great unless you need concurrent writes
• seq_page_cost - related to rand_page_cost. cost of fetching from disk. with raid10 set the same.
• cpu_operator_cost - default is 0.0025. 0.5 is better
• cpu_tuple_cost
• connection pooling - use pgbouncer
• skype uses postgresql
• plproxy
• prepared queries hang onto plan for the life of the connection. pooled connections can be problems.
• functions are great for ORM
• execution_cost - higher means function costs more
• no function hints to the planner
• 9 has replication. not ready for production data yet.
• londiste - some kind of skype-made replication
• drbd