home | projects | about

Saturday, April 25, 2009

Managing ZFS Snapshots

Recently I've been using ZFS for home backup, for the impressive data integrity and painless snapshot features it has built in. I've put together an OpenSolaris system and migrated my data to it, and came to the point of deciding how to manage snapshots.

Creating a snapshot with zfs is simple, but zfs doesn't help you decide when a given snapshot isn't needed anymore. For this I've created a simple script. Give the script your daily / weekly / monthly / quarterly / yearly snapshot retention policy and your snapshot list, and it will tell you which snapshots from the list you can safely delete.

This makes it simple to create and maintain snapshots with a retention policy from your own scripts. For various reasons, it is difficult to do this with the auto-snapshot service that is part of OpenSolaris.

Read more and download here.

Sunday, January 25, 2009

Installing Debian on a MSI Wind PC

I've been putting together a new backup box using the MSI Wind PC. I ran into a couple of issues that I'll mention here - maybe it will save someone else some time!

My approach was to boot from a USB key, and install the appropriate network install (netinst) .iso. This installs a basic system, and from there you're off to the races. The first set of issues for me was discovering how to create a bootable USB key, and what to put on the key. The second set of issues surround the RealTek 8111C network controller. The default Lenny kernel (2.6.26-1) has a driver r8169 built in, but it just doesn't work reliably, most notably the network device would hang for me when doing large transfers. I ended up going to the Realtek website, and getting the source code for the latest driver, which is r8168 (they clearly rolled back the driver), compiling it and modifying my install to use it. This is very easy to do. I'll list the key points below.

To boot from USB key and install the latest Lenny netinst.iso:

1. Follow these instructions:


But for boot.img.gz, use this image:


and for the netinst.iso, use this image:


2. Mark the USB partition as bootable, if it isn't already. You can do this with /sbin/parted.

3. Install a master boot record:

install-mbr /dev/xxx (where xxx is the usb device)

install-mbr is part of the mbr package on Debian.

Now plug your USB key into your Wind PC, and boot. If the hard drive does not have a bootable image on it yet, the bios will boot from your USB key. If it does, you'll need to go into bios setup and tell it to boot from USB first, before the hard drive.

2.6.26-1 has the r8169 driver built in, so Debian installer will detect your network and get an IP address over DHCP. I was able to go through the install without problems (including installing software from the repository). However if you use it for any length of time, you will encounter this bug where the network device simply stops working. You need the latest version of the driver. Once you are done with your install, follow the directions listed here:


I suggest a small modification to the instructions: build the driver before removing the old one, because you'll find you will need to install components to build correctly, like kernel headers, and possibly make. After building and before installing, remove the current r8169 driver, install the new one, and proceed as usual.

Cheap backup box vs. Drobo

Recently I needed to add another backup destination to my pool. On the search for how to build a cheap Linux box, I found these components and have put one together:

- MSI Wind PC, comes with Intel Atom cpu, for $140 from newegg
- 2gig memory for $27 from newegg
- 1T drive for $120 from newegg

For a total of $287. For comparison, the cheapest standalone, network addressable drobo is $430 (for the drobo, which is a USB device), and $190 for the droboshare (a network device that the drobo plugs in to). That comes to $620. Clearly when it comes to the drobo, you're paying a hefty premium.

Why use a drobo? I have borrowed one that I am experimenting with to understand the tradeoffs. It has hung i/o on my Linux box twice(!), so I am now trying it with a droboshare. It specializes in ease of use and handling hard drive failure gracefully, at the tradeoff of higher cost, less flexibility, and less transparency (it is a magic box and your files are stored in an opaque format, for example).

An advantage of the Linux box over the drobo is flexibility. I can install the necessary software on it to use it remotely in a secure way, for example. Drobo has added "droboapps" but the capability is limited, requires building tools with a special toolchain, and there are hardware limitations, like not a lot of memory to play with. With my Linux box, I have 2gig for $27.

Another advantage of the Linux box is use of a standard file system for storing files (ext3 for example). If I run into trouble with a corrupt disk, it is comforting to know it is in a public format. With drobo, the file system is opaque.

From my point of view, the Linux box with a single drive is also simpler than a drobo. The drobo is emulating a hard drive, with private firmware and 2+ drives. It's a neat idea, but it is more complex, and there are more things to go wrong that are less easy to troubleshoot. A linux box with a single drive is a very standard set up.

The biggest selling point of a drobo (for backup) in my opinion, is the ease of use. No hats with propellers on them required. However if you're evaluating ease of use that highly and you're on a Mac, Time Machine is even easier.

Wednesday, May 14, 2008

fwd: a tool for peer to peer port forwarding

Today I am releasing fwd, a tool for port forwarding between peers, through the firewalls and NATs separating the peers. Fwd achieves this without the need for firewall configuration. The only requirement is that each peer is running an instance of fwd.

To set up a forwarding session, fwd negotiates a peer to peer tunnel between the local and remote peers using a technique called Interactive Connectivity Establishment (ICE). Forwarded data is sent through this tunnel.

Fwd is cross platform and runs on Mac, Linux, and Windows. Read more and download fwd here.

Wednesday, April 23, 2008

Fast 2D graphics w/OpenGL ES

Say you have an existing 2D game with a frame rate that generates new frames totally in game code, and you are porting to the increasing popular OpenGL ES v1.1 api. You can read about OpenGL ES from the public specification here:


Let's assume, hypothetically, that your game already has optimized 2D graphics code, and only needs a way to get a composed frame to the screen quickly (this is the case for Warfare Incorporated). There are two overall approaches:

1. Turn all your graphics objects into OpenGL texture objects, then compose your frame by texture mapping these into a color buffer.

2. Keep all your optimized 2D graphics code. When a frame is ready, turn it into a texture with OpenGL ES, and texture map this into a color buffer.

There are pros and cons to each of these. #1 will mean a lot of code change: if you have optimized 2D graphics code already, it's possible your graphics are stored in a custom format, and that drawing a particular object is done in a custom way. Porting the format, and the drawing for every object over to OpenGL ES api is a lot of work. However, it will likely be the fastest executing if the device your game is running on has a graphics accelerator. It is also likely to take the least amount of battery life.

Approach #2 is the least amount of work: it means you keep all your 2D graphics code and data formats. When a composed frame is ready you "upload" it to OpenGL ES, and have it draw that to the screen. The problem with this is that uploading a texture is often slow. First, regular texture objects aren't designed to dynamically update. When a texture is uploaded it is converted into a gpu specific native format that is optimized for fast texture mapping. This "conversion" is quite slow. Putting this texture conversion into the middle of your game's frame rate won't work.

Thankfully, OpenGL ES has created the concept of a "PBuffer". A PBuffer is both a surface for rendering onto, that can also be used as a pixel buffer for a texture. It is assumed that PBuffers dynamically update, so using a PBuffer as a texture is designed to be inexpensive. They key thing for your 2D game, is that uploading a new texture into your PBuffer is also fairly inexpensive, using glTexSubImage2D().

Once your frame is composed in your PBuffer, you'll want to draw it onto your windowed surface. OpenGL ES v1.1 has a nice "extension api" called glDrawTex(). This is the fastest and easiest way of drawing a textured quad. Use this to draw your PBuffer contents to your window surface. Before you call glDrawTex(), be sure to set the clipping rectangle with glTexParameter / GL_TEXTURE_CROP_RECT_OES. Finally, your platform specific swap buffers api will copy your color buffer to the associated native window. This whole process will give you decent frame rates with minimal change to your existing game. To recap:

1. Use glOrtho to set up a parallel projection, useful for your 2D game.
2. Create a PBuffer surface and associate it with a texture name. Use powers of 2 dimensions.
3. Use glTexSubImage to update only the parts of the PBuffer that are changing (worst case, the whole frame).
4. Once the PBuffer is ready, draw it to your window surface by setting the cropping rect and calling glDrawTex.
5. Swap buffers. Goto 3.

I will post more information about this approach in June.

Tuesday, March 04, 2008

Using Leopard's RSS-fed picture screen saver

I have a lot of pictures - actually it's my wife who takes them all. I mainly ensure they are accessible, safely backed up on and off site, and useful in various ways. They are stored / accessed from a home Linux machine. So when my wife asked to see them on her Mac in a screen saver, I was pleased to discover that Leopard had an RSS based picture screen saver built in. This should be a snap I thought. First thing - our pictures are simply stored on the Linux machine's file system in a large directory tree. I needed a way to 1> serve an RSS feed, and 2> make the pictures in the feed accessible from http. In the process of writing a script to do this I found out a few tidbits about Leopard's RSS screen saver:

- The first thing I did was make a single RSS feed of all the pictures - 60K or so. This just didn't work. (a) For some reason, the screen saver takes forever to validate the large RSS document I enter. It literally hung and after 5 minutes I killed it. (b) every picture in the RSS feed is read right away, rather than reading the picture just before it is displayed. I decided a smaller number of images (< 1000) randomly chosen every 24 hours from the archive, would be a better approach.

- The screen saver ignores the EXIF orientation setting. Modern cameras know the orientation of the photo because it is recorded in the image itself how the camera was being held at the time the picture was taken. I added support to orient the image properly. Doing this required storing the normalized image in a local cache (because I didn't want to modify the original). While doing this, I added support for resizing to a desired size. There is no need to make the image larger than the screen it is displayed on, and resizing makes the file sizes much smaller than the original typically. This is helpful since the screen saver downloads and caches all the items in the RSS feed, so smaller images use less disk space on the Mac.

- Originally I named the pictures in the normalized image cache by index. When the images were changed, the same names were used since it was index based. This caused a problem because the screen saver cached the images by name. So now the images are named based on md5 sum, ensuring uniqueness.

- Finally, I noticed that using an enclosure in the RSS item for the picture caused the RSS screen saver to log a warning message to the console utility about not being able to find the "img node". I changed the script to not use enclosures.

BuildRss is the result, a simple script that works around the Leopard's RSS picture screen saver's various behaviors. I run this script nightly from a cron job to update with a new batch of pictures. You can download the script from here.

Sunday, December 24, 2006

Link-Backup v 0.8

New version of Link-Backup released, with a few enhancements:

v0.8 12/24/2006 scottlu
- allow backup of any file while it is changing
- added --verbose logging to tree building
- minor --verify command fix

Link-Backup can be found here.

Sunday, September 03, 2006

Link-Backup v 0.7

v 0.7 09/02/2006 scottlu
- Ignore pipe, socket, and device file types
- Added --ssh-i to select ssh id file to use (see ssh -i) (Damien Mascord)
- Added --ssh-C to perform ssh compression (see ssh -C) (David Precious)
- Added --ssh-p to specify remote port (see ssh -p) (David Precious)

A friendly user reported a failed backup, and on close inspection it turned out that link-backup was attempting to backup atypical file types such as pipes, sockets, and devices. Now link-backup knows how to ignore those types of files.

Added --ssh-C on request to compress the transfer stream. If you're mainly backing up already compressed files such as pictures and music, --ssh-C is likely not to be a win. For other types of files, it'll save some bandwidth.

Added --ssh-i and ssh-p on request. Good options, thanks for the suggestions and code!

Get Link-Backup here.