edouardp

10:25 pm: Back in Raro
I'm back in Rarotonga on holiday again - since meeting Theresa, I've been here number of times, and I imagine I'll be back here a number of times more in the future. It's truly a wonderful place.

And the accommodation keeps getting better and better too. The overwater bungalows for our honeymoon were stunning, and on a later trip when Kathy and Tony put us up for free at their amazing house in the hills I thought it was about as good as it was going to get, but then, then, this time, we get to stay at the NZ High Commissioners residence! Because, if I hadn't mentioned it before, Theresa's daddy is now the High Commissioner to the Cook Islands.

And there's even broadband on the island now - with enough bandwidth for Skype. Web browsing is a bit sluggish though - I'm not sure, but it feels like a satellite link, as the latency to start a transfer is really long. Ping times are around a second for example. But once you've got a connection up and running, data flows quite quickly (well, quickly for the islands anyway).

Catherine is enjoying it here - she loves the beach, and will spend hours just poodling about in the sand and the sea. So everyone is having a great time!

11:07 pm: Burning DVDs
It's only recently that I've actually had a machine that can write DVDs (my 7 month old iMac).

I tried to write a DVD once a few months ago, but it didn't play in my actual DVD player. I'd just naïvely used Disk Utility to create a disc image that I then burned. I didn't know what had failed, and I didn't really have the time to find work it out at the time.

But I've just got my hand on the contents of Godfrey Reggio's IRE release of Koyaanisqatsi. I already had the MGM release, but I recently found a webpage that showed the difference between the IRE and MGM releases: http://www.dvdbeaver.com/film/DVDCompare2/koyaanisqatsi.htm. When I first got the MGM edition, I remember being slightly disappointed by the quality and softness of the transfer, but back then I had no idea how much better the IRE release was. Or even that the original was actually 4:3 and not widescreen.

And now I wanted to burn it to a DVD, and play it on my DVD player through the projector.

So it was back to working how to make a valid DVD image on my Mac. And, on some level, I objected to paying money for some software to do it when I should be able to work out how to do it for free.

To cut a long and frustrating story short (including several burnt DVDs that would play on the computer, but not on the DVD player, or PS3), I eventually found a post from an Apple guy that states that the built-in image creation/manipulation tool on the Mac, hdiutil, is incapable of creating compliant DVD Video discs (disregard the other guy in the conversation that repeatedly ignores what he is being told).

That made my task easier - as I could ditch plan A for plan B, safe in the knowledge that hdiutil was not capable of producing what I wanted. And Plan B was also satisfying, as it was the old and always reliable tool, mkisofs.

I downloaded the latest cdrecord source, typed "make", and a few minutes later had a static mkisofs binary.

Then, after changing into the directory that had the VIDEO_TS folder (and creating an AUDIO_TS folder, just to be safe) I issued the following command:

mkisofs -dvd-video -udf -o ~/Desktop/dvd.image .

I used Apple's Disk Utility to burn the image to a DVD-R, popped it in my DVD player, and was happily watching the IRE version of one of my favourite films of all time for the next 87 minutes.

Now that I know how, I can duplicate a few DVD's like the scans for the new baby, and a couple of DVD transfers of old family movies Theresa's family has. And hopefully it may help someone reading this blog to know to use mkisofs, and not hdiutil, esp since there are a lot of Google hits out there that tell people how to use hdiutil without any indication that the discs created with it can usually only be played on a computer, and not a real DVD player.

11:20 pm: Google Spreadsheet not quite ready to take over the world yet
So I had used my sophisticated data capture toolset (ahem, tcpflow + vim) to grab all the PS2 compatible games from http://faq.eu.playstation.com/bc/ so I could make a spreadsheet out of them, since the web interface on the official Playstation site sucks really hard.

That was easy. 10 or 15 minutes of clicking next, plus a few simple macros in Vim gave me the CSV file I wanted.

Then I thought "perhaps this might be useful to someone else out there" and thought I should upload the data. And what better way to upload a spreadsheet than to use Google Spreadsheets?

Well it appears that Google Spreadsheets is great for toy data; but throw a three column, two-thousand line dataset at it and you're out of luck.

Firefox has been stuck at "97% done" (done what, no idea) at 100% CPU utilisation on my new iMac for the last hour.

Oh well, live and learn. I guess I'll go low tech and upload it as a text file somewhere instead...

11:52 pm: I knew it was bad, but I didn't know it was this bad
So I like to watch the NBA - in fact that's the main reason I subscribe to Sky; for my twice weekly fix of NBA matches.

I've been getting slowly more dissatisfied with what I'm getting though...

A few years ago I used to get either 2 or 3 games a week - usually one mid-week game, and then, if I was lucky, a double header Friday or Saturday night match. I didn't get a double header every week, but often enough.

These days they are still playing the double headers, but we now only get one of the matches on Sky. As the broadcast is part of the feed from the US, they tell you about the double header, and preview the exciting second match. That I won't be watching here in NZ.

The other thing they tell you, is that the matches are available in High Def. But again, that's for people living in other countries. Not for me. Not in New Zealand. Not on Sky.

(Sigh - the projector I bought four years ago does High def. 720p and 1080i, but I've never had anything High-def to feed into it.)

And, frankly, the NBA "standard def" picture we get here isn't even that. It's noticeably worse than, say, TV1 via Sky.

I can understand why - the signal is originally NTSC, which is only 480 lines, as opposed to PAL which is 576 lines (20% more). And NTSC's signaling means that the picture isn't even as good as a lower resolution PAL would be anyway. And then it's compressed and uploaded to a satellite, and downloaded by Sky TV, transcoded to PAL, re-compressed and uploaded to another satellite, and downloaded to my Sky decoder. The signal is digital, but there's an analogue step at my end to transfer it to my HDD recorder, and, I'm guessing, there are probably analog steps all along the way.

The quality of the picture at the end of this is pretty dire.

But let's quantify how bad the quality is. I ended up forgetting to record the NBA all-star match due to one thing and another, so I decided to download it from the internet via bit torrent. The torrent I downloaded was only 360 lines, but had originally been recorded off a 720p signal in Hong Kong, and then down-sampled and h.264 encoded.

As a comparison I grabbed some frames from a match this week via Sky.

First up, Sky TV:



Second, a Torrent downloaded from the internet:



The second picture is the 360 line h.264 file, while the first is the PAL 576 line picture, but scaled to make everything the same size (so the players are the same height).

How is it that a file downloaded from the internet is better quality than the digital Sky picture I'm paying for? Look at the players with the ball in each. LeBron James in the Bit Torrent, and, well, I can't tell you who in the Sky TV picture. I can't even make out his number. And compare the backboards in both. And, ummm, everything else. Absolutely unbelievable.

Sky says they are planning on moving to High Def. Sometime in 2008. And I'm sure they'll want to charge me several hundreds of dollars for a new decoder at that point as well.

Honestly, if I didn't believe that I have some sort of moral obligation to actually pay for things, I'd just cancel my Sky account and download everything from the internet. I'm pretty sure that's what everyone else does...

05:09 pm: Great Idea
I had a great idea the other day for the name of a PR firm.

"Clientology".

Cool and cultish at the same time. Emphasises the customer is paramount. Ironic.

The only drawback would be the Scientologists might ring up and complain about you using a name too close to theirs.

The defense to that is, of course, that you can use similar, or even identical, names, as long as you are in different industries. Hence Apple Music and Apple Computers (well, until the iTunes store anyway).

So you would just say "Yes, but you are a bunch of dishonest charlatans trying to steal other peoples money, while we're in PR."

...

Ah. OK. Forget I said anything then.

05:17 pm: XML Language
So I ended up forgetting to record the NBA all-star match last week.

I was keen to see it, as it's usually an entertaining couple of hours, so naturally I started downloading a torrent of it. Only after I'd downloaded it, I found that an h.264 encoded
640x360 stream (originally ripped from a HD 720p broadcast) was in fact to complex for my poor old 876MHz Powerbook to play.

I mentioned this ("my computer is too slow to play these files") to my wife, and she said "well you've been saying you need a new hard drive". We had a friend around with her laptop, and I wondered if it was fast enough to play the files, so I asked her how fast her computer was. She said she'd upgraded it from 256M to 512M of memory.

Now I realise that non-computer literate people don't really understand the parts that make a computer go. That fine. I simply mentally say to myself "Oh. OK. They don't have an understanding about that."

But you know what, I get the same feeling whenever I read someone describing XML as a "language". Occasionally, in the right context, like someone explicitly talking about markup languages, it's OK. But mostly it's people talking in the context of programming, and who don't quite get the concept of "data".

At least it's not like when I read the phrase "RESTful Architecture", where I can immediately close the page I am reading, secure in the knowledge that I will be missing nothing of any possible value by not reading further.

10:04 pm: Six word story article on WIRED
New meme: WIRED's "Six Word Story"

10:20 am: Pinball
Thought I'd put a picture up of the new office pinball machine!



Pretty sweet!

02:20 pm: likebetter.com
Huh - I played "likebetter" for twenty minutes, and the brain never turned pink. What's with that then?

11:30 pm: Laptop Stand
I've been wanting to buy a laptop stand for my 12" PowerBook for a little while now.

Serato makes laptop stands but that's not quite what I was after.

I wanted a laptop stand that allowed my PowerBook to stand vertically - raising the screen up at the same time as minimising the footprint on the desk.

Turns out there really are a lot of them out there, but I'd suspected I'd have to order one from Australia at least, and the cost would end up being near $100, which sounded steep, esp. given that I'd be buying sight unseen.

Then last week I visited Miriam at Shift, and everyone there had vertical laptop stands! I asked where they got them from and the answer was "Stephens".

Huh? I thought they sold kitchen stuff, not computer gear...

Turns out the laptop stands were actually cookbook holders! And the cost? $11.95!

Bargain!

Obl-pictures:

10:15 pm: Memory Stick, Memory Lane
So I bought a 2GB memory stick for my PSP the other day. I wanted to be able to put photos, music and movies on my PSP, and 2GB seemed like a good size to do that. It should be able to fit four feature length movies I reckon. At the moment I've dump a bunch of Placebo videos onto it.

Looking on Sony's NZ site, it looks I could have bought one from them for $299, but instead I decided to buy one from a trader on TradeMe. $115. I love TradeMe! Still wouldn't pay 700 million for it, but maybe that's just me.

The guy I bought it off turned out to be someone I'd met 20 or 25 years ago playing D&D - there was a club called "Auckland Dragonslayers" and it met every month to play role-playing games on the top floor of the Customhouse on Customs street.

On a whim I searched for "Auckland Dragonslayers" on Google, even though it pre-dated the web by a decade or more. Amazingly it did appear in an article about Mark One comics, which opened it's first store in the basement of the Customhouse. Wow - that brings up some memories as well!

One New Zealand comic artist mentioned in the article is Dylan Horrocks. Apparently he was also a role-player at Auckland Dragonslayers, although I don't remember him (it was over 20 years ago). He's got a blog over at blogspot. I'll try giving it a read since the only NZ blogs I see linked to are all political, and, with the exception of the ones hosted on Public Address, all truly awful. It's nice to see some NZ blog content that isn't from the usual self-opiniated nut-cases.

And now that I've got the 2GB memory stick, and have dumped a bunch of content onto my PSP, I'm noticing that it's got some sort of "rainbow effect" going on - presumably to do with the LCD having a very slow refresh frequency. Of course my wife couldn't see it at all, and neither will most people, just like with DLP projectors. I've just got some special sensitivity to the effect. I didn't notice it on the Spiderman 2 UMD disc, so perhaps it's isn't as noticeable on normal content like movies (rather than quick-cutting, high-contrast music videos), or maybe Sony do something clever in post-processing the decoded video stream given that they know the characteristics of the display surface. I'll try to put a couple of movies onto the PSP and see how they look...

08:48 pm: Real life
How long is it since I wrote something here? Months at least.

I'm starting to see the point of Tim Bray's extremely short entries - if you're only going to write a few lines then it's actually not that much effort at all to get a post out. I've got several posts that are pages long that I've been "working" on for months, and are not even half-way finished. By the time I do actually manage to complete one, I'm sure that what I have to say will be so out of date as to appear almost comical. Let's hope that Graham Nelson isn't currently putting the finishing touches on Inform 8, for example. And that one about the cult I joined - it's been on the back burner since February.

Real Life has been being good to me - My wonderful daughter is growing quickly, and is now smiling and giggling back at her Daddy. She's also started to pick up things like her rattle that Susan gave her (it came in a package addressed to "Catherine Poor" when she was about 2 days old - that was so neat!). Apparently she's doing everything a 3 month old is supposed to be doing, although Daddy still secretly thinks she's Very Advanced For Her Age...

I'm enjoying reading the various blogs people have - Nigel's last introduced me to Kevin Smith's blog. I've been following Agnieszka's blog - I'm looking forward the second installment of Catholic Marriage School! And Aaron's blog too - and my advice is to get a cheap Microsoft wheel mouse. That's what every other hardcore Mac user I know does...

More soon. I hope.

12:42 am: Baby's First Blog
So I bought "catherinepoor.com" last night (great news - tiffanypoor.com and kublaipoor.com are both still available) and I'd also picked up iLife '06 a month or so ago.

And so now I've got www.catherinepoor.com pointing to a website created with iWeb for Catherine!

iWeb is actually pretty good to be honest - if all you want to do is create a good looking website based on some nice templates Apple supplies, then it's a very easy and attractive way to go.

The website itself is on my .mac account, but I've used Yahoo's "masked domain forwarding" to make it (kindof) appear as if it's coming from catherinepoor.com. Their method uses hidden frames to wrap the target site, which is, well, quite tacky, but I'll leave it as a trial for now.

11:05 pm: Photocasting the baby pics
http://photocast.mac.com/epoor/iPhoto/catherine/index.rss

The Photocast (an RSS feed for photos) of Catherine's pictures. Subscribe, and new ones wil show up automatically (in iPhoto, or other photocast-aware RSS feed reader).

10:21 am: And a couple of photos
Busy busy - I've been up at the hospital almost every hour of the day, and there's no interweb there.

But, as promised, here's two photos of Catherine:

DSC07866.JPG

1 and a half minutes old!

DSC07875.JPG

Being held by Nana later that night.

More to come (perhaps I'll try that new Apple Photocasting thingy)!

11:39 pm: Baby!
Catherine, 2.95kg, born 4:37pm by C-Section. Mother, Daughter and even Daddy doing well. Picture will follow in a day!

12:30 am: Where's my four times faster?
So I was grabbing a dump of our forum database from serato.com/scratchlive.net, and to speed up copying it from the U.S. hosting site to my Mac, I ran bzip2 on the database dump file first.

I could have used gzip (or scp's built in "-C" flag which is also based on the DEFLATE algorithm), but I wanted more compression as a smaller file would make the transfer quicker, as the compression and decompression times at both ends would hopefully be outweighed by the lower transmission time.

Therefore I compressed the files with bzip2, as this is the best compression available as standard on Linux and Mac OS X. It is based on the Burrows-Wheeler transform (BWT), which is one of those amazingly cool algorithms that every programmer should know just for the sake of pure knowledge. Look it up now if you don't know it!

The uncompressed size of the database dump file was 208.4M, and the following table shows the compression sizes, and times achieved by all three standard compression programs (by the way, where on earth did pack(1) go? Or am I having one of those '80s flashback again?)

Method      Size    % Size   Compress  Decompress
-------------------------------------------------
bzip2      34.4M     16.5%   2m 48.9s       26.5s
gzip       45.5M     21.8%      19.5s        2.8s
compress   61.4M     29.4%      16.5s        4.1s

So Bzip2 saves an additional 11M, but takes a lot longer than either Lempel-Ziv based algorithm.

Does the extra (almost) three minutes actually make up for the shorter transmission time? Well 11M in 173 seconds equates to 65Kbytes per second, which frankly is around the speed I get downloading anyway. Actually I usually get a bit more than that, so I think the answer is "No".

Only Bzip2 is a block compression algorithm, or at least the Burrows-Wheeler transform is, and I believe that the move-to-front and entropy encoding also operate on the natural BWT block sizes. The LZ compression family are, in contrast, stream based.

Block algorithms operate on independent chunks of data, whereas stream based algorithms are dependent on previous data (or the most recent previous data, but that data is in turn dependent on it's most recent previous data, and so on back to the beginning of the stream).

Since the chunks of data in a block based algorithm are independent of each other, they can be computed in parallel. This isn't possible on a stream based algorithm as the data at position N relies on all the data from positions 1 to N-1 (well you can restart the stream algorithm periodically to get a block like effect, but this tends to hinder the performance, and, for a compression algorithm, you get worse compression as a result).

And here's the thing that struck me while waiting for bzip2 to finish on my dump file - The machine I was compressing it on (our hosted Linux server) has 4 CPUs. And the machine I was decompressing it on (my Mac) has 4 CPUs.

So why isn't there a parallel version of bzip2 that can use all four CPUs?

Well, OK, the answer to that is because back when compress, gzip, bzip2, and most of the rest of the open source was being written, multiple CPU machines were the exception, and not the rule. Today we are moving (or have already moved) from single CPU machines to dual CPU ones. Tomorrow multiple CPU machines will be the standard, expected configuration of every computer.

Well, OK, so why isn't someone re-writing bzip2 to be parallelisable now?

I really want to know!

Oh - and for those of you writing commercial software - if you aren't looking at your core code to see what needs to be re-written to take advantage of multiple CPUs (and I don't just mean threading - I mean the core, heavy workload code like compression, searching, encryption, etc) then how are you going to feel in a years time when your competitors products suddenly run twice as fast as yours? That's not a situation you want to find yourself in.

11:41 pm: Synergy - Cool Utility!


Well the new job is turning out great, lots (lots!) of really interesting work, but I ended up with three machines on my desk - my dev machine, the new Intel iMac, and a Windows laptop that was exhibiting the USB timing issue we've been looking at for Scratch LIVE 1.5.

I started off trying to have two keyboards and mice on the desk for the Macs, and also use the laptop trackpad and keyboard for that machine. What a pain - you were reaching all over the place to do anything, plus two keyboards and mice take all sorts of room my desk doesn't really have.

I tried running VNC, which definitely was better, but not perfect. Back at the last job we had Black Box KVM switches - but the thing was I had three monitors here. All I really wanted was a keyboard and mouse switch for them.

I remembered, years ago, reading about some utility that allowed you use a keyboard from, say a laptop, and send the keystrokes to another computer, say a rack-mounted server. I'm pretty sure it relied on X-windows, which makes sense as any X client can generate keystrokes across the network. I wondered if something similar to that existed for Mac OS X - this would mean I only need one keyboard for the two Macs.

What I found was even better - synergy2.sourceforge.net

It's a multi-platform keyboard and mouse virtualizer - a single computer can send it's keystrokes to any other computer. Two move focus you simply move the mouse the the edge of one computers screen, and it appears on the next computer across (just like having multiple monitors attached to the the same computer). Copy and paste work as well - cross platform for text - and on multiple Windows machines, you can copy and paste HTML, pictures, etc.

Bestest utility I've found in the past few months!

10:06 pm: An Itch I Can't Reach to Scratch
So I had this file on my desktop - a quicktime movie of a really cool but amazingly low-tech, simple-to-make, grid of oscillating lights someone had done.



And I wanted to know where I'd gotten it from. Which was only a few hours ago, but it was via somewhere like JWZ's Live Journal, or Metafilter, or Slashdot, and I couldn't really remember where.

But I'd like to send the page to a friend in email. Plus I kinda recall it had some other links on it, and seeing as the quicktime was so cool, I'd like to take a look at them too.

If only I could, you know, remember out where it was I saw it...

At this point you are no doubt saying "Dude - why are you whining to me about it? Just go through the pages in your browser's history till you find it. It was only a few hours ago."

Well, true, but in this day and age isn't my computer's job to help me manage my bits? To make my interaction with the Great Information Appliance both rewarding and painless? Isn't that the brave new world we're all striving towards? (And anyway, what if the file wasn't just a few hours old, what if I'd downloaded it "sometime last week"? The job goes from being tedious to not even worth trying.)

Now back in the '90s, when I had my previous Mac, the FTP download tool that everyone used was Anarchy (now Interarchy), and it had the nice feature that it would put a copy of the URL the file came from in the comment field (now renamed "Spotlight Comment" in Tiger) of the downloaded file.

You could go "Get Info" on the file, and it's original location would be sitting there for you. Nice feature.

Why can't Safari do that for you? Like:



Well, it certainly should, no question. But as you can see this doesn't really solve the problem of me finding the page which had the link, that page I chose "Download Linked File" from. Things were simpler in 1995...

The information I need is the page that referred to the URL I downloaded. Or, in this web 2.0 world of a blogs linking to blogs linking to blogs linking to the actual file, the reverse list of pages I followed that ended up me eventually downloading it.

What I really need is for the file and my browser history to be linked somehow, and for that information to be easily accessible.



Ahh, now that's starting to look promising.

And since my train of thought is unconstrained by actually having to implement this, I'd like it to be displayed in an easy to understand, simple to navigate, sexy Apple way. Or, alternatively, like the mediocre mock-up I've done below:


(click for a bigger version, or the full-size version)

Now that's not asking for too much is it?

10:55 am: My MP3s
I've been working on my MP3 collection now for about 7 years now, and am pretty happy - proud, even - of it on a technical level (and I'm pretty happy from a content point of view too, but there's always new music to be discovered!)

I've got a good standard for naming and organising my tracks, all of which live on my Linux server (as my Powerbook's hard drive is actually smaller than my music). iTunes accesses all the music via a SMB share, usually wirelessly, and can stream it to the AirPort Express in the lounge, again wirelessly, without a problem (audio is pretty low bandwidth to be honest).

I have encoded everything with the same encoder, with the same settings, over those seven years. I use Fraunhofer's MP3 encoder at 128kbps, with the quality flag at max (that is, 9).

I find the Fraunhofer encoder to be really really good - in fact when I got my Etymotic ER•4 Earphones I actually heard things in very familiar music that I'd never heard before. The fidelity of the encoded music had exceeded the quality of my audio gear up till that point at least!

(An aside: I remember when the whole "file sharing" thing happened on the internet, and when I listened to peoples collections of downloaded music I was always horrified at how bad almost all of it sounded - I didn't understand how anyone could think that those MP3s were worth listening to. I guess one the reasons was that almost no-one had heard properly encoded MP3s before. I, on the other hand, was one of the lucky ones - I'd had a trial version of the Fraunhofer encoder on the SGI I used at work in 95 or 96. I'd been blown away by the fact that you could get 10:1 compression on audio files, with virtually no loss of quality. Of course this was no use for playback, since no computer back then was fast enough to play an MP3 stream back in realtime! Hooray for Moore's law.)

I guess if I'd never encoded any music up till this point, and had decided to start today, I'd simply encoded it all as uncompressed WAV or AIFF files since it would easily fit on a single hard drive these days. But I don't regret the choice I did make - it has served me very well over the years.

I've spent quite a lot of time downloaded album cover art by hand, mostly from Amazon. I've also spent a fair bit of time fixing the "Song Name", "Artist", and "Album" tags. My current approach is to make sure that stuff is right when I add it to iTunes, as I learnt the hard way that letting that slide, and then having to go back later and check and fix-up 200 albums is no fun.

But the thing that has been dawning on me lately is that this is actually a very long term project (seven years so far, and many more years to come).

And that there is a lot more work to do to get everything in the state I'm completely happy with.

For a start I haven't rated all my music - in fact I've only rated about a third of it. Since what goes on my iPod is determined by the rating being a 4 or 5 (or the album being a recent addition), I know that means I'm missing out on hearing quite a lot of good tracks.

(The rating system I use is as follows:
0 stars - I haven't rated this song yet
1 star - I never want to hear this track ever
2 stars - I don't like this track, and don't want to hear it again
3 stars - this track is neither good or bad, but I don't really want to hear it again
4 stars - I want to hear this track again
5 stars - I want to hear this track again, as soon as it ends)

Jamie Zawinski posted a journal entry recently about having completed rating all his music. It took him about 5 months, so that gives me an idea about what sort of time-frame I should be looking at (taking into account: he doesn't have a wife, a child on the way, or a 9 to 5 job - but I don't have quite as much music as he does.)

Another recent project is download lyrics for the music. I'd actually started this a few years ago by hand, but, unlike the album art, this wasn't even remotely possible to do for all my songs. Luckily, in the last year or so, people have started to write decent programs to automatically scrape the lyrics from the net and put them into the MP3 tags of each track. I haven't run the programs over my entire MP3 collection yet (I currently run the best program in "download when I play" mode), and even once I do decide to apply the program to everything, there will be a lot of tracks for which lyrics cannot be found (a lot of NZ and Australian mainstream music falls into that category for example), and even the lyrics that do automatically get downloaded still need a thorough QA process, as the quality of the lyrics are quite variable, and even the good programs still hit "false positives" and download lyrics to some other song from time to time.

Mentioning MP3 tags leads me to a bigger question: When I started off I made the decision that I'd use version 1.0 ID3 tags since I didn't want to exclude any device from being able to play the songs, and practically everything supported 1.0 tags. Now days 2.X tags are pretty well understood, and I'm tossing up whether or not to tell iTunes to bulk convert everything to 2.X tags instead. I guess I'll have to find out whether all 2.X tags are equally well understood, or whether I should pick a particular (I guess, older) version that most things can read. The lyrics scraping programs have been converting any song that they add lyrics to a 2.X tagged file anyway, so in some ways this decision is turning into a fait accompli...

And, again on tags, the album art I've downloaded actually lives in each album folder as a JPG file, and not, so far, embedded into each MP3 file. Doing so would require 2.X tags, but I'm not actually sure if I should - firstly it will make all my MP3s a few percent larger, and secondly, it's not third normal form dammit!

Plus I'd have to write code to scale the cover images, and add the resulting files to the IDs tags. I guess I'll either use http://id3lib.sourceforge.net/, which is C++, or perhaps one of the Java ones (http://jid3.blinkenlights.org/, http://javamusictag.sourceforge.net/ or http://sourceforge.net/projects/jd3lib/ I guess)

Another open issue is the volume of my music. As you've no doubt noticed, old CDs tend to be much quieter than modern CDs, and some very recent CDs can in turn be louder than the majority of CDs in the last 10 years. Rip Rowen explains it best in the article LOUDER IS BETTER.

The solution (to my problem, not Rips) is to listen to every MP3 in my collection, and then change each one to make them all have the same loudness (by adding metadata). Which I am obviously not going to do by hand, so what tools can I use to do that for me?

iTunes itself has a built-in feature called "Sound Check" that purports to do just that, but, from what I've read, it doesn't perform to expectations.

People recommended iVolume, and I can say it seems to do a pretty good job.

I'm running it set to a perceived output loudness of 90db, with "Allow clipping" off. Currently I've been running it in "Single Song Mode" where each song is independently analysed. You can run it in "Album mode" where the entire album is analysed, and a single loudness adjustment value is calculated and applied to all the songs - thus preserving the artists original intent for relative volume within the album. I'm still tossing up which of those two modes is best. (iVolume keeps a database of the tracks its processed, so changing the mode later doesn't require re-analyses of the tracks).

Which is good, as iVolume's algorithm is really slow. It will take over a week to analyse all my music (on my admittedly aging powerbook).

What else is outstanding? Well most of my MP3s have the wrong year in the ID3 tag, and I'd like to fix that with the date the original track was published (not the album it's on - as compilations and greatest hits albums would then mark the track wrongly to my mind). That may be a "by hand" task.

Also - the genre tags are just worthless noise at present. I'd like to be able to create smart playlists like "Top Rated Rap", but simply cannot at present.

Partly the plethora of micro-genres are useless ("Is that Hard House, Deep House or Progressive House?"). Then there's the definition of the genres (what's the difference between "Rap" and "Hip-hop", what exactly is "Alternative?"). And of course do I set the genres per artist? Per album? Per song?

Perhaps using grouping (aka tags) might help with giving multiple categories to my Japanese Jazz/Drum'n'bass fusion tracks or my UK cockney-hip-hop garage remixes. That's something I need to investigate further.

A blue skies project I'd like to implement is a way of adding cross-referenced notes to my music (think of a wiki that knows specifically about artists, albums and songs). I find I sometimes listen to a song that makes me think of another song, which I then listen to, and so on through multiple songs and artists until I end up at Philip Glass' Einstein on the Beach 20 songs later (well, sometimes anyway). Having done this dozens of times I want to capture the connections between various parts my music.

No wonder I hardly bought any new albums in 2005 - I've already got enough trouble with the ones I do own!