Monday, 29 December 2008

Seaside 2.9: Exception Handling

Ok, I promised well over a month ago to start documenting some of the new bells and whistles in the Seaside 2.9 Alpha series. I thought I'd start things off with a discussion of the new exception handling mechanism.

Have you ever wanted to customize the look of Seaside's error pages? Want to send yourself an email whenever an exception is raised? Maybe save a copy of the image on errors so you can look at the stack later? It has been possible for ages to implement a custom error handling class for your Seaside application but the process was not necessarily obvious and you were limited to catching subclasses of Warning and Error. In the upcoming Seaside release, we have cleaned up the exception handling code to really simplify the error handlers that come with Seaside and to make your custom handlers simpler and more flexible.

Basic Usage

To create your own exception handler, the first question is what class to use as your superclass. If you are just interested in customizing the appearance of error pages or performing some additional action when an error occurs, you probably want to subclass WAErrorHandler. If your requirements are more complex, you may want to subclass WAExceptionHandler directly (see Advanced Usage below).

Subclassing WAErrorHandler

WAErrorHandler is already configured to handle Error and Warning and is a good starting point if you don't need to drastically change the error handling behaviour. In your custom subclass, you can implement #handleError: and #handleWarning: to define the behaviour you want in each case. The methods take the exception (either an Error or Warning respectively) as a parameter and should ultimately either resume the exception or return an instance of WAResponse. By default, both methods call #handleDefault: so you can provide common implementation there.

For example, assuming you had defined a routine to notify yourself of errors by email, you could provide an implementation like this:

handleError: anError
self notifyMeOfError: anError.
^ WAResponse new
contentType: WAMimeType textPlain;
nextPutAll: 'Eek! An error occurred but I have been notified.';

Returning an HTML Response

You could of course provide an HTML response but you would want to make sure your HTML is valid (and that means a head, a title, a body, etc.):

handleError: anError
^ WAResponse new
nextPutAll: '<html>
<body><p>An error occurred</p></body>

If you want to use the Canvas API, you can subclass WAHtmlErrorHandler instead. Just implement #titleForException: and #contentForException:on: (it gets passed the exception and a canvas) and you're done.

Using Your Exception Handler

Once you have written your exception handler, you need to configure your application to use it. Open your web browser and navigate to the Seaside configuration page for your application. There you will find a preference called 'Exception Handler' and you should be able to choose your new class from the list.

Alternatively, from a workspace, execute something like:

(WAAdmin defaultDispatcher entryPointAt: 'myapp')
preferenceAt: #exceptionHandler put: MyHandlerClass

Advanced Usage

If your needs are more complex than the above, you may want to subclass WAExceptionHandler directly. An exception handler needs to do two things:

  1. choose which exceptions to handle
  2. define how to handle them

Selecting Exceptions

The easiest way to select which exceptions to handle is to implement #exceptionSelector on the class-side of your new handler (note: in Seaside 2.9a1 this method is named #exceptionsToCatch). Your #exceptionSelector method should return either an ExceptionSet or a subclass of Exception. You probably want to continue handling the exceptions handled by your superclass so your implementation might look like:

^ super exceptionSelector, MyCustomError

If your needs are somehow more complex, look at implementing #handles: and #, yourself on either the class- or instance-side, as appropriate (note: these two methods do not exist in Seaside 2.9a1).

Handling Exceptions

When an exception is signaled and your handler indicates (via #handles:) that it wishes to handle the exception, #handleException: is called and passed the signaled exception. To define your exception handling behaviour, simply implement #handleException: on the instance-side of your handler. Note that the same handler instance is used throughout a single HTTP request even if multiple exceptions are signaled.

The #handleException: method is expected to return a WAResponse object. It may also choose to resume the exception, cause the Request Context stored in its requestContext instance variable to return a response (by using #redirectTo: for example), or otherwise avoid returning at all; but if it returns, it should return a response.

Internal Errors

WAExceptionHandler also provides a class-side method called #internalError:context:. This method creates a new instance of the handler and calls #internalError:, which should generate a very simple error message. This method is used whenever the exception handling mechanism itself signals an error as well as any other place in the system where we cannot be certain that a more complex error handler will succeed. This method should not do anything that has potential to raise further errors.

Setting Up an Exception Handler

Exception handling for Seaside applications is set up by WAExceptionFilter (more information on filters in a later post) and you do not need to do anything else if you are using WAApplication and WASession. If you are implementing your own request handler, however, and want to use the exception handling mechanism, you will need to set up an exception handler yourself.

Although you could do everything yourself, the easiest thing is to call #handleExceptionsDuring:context: on your handler class, passing it the block you want wrapped in the exception handler and the current WARequestContext object. You can look at RRRssHandler for an example.


The Seaside distribution includes a few exception handlers that you can look at for examples of usage. Beware of WADebugErrorHandler and WAWalkbackErrorHandler, though, as they do crazy things to get debugging to work the way we want in Squeak; although they are well commented and may be interesting to look at, they are probably not good examples of general usage. The Seaside-Squeak-Email package includes WAEmailErrorHandler, an abstract class that can be subclassed and used to send an email whenever an error occurs.

Monday, 15 December 2008

Christmas giving

I just stumbled across this site:

As it happens, I've been reducing my Christmas gift giving over the past few years anyway, agreeing with more and more people just to forgo the exercise. I love giving gifts (really) but have been finding searching them out, selecting them, and buying them increasingly stressful. At the same time, when people ask me what I want or need, I have a hard time coming up with anything much and the same seems to hold true for everyone else. In order to perpetuate mass consumerism, this situation seems to have brought about the proliferation of gift cards. I don't have a problem with gift cards per se (they've been around in the form of gift certificates for ever), but the fact that I can now buy gift cards for a hundred different stores at the Safeway checkout counter makes me feel like things have gone too far.

Sure, I remember many childhood years, waking up with eager anticipation of gifts to come–at least one of them usually big and/or expensive. There were even a couple of years there, not too long ago, when Christmas seemed to have lost its luster and I just couldn't eke much enjoyment out of it.

But things have mellowed again and Christmas has been reborn. Now it's about cooking "Christmas" dinner (weeks in advance to fit everyone's schedule) with friends. It's about brisk walks down brightly-decorated streets and the smell of a wood fire (where still permitted by law). It's about spending time with family. And sure, sometimes it's still about giving gifts but the joy is back (on both sides) because it's now about an occasional gift that you didn't even expect. One of the most meaningful gifts I received was an album of family recipes and baby photos. In fact, I think that's when I realized that my perception of Christmas had changed.

I don't know if it's just the result of aging or if society is actually start to push back against consumerism-for-its-own-sake but it feels good. And if you still feel you need to get me a gift, try something consumable: wine, herbs, spices, oils. It's so nice to get a bottle of wine I've never tried and would never have bought. And if I don't like it, at least it's gone instead of staring at me for the rest of time from a dusty shelf or the back of a closet.

Thursday, 11 December 2008

The will of the people

Steven had a post about the recent Canadian political events wherein he concludes:
So best I can tell, when the Governor General does allow the opposition to take power without an election there is noise, controversy and eventually change so it cannot happen again. And when the Governor General does not give power to the opposition then history seems to just continue along. I guess I must have missed McArthur's convention.
Perhaps this is nitpicking, but we're not talking about the Governor General giving power to the opposition; we're talking about the Governor General giving power to a new coalition, one with the support of the majority of parliament.

Now I agree that the Governor General has this option but certainly no obligation to choose it. On the other hand, I still can't fathom why we would want another election right now: Canadians elected 308 MPs to represent them and if the majority of them want a change in parliament then that is the will of the people, particularly this soon after an election. The incessant cries of Canadians who seem to think they live in an the US and therefore elected the Prime Minister are driving me nuts.

I also disagree that the Governor General has lost her authority to act against the will of the Prime Minister. By convention, she normally follows the advice of the Prime Minister, but, at least as far as I'm concerned, this is only in his role as the head of parliament. The Prime Minister in Canada is not a political office; he does not have any special privileges and should not; the Governor General is bowing to the will of the people (as represented by their elected MPs in parliament), not to the Prime Minister.

The Governor General has a tenuous position in that being seen to oppose the will of the people would likely result in a swift call to remove what has become a largely ceremonial position. However, I also think we have a history of intelligent, reasonable, and competent Governor Generals and I'm glad the position is still not technically powerless. In this case, I think calling a timeout is a reasonable decision, allowing both sides a chance to back down. If the non-confidence vote goes ahead in the new year, though, I'd hope she won't waste our time and money with another election.

Tuesday, 28 October 2008

One, Two, cha-cha-cha

We just got back from our first dance class: Cha-Cha and Waltz. It was lots of fun, ours shoes don't look too stepped on, and we definitely weren't the worst couple in the class. What more could you ask for?

I've been wanting to take ballroom lessons for years (at least since my first friend got married). We just need to practice enough to let muscle memory kick in; I think there's some worry from my better half that I'll be scheduling nightly drills so I'll have to control myself.

Sunday, 26 October 2008

Notebook Shopping

So I'm in the market for a new laptop. This kind of annoys me because I don't actually want a laptop. Expensive, barely upgradeable, tiny keyboards, and (worst of the all) the temptation to sit awkwardly in front of the television with the thing precariously balanced on your knee, cooking and radiating your organs and destroying your body through poor ergonomics. But I'm traveling a fair bit, doing consulting, and there are occasions where a laptop is, if not essential, at least very very useful.

So here I am looking around at the options and, while I've been telling friends and colleagues for years to buy Mac laptops if they can afford them, I now find myself unsure. I was intending to get a MacBook Pro in September in order to get the free iPod Touch but decided to wait for the new Macs to be announced. I was underwhelmed: there's really not much new there despite the heavy sales pitch and I don't like the new look. Even with the education discount, the cheapest MacBook Pro would cost me over $2400 after tax.

The problem is that I really like Mac OS X and I really like that the freeware people write for it actually looks good (unlike for Linux or Windows, at least in general). I discovered over the last 9 months that, for the things I do, I can manage fine with Windows. And as a major plus, I don't spend hours endlessly "tweaking" my system like I do with Linux. Sure, every time Apple releases a software update, Mac owners everywhere cower in fear wondering what will be broken, but I'm still happier in OS X than in any other operating system.

I'm looking around at Dells and Compaqs and I can get a laptop that would probably serve my needs just fine for $700 or so plus tax. That makes the MacBook Pro something like three times as expensive! If Apple would sell me a copy of OS X to run on a PC, I would buy it in a flash but $1500 is a hefty tax to pay for it. Even refurbished old-model MacBook Pros are selling for $1449 plus tax but that's still double.

Now I just read Jeff Atwood's post about netbooks. Netbooks are an emerging category of small ultra-portable computers basically designed for running a web browser, checking email, and maybe doing some word processing. Their specs seem to be in the range of 1.5GHz processor, 1GB RAM, 160GB hard drive, and a 7"-9" screen. Aside from the screen, that's better than the computer I'm writing this post on. And the price? Under $500. Jeff says:
They still have a way to go, of course, but the $299 or $199 no-compromises, go-anywhere, zero-monthly-contract-fees web browser in the palm of your hand -- with the requisite 9" or larger screen -- is almost upon us. I guess I hadn't been paying enough attention, because that's a shocker to me.
And that's exactly how I feel. I've heard people talking about Netbooks for a while but hadn't really realized how far they'd come. I'm still considering this new laptop my primary machine so I can't handle the small screen but if I had a desktop computer I would seriously consider getting one of these (with an external keyboard) to cover those "useful-to-have-a-laptop" situations.

The social pressure to display an Apple logo is intense (do you have any idea what percentage of people at computer conferences have Macs?) and I would probably spend $200 or $300 more for the privelege. If I can find something with a comparably beautiful screen, though, I think I may have to make do without the metallic finish.

Monday, 13 October 2008


After reading this article, I ordered a copy of George Lakoff's Don't Think of an Elephant. The book examines the use of language to "frame" arguments to your advantage. Take, for example, the phrase "tax relief". If there is relief to be had, tax is obviously a burden. How can anybody expect to succeed in arguing against something called "tax relief"? If you enter into the debate without reframing the issue, you lose.

Lakoff draws his examples largely from the context of the US political system and he makes no effort to conceal his Democratic leanings. Certainly not unbiased, the book is in fact a call to arms for Democrats, who Lakoff claims are decades behind Republicans in their understanding of these issues. All this means I take his words with a grain of salt.

Nonetheless, the discussion of mental framing and its effects fits with my observations of the world and seems broadly relevant. His analysis of the progressive and conservative movements themselves and their origins was also intriguing: I have no idea how accurate these theories are but I was able to look at conservative policy in a new light.

I was annoyed throughout by his repetition, both within and between chapters. In a few cases, I discovered several-paragraph-long sections that were taken almost verbatim from an earlier chapter. I discovered part way through that the book is a collection of essays and I have the impression they were thrown together quickly to get the book published in time for the last US election. This is partly to blame for the repetition but I think Lakoff is also overapplying one of his own messages: that mental frames are adopted through repetition.

That said, the book is short (100-and-some pages), cheap ($8 on Amazon), interesting, and a quick read so I recommend picking up a copy even if you only make it through the first two chapters.

Don't toss that margarine!

I grew up with butter. As a result, margarine just always seemed weird, even before the trans-fat scare (which, apparently, is no longer an issue with margarine).

Well, as of a couple of weeks ago, I have a reason to keep margarine in my fridge again: it takes tree sap out of clothing! Really. Rub margarine thoroughly into the clothing and use dish detergent to wash it out (no need to let it sit). Then run the clothes through the washing machine and you should have nothing but a pine-fresh scent remaining. It worked for me.

Friday, 10 October 2008

Graph Theory

I'm not sure what happened. I don't remember finding graph theory particularly interesting in university but over the past month I've had two occasions of simple pleasure while figuring out (pretty basic) directed graph algorithms.

The first was at our last Seaside sprint, where Lukas and I were working out how to modify his Package Dependency tool to mark cycles on Seaside 2.9's dependency graph in red, so we could immediately spot problems. It's not like there aren't already algorithms for this but it was fun just trying to work one out for ourselves.

The other was this Tuesday. I woke up wondering whether I couldn't make the graph a little easier to look at by removing direct edges when we have another indirect path to the same node. In other words, if package A depends on B and package B depends on C, we could omit a dependency from package A to package C because that dependency is already implied through transitivity anyway. I pictured this as the opposite of a Transitive Closure.

It turns out that this operation is called a Transitive Reduction and there are algorithms for it but they don't seem to be very well documented online. I ended up just working something out myself using depth-first searching and path-length tracking to prune shorter paths to objects. I don't know if there's a faster way but it doesn't really need to be that fast in this context anyway. I currently abort if the graph is cyclic but I'm pretty sure you could do something like pick an arbitrary root for the cycle, treat everything within the cycle as being distance zero from each other, and you would get a reasonable (albeit non-deterministic) answer. In our case, we don't want cycles in our graph so, realistically, you would fix the cycle and rerun the dependency checker anyway. Here's the result.

This got me thinking... is there a good implementation of graph theory algorithms in Smalltalk? I would love to be able to create a Graph of my model objects (maybe implement #edges on each one or something) and then call: "graph isCyclic", "graph transitiveReduction", or "graph topologicalSortFrom: aNode". So many things can map to directed graphs and, if you could work out domain-specific solutions by really easily layering generic graphing algorithms, that would be really cool.

Sometimes solving simple problems with definable right answers is so satisfying.

Gartner on Seaside

Well... praise for Seaside from a Gartner analyst:
If you are BIG fan of dynamics languages (closures, meta programming, and all that cool stuff) then consider giving Smalltalk a look. You might like what you see. Its like Ruby but with bigger muscles. You think Rails is cool? Check out seaside.
His general comments about Smalltalk are in line with Gartner's changing position but the specific reference to Seaside adds another touch of legitimacy when selling a Seaside project to those who hold the purse strings. Very nice.

Saturday, 4 October 2008

Vancouver Olympic Tickets on Sale

For those who haven't heard, tickets for Vancouver 2010 are on sale to Canadian residents.

For Round 1, you have until November 7 to put in a request for tickets. After that, events with too much demand will have a draw to determine who gets tickets.

Saturday, 20 September 2008

Seaside History

There are a lot of questions around the origins and evolution of Seaside, particularly after Avi and I gave up our old domain and the old Seaside and Seaside 2 websites with it.

A couple of months ago, I began (but never finished) a history page for the Seaside website to provide some background information for those who are interested. I had to dust off some of those notes to prepare my ESUG presentation on the past and future evolution of the framework and figured I might as well dust off the history page as well. So here's the story as best as I can recall it (and by "recall", I mean "find in Google" because Google seems to hold the majority of my memories these days).

[Update: now posted at]


Seaside made its public debut (version 0.9) in an announcement to the squeak-dev list on February 21, 2002. Avi Bryant and I developed Seaside to support our web application development consulting, specifically the development of a web-based theatre boxoffice sales system.

Seaside took heavy inspiration from Avi's Iowa framework (now here), which was written in Ruby and was itself inspired by NeXT's (and then Apple's) WebObjects. This first release of Seaside provided action callbacks for links and forms, session state management with support for call/return and the back button, and a component system with templates.


Almost immediately after the release of 0.9, we began work on Seaside 2.x (codenamed Borges, a reference to Jorge Luis Borges' short story The Garden of Forking Paths and an allusion to Seaside's support for forking session states). Seaside 2.0 was essentially a complete rewrite with a layered architecture: a Kernel layer providing a continuation-based HTTP request/response response loop and state (back-)tracking; a Views layer providing action callbacks and a rendering API for generating HTML; and a Component layer providing call/return semantics, embedding, and development tools.

Seaside 2.0 was released in October, 2003 with the templating system conspicuously absent. This was an experiment to see whether the development of the HTML rendering API and the wider acceptance of CSS had reduced or eliminated the need for templates. The new layered architecture made it easy for others to experiment with developing their own template engines. Seaside was also ported by Eric Hodel to Ruby, where it kept the name Borges.

Several versions followed in quick succession with major refactorings to the session state tracking and backtracking mechanisms. Seaside 2.3 (mid-2003) also introduced an even more layered architecture that tried to make some of the internals clearer and more accessibly to the project's growing number of users and contributors. It also confirmed that Seaside would not have built-in templates in the near future. Seaside became increasingly well-known around this time with a presentation at ESUG 2002 by Lukas Renggli and Adrian Lienhard and a hands-on development workshop at Smalltalk Solutions in 2003 by myself and Avi.


Seaside 2.4 and 2.5 addressed some growing pains in some of the core parts of the system: the Renderer API, collapsing under the weight of combinatorial explosion, was replaced by the now-familiar Canvas API; and some of the internal workings of the Session object were reified to make its application main-loop metaphor more obvious. Version 2.5 also saw the introduction of Component Decorations, Halos, and response streaming.

As first I and then Avi began to work full time developing applications using Seaside, the community began to carry more of the development load, with the release of Seaside 2.7 being  entirely (and very successfully) managed by the community, with Lukas, Philippe Marschall, and Michel Bany leading the effort. This release focused heavily on cleaning up the code base by fixing, deprecating, refactoring, and removing code.

Tuesday, 16 September 2008

Teaching a nation how to wave (part 2)

Ok, so it's been rather a long time since I ended Teaching a nation how to wave (part 1) with a "to be continued..." China already feels so far away that I barely remember where I wanted to go with that series but here we go anyway...

The Beijing Olympic Games were not the party I was hoping for. This is not to say it wasn't interesting (it was), nor that it was the fault of the security measures (it wasn't) or the Chinese organizers (not sure). For all I know, the Olympics are never as much of a party as you would expect. When Vancouverites put down our own Olympic bid as a waste of money, though, I countered that it was like throwing a house party: of course you'd rather go to somebody else's house party and avoid the costs and cleanup but eventually it comes your turn to step up and host one of your own.

And yet, while there were more people on the subways, more accreditation-pass-sporting foreigners on the bar streets, and Olympic sponsor booths scattered here and there, on the whole, life outside the sports venues seemed to be largely business as usual. The athletes (and those who could afford to drop $400 on a one-night admission) could seek out one of the many national houses or embassy-sponsored functions. But the rest of us were left to the usual collection of bars and restaurants, now lined with flat-screen televisions and sporting 15% surcharges to cover "the increased costs of food and labour" during the Olympics. I can't help thinking that if each country opened their national houses and threw a big party (much as the Dutch Heineken house did nightly) even just once during the event the atmosphere might have been a lot more festive.

That said, the atmosphere at the sporting venues was often electric. Because of the large number of individual competitions combined into a single ticketed session, many people either arrive late or leave early rather than sitting on hard bleachers for 6 or even 8 hours (way, way, way too much tennis for one go). But when the Chinese athletes were competing you could count on a pretty full house. I imagine that for many of the spectators, attending a major sporting event would have been a novel experience and the Beijing Organizing Committee had been circulating instructions on how to perform various "suggested" cheers. There were also cheering squads with bright yellow shirts scattered throughout the stadiums to provide guidance. The main cheer, quickly adopted (and adapted) by foreigners from all countries was a rhythmical four-beat chanting of zhongguo jiayou!, which means, basically, "Go China!".

At one particular basketball session, the stadium quite full of Chinese fans awaiting an upcoming game, a rowdy group of Russians behind us was trying to initiate a Mexican Wave (first time I've heard it called that) in support of their team. A few tentative participants at first. Then a few more. Maybe a section now. A few sections. Finally, after 8 or 9 attempts, the first wave trickles around the stadium, picks up a few more people, builds a little momentum and completes several more rotations before petering out. The slightly surprised but enthusiastic looks on the faces of people around me are contagious...

Several more attempts were made with limited success. These attempts are (and I have never seen anything quite like it) best described as "square waves". Each section seemed to stand up en masse, cheer, and sit down. Only then would next section do the same. The result is a sort of pulsing roar that is really quite off-putting. By the end of the Olympics, however, the stadiums full of fans were waving, clapping, and stomping their feet to "We Will Rock You" like they had been attending NHL hockey games since before they could walk. And when the Wave got started, not only could you see and hear it, but you could feel its energy passing over you: the roar would come barreling towards you and almost literally pick you up out of your seat. Teaching a nation of 1.3 billion people how to wave? I'd call that mission accomplished for the Olympics.

Thursday, 28 August 2008

ESUG 2008 Presentation

I'm in Amsterdam and just finished my ESUG 2008 presentation on the evolution of Seaside this morning. The gist of the talk was that Seaside has evolved through Experimentation, Stabilization, and Optimization and is enjoying a bit of an Adoption phase which should be encouraged. Sort of a call to arms.

I also took the opportunity to try to communicate the architecture and key metaphor of Seaside through illustrations. Finally, I rounded it all out with a few examples of less obvious (read badly documented) places people might consider extending the framework.

I think it went pretty well overall. I had a few people come up afterwards saying either that it had helped them understand some part of Seaside, that the historical perspective was interesting, or that they agreed with the need for documentation and advocacy.

I imagine James Robertson will post video on his blog at some point as he was filming all the talks. In the meantime he's posted a quick summary (wrong title and name spelled wrong, though! :) ). I'm guessing the slides will be available on the ESUG website sometime soon but I'll post them if not.

Update: Slides are available here.

Sunday, 10 August 2008

Teaching a nation how to wave (part 1)

Friday was a national holiday, declared only days in advance in recognition of the Olympic opening ceremonies. My poorly judged trip to Tiananmen Square that day revealed more people than I have ever seen in one place in China. Flags, stickers, and headbands were being sold, purchased, and worn everywhere I looked. The Olympic excitement seemed to have finally hit.

Trying to get home, I eventually give up on taking the subway after finding stations closed and streets blocked off at every turn. With the heat finally getting to me, I fall into a taxi (the best 30 quai I ever spent!) and head for home. All of a sudden, the driver is trying to talk to me. We manage to communicate that I come from Canada and he from Beijing. He pats my leg and says jianada a few times with a big grin on his face. I try out jianada yo shan yo shui (Canada is beautiful, literally: there are mountains and there is water), messing up the pronunciation but earning a smiling correction from him instead of a disinterested grunt. Finally, he rests his hand on my arm, rubbing my arm hair between his index finger and thumb. As I'm beginning to find this creepy, he starts miming shaving his face. Now I get the joke! I mime back that it may be stinking hot in China but in Canada I need this fuzzy layer.

Heading to a bar to watch the opening ceremonies, I carefully write out the chinese address, copying it character by character from a book. I hop in a taxi and the driver studiously examines my scrawled lines, picking out the characters he recognizes and piecing it together. Finally, he gets it. He turns to me and mimes, "Did you write this?" Yes, I laugh. Hen hao, he smiles (very good) and gives me a thumbs up.

It seems the city is suddenly filled with high spirits and friendly good will. The drive to the bar is the quickest I've ever experienced, despite two of the four lanes being set aside for Olympic vehicles; apparently residents were asked to keep off the roads. The street is lined with police officers and vehicles and, at every intersection, bus stop, on-ramp, off-ramp, and pedestrian overpass, a soldier in dress uniform stands at careful attention. Beijing is about to throw a party and everything seems to be ready.

(to be continued)

Friday, 1 August 2008

Random Thoughts

I'm not quite sure what to write today so it will be a bit of a stream of consciousness.

I saw a man on a subway platform today repeatedly slapping a woman (presumably wife or girlfriend) in the face. It was truly horrifying. I thought we were all supposed to be desensitized by violence in movies, games, and the news but the sheer brutality of it was absolutely shocking. I mean this is something I have literally never seen before in my life. Hours later, it still makes my blood flash boil thinking about it. All I could think to do was to start screaming at this guy in a language he didn't understand, indicating by gestures that he could try it on someone else, but the subway doors had closed and all I could do was grimace and talk myself out of taking the next subway back. Apparently domestic violence is a big problem in China.

Only three weeks to go before my departure from China and (much like my last job) I don't really want to stay but the departure is still sad. I won't miss the slow, spotty Internet. I won't miss ordering bottled water. I definitely won't miss the pollution or the heat. I won't miss the one channel of English propaganda on television, or the staring, or the spitting, or the manually controlled traffic lights that don't give pedestrians enough time to get across the road, even if you get a head start before they go green.

What will I miss? I don't exactly know. I'll miss the cheap taxis and cheap beer. I'll miss the variety of food choices (at least compared to where I'm going next!). I'll regret not having got the most out of the country (no travel!) and not having learned enough of the language to even have a basic conversation with the taxi drivers. I'll miss the way something noticeable has changed every single time I go outside. But these all seem insignificant and I think there's something more to it. Or could it be just sentimentality because it was "home" for a while? It may take some time for me to figure this one out... and I think I may be back sometime to see a bit more of the country (maybe they'll have fixed the traffic lights!).

One final observation: I seem to be getting restless. It's been 5 or 6 months since I went on leave and this appears to be a pattern. When I returned from Europe in July of 2002, it was just after Christmas that I started working at UBC. When I returned from Australia, it was about 5 months before I started working at Emily Carr. And I became restless in Australia after about five and a half months of being there. It seems that's about as long as I can stand being unfocused (at least in one place). Julia commented that for her it would be more like a few hours but it's good to know what my limit is.

Friday, 18 July 2008

It's raining, it's pouring...

... the Old man is... yeah well it's really raining here again this morning. This is serious heavy rain (and I come from Vancouver, so I know whereof I speak). And the thunder and lightning is setting off car alarms again. It's really cool to live somewhere where storms are accompanied by huge rolling thunder crashes and the tremendous, powerful cracks of lightning bolts.

With the exception of the tropics, I have never been anywhere where it rains so much during the Summer. For a month or more now, we have been saying we should follow everybody else and buy umbrellas to hide away in a pocket or a bag. But it always feels like we're about to leave and the rain has to stop... sometime. It doesn't seem like it's going to.

On the bright side, it does seem to help keep the ground and the air clean. I wonder how much of it is caused by meteorologists seeding the clouds (they also apparently have technology in place to reduce the chance of rain for the opening ceremonies if necessary). The western media is acting like they've only just been made aware of the air quality problem in Beijing and its potential impact on the Olympic games. All construction in the city is slated to stop this Sunday so we'll see if that makes any difference. Sadly, as a result, streams of migrant workers are packing their few possessions and returning to their home provinces for the duration of the games. I say "as a result" but of course it is equally as a result of the desire to maintain "security" and a "clean" city for the games.

On a related note, I heard a news story on the World at Six on CBC saying that spectators would be searched for banners, flags, and noisemakers and that they would be confiscated. "No flags?", thought I, "That's kind of lame." After allowing this misconception to carry on for the length of the story, they added the extra detail that only flags smaller than 1m by 2m would be allowed. Who the heck needs a flag bigger than that?! Imagine you are sitting behind the guy with a flag bigger than that! Sheesh... Apparently umbrellas will be allowed, which seems like it may be a good thing.

It feels like the anticipation in the city has, if anything, declined over the past few weeks. I really felt like it has been growing since I arrived (maybe it was just my anticipation) but, as the last details get sorted out, I just don't feel any excitement. Oh, sure, there are a few more banners, the signs for the Olympic traffic lanes are going up, and workers are setting up venues around the city with big televisions for people to watch the events. People even lined up for hours at the bank to receive newly-printed 10 RMB notes with the Olympic stadium on them. But I just don't sense the excitement. I'm guessing it will turn on by decree.

Monday, 7 July 2008

The Rain!

It's 9:00am, Julia left for work 2 hours ago, and I'm listening to the morale-building going on outside one of the real-estate offices across the street. The workers in the office building kitty-corner to us often file out and do calisthenics in the parking lot early in the morning. But every morning these real-estate agents line up in two rows (one of men, one of women) on the sidewalk and do cheerleading routines involving chants, clapping, singing, and hand gestures, often to music.

The heat and humidity are already up; the former due mostly to two rare days of clear skies over the weekend. I no longer trust my key-chain thermometer as I have long suspected that it always reads between 27 and 29 degrees. I attribute this defect to the photograph I took of it maxed out at 50 degrees in northern Australia nearly 3 years ago. Google, however, says it is 27 degrees and 74% humidity right now... not bad for nine in the morning. It's amazing to me that a place can be this humid in the summer (wait for August!) and so dry in winter. If the heat doesn't dehydrate you here the air conditioning will and I wake up every morning with a dry throat.

The clear skies this weekend probably came thanks to the rain on Friday (rain seems to settle the fine particulate pollution). Julia and I met for dinner after work on Friday and then decided to go to a movie. As we emerged from the restaurant, it began to rain lightly. As we emerged from the taxi and sprinted to the cinema, it began to rain heavily. Forty minutes later, as our film was about to start, they informed us that Cinema 6 was being shut down because of water flooding into it and that we could get our money back. Upstairs, parts of the street were now flowing like rivers as the rain alternately poured and paused in 5-minute cycles. After half an hour trying to find a taxi that was (a) empty, (b) willing to pick us up (not sure why this is a problem), and (c) closer to us than to somebody else looking for the same thing, we gave up and walked to the subway. Unfortunately the subway was closed, due (as far as we could tell) to flooding. Tiring of waiting, we crossed the street and attempted to decipher the bus system. We did eventually manage to get close enough to home that we could walk the rest of the way, having become wet enough by then that we'd given up caring about staying dry anyway. In total, it took us two full hours to get home... this city doesn't deal well with heavy rain.

Friday, 4 July 2008

I bowled a 162!

For as long as I've known Julia I've been trying to get her to go bowling with me. It's not like I bowl often but it's fun occasionally. Well we finally had the opportunity last weekend at a social event organized by Julia's company and while I couldn't knock down more than 9 pins to save my life in the first game, I managed nearly all strikes and spares in the second as I found my stride. People said I had "good form" which I guess I can only attribute to a few years of regular curling; I did feel pretty consistent.

Sunday was a busy day for us. Julia had uniform measurements and team building with her hospitality staff, then we had lunch and bowling, and then we had a few hours to kill before she and I went to a noodle and dumpling cooking class. Unfortunately, we really only had a demonstration and a little practice of the noodle pulling so I still can't claim to have successfully mastered the art but at least I have an idea how the dough should feel. The dumplings, or jiaozi, were quite successful: pork-and-celery and egg-mushroom-and-bok-choi. And I had left overs for lunch!

In other news, I'm currently struggling under the weight of decisions regarding employment, flights out of here, and their destinations so more to come once those are settled. I won't go into all the details of the complications of obtaining a Russian visa but, after a frustrating experience with the Russian embassy in Kuala Lumpur coloured by view of Russian hospitality, I have essentially decided to put off my trans-siberian trip for another occasion.

We're nearly within a month of the Olypmics here now so the pace is accelerating. Julia is busy at work, we have our spare room rented out, and I have an accreditation that removes the need for me to leave the country one last time. The new subways are nearly open and construction sites are racing to get their landscaping done and their walls down before the construction ban kicks in a few weeks from now. Should be interesting times ahead...

Friday, 27 June 2008

RSS Feed Change

Just a quick note to anyone (Steve) who may be reading the feed for this blog: I've decided to start using FeedBurner, mostly just to provide email subscription for people who don't know what RSS means but also because I'm curious. I've been told that many news aggregators don't follow redirects, so you may need to update your feeds to the new URL:

And for anyone who would rather receive an email when there are new posts, use this link:

Friday, 20 June 2008

Bash: using the read command

While I normally use awk or sed to pull a piece of information I want out of a line of text, it doesn't work as well when you need multiple pieces of information. Well it works fine for displaying multiple pieces of information, but if you want them in variables to use later?

I once wrote a script that used declare to do the equivalent of this:
declare `w | head -1 | 
awk '{ print "DAYS=" $3 " " "USERS=" $6 }'`
# now do something with $DAYS and $USERS

Ewww... Now I know there's a better way: the "read" command. You could do something like this:
w | head -1 |
(read -a FIELDS
#do something with ${FIELDS[2]} and ${FIELDS[5]})

You can also do like so:
ls -l | tail -1 |
(read perms dunno user group \
size month date year file
echo "$user: $file")

You can even loop over all the lines of input. Let's take the /etc/passwd file as an example. Since that file uses a colon as a separator, we first need to set the IFS variable which determines what characters are used as field separators.
cat /etc/passwd | (IFS=:
while read user pass uid gid desc theRest; do
echo "$user: $desc"

As pointed out here, this is also one way to deal with a file containing a list of filenames that may have spaces in them:
cat list | while read f; do
ls -l "$f"

Monday, 16 June 2008

Teenage Voting

I was reading an article on about the education system in the US and there was a quote that read:
Our educational system is the only major institution in our country that officially recognizes autocracy (in contrast to democracy) as its principle type of government.
I don't think that's correct. Rather, the education system is one of the few major institutions to deal essentially entirely with children; and children are dealt with pretty much universally as if they were living in an autocracy. That got me thinking about engaging teenagers in democracy. I wondered if there mightn't be some forward-looking European nation that included the under-18's somehow in the process.

The only case I can find is a BBC article stating the the Isle of Man has lowered their voting age to 16. This seems a reasonable enough action to me, despite the call by others (one example from a Maclean's article parodied here) that the age should be raised to 21 because "young people today have essentially tuned out". Yeah, preventing them from voting will really help them tune back in.

New York City apparently proposed a bill to lower the voting age to 16 and so have California and a number of other states, but as far as I can find, none have yet passed. The arguments in favour include: getting teenagers used to voting; that they pay sales and, often, income tax without any control over where the money goes; that they can be tried for crimes as an adult after 16 but can't affect the laws; that 18 is a bad age to start voting because students are heading off to college and have too much other stuff on their plates; and (most important to me) that they have more invested in the future than anyone else. I'm not even that old and I already have a slight trepidation at trusting "kids today" to cast a vote, but I'm still young enough to remember that I had perfectly valid and rational opinions in highschool.

I don't even know if you need to lower the voting age. The Californian bill initally proposed that "each 16- or 17-year-old's vote count as half an adult vote and each 14- and 15-year old's vote a quarter" (reference) but that was dropped before the bill was floored. Or what if, for example, everyone over 12 but under the voting age (whether it be 16 or 18) was invited to the polling stations and given a different ballot to cast. It would have the same choices but would only be advisory: the results would be reported, which would provide feedback to government, engage youth, and introduce them to the voting process. If you're concerned about children just voting randomly or following their parents (not that it really matters: adults do the same anyway) there could be a place on these ballots for written comments. They're informational anyway and what better way to get feedback from interested youth. I don't know... just a wild idea.

My favourite quote was in an article discussing a California bill to preregister 16- and 17-year-olds so they would automatically be ready to vote when they turn 18. A Republican assemblymam who was against the bill says:
The truth is, when you're young you tend to think like a liberal. As you get older and wiser … you tend to become more conservative.
Ha! So we shouldn't even help register young people to vote because they'll be liberals! The article goes on to say that Democrats killed legislation to require voters to show photo ID, and that Republicans killed legislation to allow citizens to register on voting day.

This makes me want to start my normal rant on party politics and career politicians, but I won't. In the end, though, the issue isn't so much specifically about youth. It's about politicians who have been elected by the current system being reluctant to change it. Whether the issue is proportional representation, holding polling days on weekends, or reducing the voting age (all of which could potentially increase voter turn out and the accurate reflection of their vote), the status quo seems just too tempting.

Bash: printing the alphabet

(Sorry for the geeky topic, non-geeky readers)

Most have probably run across the "seq" command that will let you generate a list of numbers:
$ seq 1 4
But you can do similar with "echo" for a list of letters:

$ echo {a..d}
a b c d
$ echo {a..d}{a..d}
aa ab ac ad ba bb bc bd ca cb cc cd da db dc dd

Wednesday, 11 June 2008

Think work sucks? Think work sucks?
Actually, yes. Yes, I do.

Ok, let's be clear here: I like doing work but I don't really like the "traditional" work environment (an ironic term given the short length of time we've been working this way). Sarah forwarded me this article on ROWE (Results Only Work Environment) and all I can say is "Yes! Exactly! Has someone finally clued in?". Well, and also "Who came up with that name?". Maybe administrators love the name...

The idea is that you stop defining an employee's work in terms of hours in the office (or even hours worked, period) and start defining and measuring in terms of results. I've been lucky that most of my career so far has allowed me flexibility around the hours I work: a late morning here, a long day there, a banked day from working on the weekend, and so on. And I've always taken the attitude that, if I'm getting the job done, I don't need to feel too guilty if I spend a few minutes looking at a friend's photos while at work. But I've still always had to count hours because ultimately that's what I'm being paid for.

The difficulty with implementing a system like this, of course, is that it's easy to see if someone is sitting at their desk but much harder to determine whether they've achieved what you asked them to do. Plus you have to make sure your employees know what they're being asked to do. This may take more effort from managers to define tasks, set goals, and establish metrics. You might have to actually have those performance review sessions and dust off the 360 degree evaluation forms. But isn't this what managers are supposed to be doing anyway?

The end result is that there is just no excuse for not getting work done. As far as I can tell everyone wins, except possibly the guy in the back corner who hasn't done any actual work in 20 years. Maybe he gets motivated and actually finds himself happier? Maybe that's wishful thinking...

P.S. I don't think this ideas at the root of this are new or particularly insightful but if it's being adopted at all (and apparently Best Buy uses it) then that seems new.

P.P.S. What happens when employees start to feel like they're just being given a "harder" task than their co-workers?

Tuesday, 10 June 2008


Wandering through the bookstore at the airport in Kuala Lumpur, I came across this book. What caught my eye was not the title, but the claim on the front cover that the book was a lipogram and contains no E's. No E's?!! E is the most common letter of the alphabet. You can't even use "the" or "we" or "me" or "he" if you can't use E. How can you possibly write an entire novel without the letter E? At first I thought it was a joke but a quick skim certainly didn't turn up any occurences. Apparently the author wants to write one book for each letter of the alphabet... sheesh.

Tuesday, 27 May 2008

Two-way traffic

A few weeks ago we hired a driver and drove for two and a half hours (on the first day of a holiday weekend, no less) to one of the Great Wall sites. We thought we had negotiated a price that included road tolls so I was surprised to find us on small toll-free highways in so much traffic. I spent a good portion of the way there looking at sometimes-solid-sometimes-dotted expressway markings on the map that seemed to be running largely parallel to our route and telling myself that perhaps it just wasn't open yet. But the expressway isn't really the point of this story.

On several occasions, I've observed a certain traffic phenomenon here and this trip provided a good illustration. In China, you see, when someone or something on the road is in your way, you just... go around. Even if it means moving into an empty space in the oncoming lanes. Ignoring the danger of an accident, it seems reasonable enough at first glance: save yourself some time, keep the traffic flowing.

An hour or more into the trip we came along a particularly heavy patch of traffic and, as we inched along, I began to suspect what had happened: you see, this section of road had five lanes of traffic in total but there were four lanes going our way. Not a normal distribution. Sure enough as we moved along we came to a point where there were suddenly four lanes going the other way and ours all merged into one. Can you picture this? There was now one through-flowing lane in each direction and, in the middle, three lanes of traffic coming each way facing head-on into each other.

You can picture how this would happen: one direction slows up for some reason, someone darts around into a gap but doesn't make it back in because the traffic is moving slowly and there are no gaps. This blocks the traffic coming the other way and cars further back in that line, in frustration, dart around the stopped traffic ahead and find they can't get back in because the traffic has stopped. This is similar to the effect known as shockwave traffic, which produces traffic jams from something as simple as one car slowing too quickly in heavy traffic. Scientists in Japan recently managed to recreate this effect in a controlled environment. If you've never heard of this, check out the video mentioned in the above article or check out this computer simulation (click on "1: Ring-road" once it loads). Then you'll understand why that huge traffic jam you're stuck in often has no obvious cause when you get to the end of it. If you want to read more, check out here and here.

So, as the traffic keeps backing up, cars keep trying to go around and end up blocking more and more lanes. If the cars would only cooperate, they'd all get where they're going faster. Because everyone's trying to maximize their own forward progress, however, they end up in near gridlock. This kind of greedy algorithm reminds me of the Prisoner's Dilemma.

On our way home, we came upon the same patch of road and came to a complete stop for 15 minutes. Eventually, somebody opened a side street that had been blocked off and we were able to begin diverting around the stalled traffic into the countryside. As we moved our way forward, guess what I saw laid out in front of us?

Yup, 5 lines of traffic all pointing in the same direction...

Tuesday, 20 May 2008

Midnight Turf

It's midnight and I can't sleep. I'm lying listening to the rumble of trucks outside our window (open because it's getting to hot to sleep with it closed and we're still resisting A/C). "Turf?", I wonder?

Sure enough, a quick look out the window confirms the arrival of four large trucks carrying turf to be laid outside our apartment. The landscaping has been an ongoing project since we moved in nearly three months ago. Most of the time, the workers deliver their materials during the day using three-wheeled bicycle carts but some activities including including larger deliveries and a large loader shoveling bricks and other debris into dump trucks seem to occur in the middle of the night. Turf was laid on about half the area several weeks ago but much of was over-watered, died, and was finally pulled up again this week. Hence my hunch about the delivery.

Why the late hour? Apparently large trucks are prohibited within the 3rd or 4th ring road during the day without special permission; I hadn't noticed the nearly complete absence of trucks until someone mentioned it to me. Perhaps I should take a midnight walk one of these days and see what other trucks are driving around...

Monday, 19 May 2008

3 Minutes of Silence

At 2:28pm today, one week since the disastrous earthquake in Sichuan, people all over China maintained 3 minutes of silence for tens of thousands of victims who were killed. I stood watching from my apartment window as the office workers emerged from their buildings and the migrant workers planting flowers stood from their work. Everyone appeared to face southwest as car horns across the city began to blare continuously and in unison. What would normally be one of the most irritating noises imaginable, this time proved strangely moving...

Wednesday, 14 May 2008

The case for generalists

Seth Godin makes a case against being a generalist.

The problem is, I think the metaphor breaks down. Sure I want the best possible surgeon to fix my heart but I don't want to pay that surgeon to look at my sprained ankle or listen to my chest cough: there's a trade off there between specialization and cost. Also, I don't want a plumber, no matter how good he or she may be, to build my house for me. For that task I want a really good GENERAL contractor. Someone who knows just enough about all the parts of the house to make sure everything is done correctly and in the right order. A contractor who is good enough to know when to bring in other experts when necessary.

It seems to me there's a huge need for people who know enough about enough different things to communicate with and coordinate the specialists. Contractors, project managers, integrated health practitioners, and so on...

I've been thinking recently about companies hiring consultants. They bring in "specialized" consultants who throw buzzwords around and tell them what they Need To Do. The problem is that often nobody at the company knows enough to know what the consultants are talking about. What you almost need to do is hire another consultant (a generalist) for just long enough to talk to your other consultants and find out if what they're suggesting makes sense. I met a computer programmer type who did an internship (not specifically computer-related I think) with the airport authority in Beijing. In meetings, he was the only one with enough knowledge to really evaluate much of what the contractors were saying so they hired him on after his internship for quite a large sum of money to continue filling that role of "translator".

Now Seth also seems to suggest that the general contractor might even be considered a specialist role itself but then... what role isn't?

Monday, 12 May 2008

How did I not feel that?

News agencies across the world are reporting that a 7.8 magnitude earthquake hit central China this afternoon. It seems I was running on the elliptical machine at the gym at the time and didn't feel a thing.

I did notice construction workers streaming out of a tall building across the street and it seemed like people in the street (and even some of the gym staff) were looking up at something on the building. I even stopped running for a moment and turned off the mp3 player to see if there was a siren but couldn't for the life of me figure out what was so interesting.

Upon getting home I found two text messages on my phone asking if I had felt the earthquake and, having smelled a slight gas smell in the lobby on my way in, decided to grab our documents and head over to a coffee shop for a few hours just to be safe. The coffee shops at the business park near our house all seem busier than usual and I've overheard a few snippets from people who felt some significant shaking. I think people must have left their office buildings after the quake and many decided just to stay in the coffee shops.

Early reports suggest damage in Chengdu, the closest major city to the epicentre, is minimal though there seem to be some deaths and injuries in the smaller towns; I'm pretty sure there won't have been any damage in Beijing where it was apparently felt as a 3.9 magnitude tremor.

Saturday, 10 May 2008


Awoke this morning to the sound of "Edelweiss" (from The Sound of Music). What's particularly strange is that this is the second time I've heard this song here this week (we came across about 20 couples waltzing to it outside the Workers' Stadium last weekend). At first I thought it was our new neighbour upstairs who seems to practice the piano, the violin, the saxophone, and what someone more generous than I might call "singing" (even calling it karaoke would be gracious) throughout the day. Turns out it was a brass band playing for what looks to be a shop opening... how did this song get so popular over here?

Now I've got it stuck in my head again... gah!

Thursday, 8 May 2008

Olympic flame summits Mt. Qomolangma

I turned on the TV this morning just as the Olympic flame was being carried up the last 100m of Mt. Qomolangma (the Tibetan name for Mt. Everest), the world's highest peak. The media here has been anticipating this for weeks, probably since the torch relay began. While it seemed like a good gimmick I have to admit to a little bit of apprehension: it was hard to imagine the climbers calling of the summiting attempt, no matter how bad the weather was. How could you fail after building it up so much?

Luckily the weather looked good and they made great time to the top. And watching the footage I had to admit it was a fairly powerful image and pretty memorable achievement.

In looking into the torch relay, I discovered I had missed this leg of the Sydney torch relay: another unique gimmick.

Olympic Tickets

Looks like we've successfully obtained two tickets to see Beach Volleyball at the Olympics. I've been cursing myself for not getting on top of this earlier and picking through the scraggly tickets left at the various official national tickets sellers for which we could possibly qualify (Canada, Ireland, Germany). Not much left, let me assure you...

Then we caught wind on the news on Sunday that the 3rd round of Chinese sales would begin Monday morning, and as luck would have it Julia qualified on account of her visa type. The selection was still somewhat limited and I spent 5 hours or so Monday repeatedly clicking and waiting for the inevitable "Sorry, we are unable to process your request at this time" which was the result of the huge load hitting the servers. I finally got an offer of two beach volleyball tickets shortly after lunch. Because of a rather complex, bureaucratic payment process, I was unable to acquire any more but at least I'll get to see something (and the tickets were only $7.50 or so each!). Some people, having failed to get tickets in the first two rounds, apparently trekked across the country to line up for tickets at Bank of China branches in Beijing the night before. Olympic fever is building...

Wednesday, 7 May 2008

A Beijing Expat Wife

People here often ask what I'm doing here. I used to say I was looking for work or that I was doing contract work or thinking of taking some classes. Or I'd give the long spiel about how my girlfriend was doing an internship, blah, blah, blah. Now I just say "I'm an expat wife" and pause for the double take. Hence the new title for the blog: "A Beijing Expat Wife: Excuse the facial hair..."

Friday, 25 April 2008

Asterisk on VMWare

Many people have reported problems with Asterisk running on VMWare and having just decided to play with Asterisk I ran into them myself. First, here's what worked for me (using CentOS 4.6):

  • On the VMWare host machine, edit the .vmx config file for your image and add:
host.noTSC = "TRUE"
ptsc.noTSC = "TRUE"

  • In your CentOS image, disable the cpuspeed process:
chkconfig cpuspeed off

It actually worked fine for me without the extra kernel parameters but the CPU usage seemed to be slightly lower at idle with them added so I left them. The issue seems to be the default clock rate (1000Hz) used by most 2.6 kernels which can't be adequately serviced in the VM guests. The kernel-vm kernel is built with the clock rate at 100Hz.

Apparently CentOS 5 has a new kernel that allows you to specify divider=10 clocksource=acpi_pm to the kernel which tells it to run at 1/10 the clock rate it was compiled with.

Some links:

Wednesday, 23 April 2008

Adjusting MTU to improve ADSL performance

So here I am in China, being driven mad by the number of dropped packets I'm seeing. Mostly it's not too bad until I make a VPN connection and then I'm seeing somewhere in the neighbourhood of 10% packet loss on a ping.

Well, after much, much debugging and internet searching I think I've finally found my problem: the MTU setting. MTU is the Maximum Transmission Unit or maximum size of an individual frame being sent on a network. Ethernet has an MTU of 1500, which is usually set as the default value for connections (at least in MS Windows). But other network equipment along your route may not be using Ethernet and may have a lower value. This is ok in theory; the equipment will break up larger frames into smaller ones and combine smaller frames into larger ones as the data is transferred between different networks.

The problem usually comes when your ISP's equipment is using an MTU slightly lower than your value. Apparently PPPoE connections always have a lower MTU than Ethernet (usually 1492) but Windows XP already accounts for that by reducing the MTU for PPPoE connections to 1480. In my case, though, China Telecom seems to be using equipment with an MTU of 1398.

What happens is that I send out 1480 bytes of data and China Telecom sends the first 1398 bytes and then holds onto the remaining 82 bytes until it gets some more data from me. Let's say I send another 1480 bytes. Now the first 1316 bytes of that frame are added to the 82 bytes from my first frame and sent and the remaining 164 bytes are held onto. This is slightly inefficient and can make the connection seem "bursty" because data is being held up before being sent. It also tends to get a lot worse if your connection is flaky (which mine is); I think this is because a lost frame along the way may hold part of the data for 2 or more of my original frames, all of which now need to be retransmitted. VPNs (because of the way TCP packets are being embedded into TCP packets) apparently amplify these effects further.

So how do we solve it?

Finding the Correct MTU

First, you need to find out what MTU setting your ISP is using. This MS KnowledgeBase article describes how to do this. Basically, you pick a host as close as possible to you on your routing path and run the following command:

ping -f -l size

The -f marks the packet as non-fragmentable and the -l size parameter specifies the number of bytes to include in the ping packet.

Start with a value of 1472 for size. If the ping is successful, your ISP's MTU is 1500. If you get the error Packet needs to be fragmented but DF set then decrease the value of size and try again. Once you find the highest value that allows a successful ping, that value plus 28 (20 bytes for the IP header and 8 for the ICMP header) is the correct MTU value.

Adjusting the MTU Value in Windows XP

Follow this KnowledgeBase article to set the MTU value for PPPoE (also seems to affect all VPN connections - not sure if there's a way to avoid that). Basically you want to set the following registry keys (all values are DWORD type):

ProtocolType = 0x00000800
PPPProtocolType = 0x00000021
ProtocolMTU = 1480

Obviously you would adjust the MTU value as desired and make sure to choose "decimal" when inputting the value. You probably have to reboot.

You may alternatively be ale to specify the MTU for the PPPoE interface here:

MTU = 1500

but I haven't tested it yet (again, adjust the MTU as desired and choose "decimal" when entering the value).

UPDATE: I tested setting the MTU for just the one PPPoE interface as above and it seems to work fine. I set the MTU value for NdisWan back to 1480 and will probably test removing that section from the registry altogether to see if it still works.

Other possibly helpful links: