Methodologies considered risable

A recent blog on The Agile Fallacy got me thinking. I've been programming for 45 years and still am, although slowly. I have seen the wheel go round many times, and I have a somewhat different perspective to that expressed in the blog. I won't mention any specific technologies because that will only yank various folk's chains. In the 1960's most big development projects delivered late and lousy. It became a standing joke that programmers used all of the project time and budget to deliver the first 90% of the agreed deliverable, and an equal amount of time and money to deliver the other 90%, assuming they were allowed to carry on that long. Peter Jackson famously remarked that movies don't actually get "released", they escape into the wild, and the same can be said for most large application development efforts. Whatever isn't done right at the time of release is done later, and called maintenance. The programmer's ultimate refuge is to claim "the users' requirements changed", even if the requirements were to implement double-entry bookkeeping which has been around in its present form for about 800 years. I rambled on about this issue in an earlier blog entry.

Fred Brooks, manager of IBM's OS/360 development project, wrote The Mythical Man Month to describe many of the pitfalls found in the 1960's. A lot of DP managers (that's what we used to call them) got fired as a result of non-delivery, and in self-defense they developed "methodology", elaborate systems of checks and balances, reports and reviews, that slowed the whole development process down to a point where there were far fewer ugly surprises. You can tell that "methodology" was a con game from the start by its name – "methodology" should mean the study of methods, not the practice thereof. It should have been called just "methods", but that was far to simple a word for the grand collusion then underway. With the introduction of "methodology" productivity plummeted, but the DP managers got to keep their jobs, and that's what counted most (to them).

The underlying problem with predicting how long development will take is that the productivity of individual programmers varies hugely. A research project at Dartmouth College back in the 1960's showed that the slowest student programmers were over 100 times slower than the fastest. The only reason that the study showed "only" a factor of 100 is that the researchers got tired of waiting for the last few laggards, closed the lab and went home. Sadly, you can't tell which programmers are fast and which are slow just by looking that them, so it's hard to pick a winning team. Ornate methodologies overlay so many admin tasks on all programmers that the difference in productivity between the best and the worst is masked.

While DP managers were happy with "methodologies", the really competent programmers were totally frustrated with them because they knew that they were being hobbled. As they say, "how can you fly like an eagle when you are surrounded by turkeys?" Some of the more cunning ones solved the problem by developing revolutionary new "methodologies" that allowed them to deliver running code dozens of times more quickly than the old, high-bulk "methodologies". On inspection, these new "methodologies" succeeded by jettisoning most of the padding that had been packed into the old ones, thus allowing the good programmers to get the job done fast. In order to allow them to use these new techniques, the practitioners had to dress them up with fancy new names, buzzwords, smoke, and mirrors.

Things went well for a while with the competent programmers delivering code at speed, but after a while the rest of the industry demanded the right to catch up. Snake oil salesmen were quick to "analyse and to codify" the new fast-track methodologies, and to come up with books and courses that would allow any pedestrian programmer to fly like an eagle. Or so they said. Institutions all over the world tried the new panacea, and found out the hard way that they didn't work for them. The new paradigms became discredited, but the fast programmers now had the bit between their teeth, and they realized that the way forward was to coin a new name for low-bulk programming every time the previous name got discredited. To date we have seen a number of iterations of this particular wheel, with the smart programmers soaring like eagles while the rest of the pack follow behind, crashing through the undergrowth and gobbling wildly as they struggle to become airborne.

So what should we do about this problem? The reality is, we can't afford to do without the really competent programmers, and if the new methodology game is the price that we have to pay in order to allow them to get the important stuff done fast, so be it. Just be aware that your mileage may vary.

Five Fingered Keyboards (3)

I have at long last put together a small demo of a five finger keyboard, as discussed in a previous entry in this blog. I have released it as an open source project on Google Code, here. It's an HTML page with embedded JavaScript that illustrates the concept of a sequence-sensitive chording keyboard. The version that I uploaded, v0.1, is limited and buggy, but it's a beginning.

DIV_SRC could cut news feeds' bandwidth by 90%

The Internet is taking over from printed newspapers, magazines and journals. It is also enabling ordinary people like me to publish their thoughts, hopes and ideas in blogs such as this one, almost free of charge. Major news feed sites such as as BBC News, CNN and Slashdot.org have millions of readers word-wide, and the amount of bandwidth consumed by people browsing news feeds is large and growing fast.

News feeds change their front pages often, and for this reason they make them uncacheable, or if they can be cached then it's only for a minute or so. Pretty much every time a person browses a news feed, they get a complete copy of the page from one of the news feed's servers. But if you look at a news feed you will find that it is made up of a number of short stories, often with links to fuller versions of the story. As time goes by, new stories are added and the older ones get pushed down the page, eventually falling off the end into an archive. If you look at a given news feed twice in ten minutes then the chances are good that while the content may have moved around a bit, most of it will be unchanged. And while you probably wouldn't look at the same news feed that frequently, if you consider all the people who share your ISP, and hence your ISP's caching proxy server, the ISP may have to fetch a given news feed page in its entirety thousands of times a day, even if 90% of it is unchanged from one fetch to the next. This places a large and growing load on the news feed's servers and bandwidth, and on the ISP's upstream bandwidth.

I have written an open source program called DIV_SRC, the package can be downloaded from Google Code. The tool allows large, composite HTML pages like news feeds to be broken into sections. The base page references these sections with <div src="URI"> tags (or any other tag types that the caller chooses). DIV_SRC.js is a client-side JavaScript program that finds all of the section references of this sort in the base page and fetches the referenced content using AJAX. A demo news feed shows that if news feeds used DIV_SRC then their main pages, which are necessarily dynamic, would shrink to about 5% of their current size. The individual stories that they carry could then be cached by browsers and caching proxy servers, reducing the news feeds' bandwidth bill by 90% and giving better response times to users.

No server-side coding is required to implement DIV_SRC, it can run off a vanilla web server that delivers nothing but the usual static files of HTML, images, style sheets and JavaScript. The DIV_SRC.js JavaScript code is compact, less than 5KB in terse form.

DIV_SRC could also be used by blogging engines to reduce bandwidth consumption and to improve response times. Blog pages delivered from blog engines such as Blogspot.com come with the HTTP header Cache-control: max-age=0, which prevents browsers and caching proxy servers from caching blog pages. This ensures that every viewer always gets to see the latest version of the page, which is good, but at a cost. If the blog is popular then browsers all over the Internet will be getting fresh copies from the blog server every time the blog is viewed. By using DIV_SRC, blogging engines could ensure that browser users continue to see the latest version of each blog, but the unchanged sections of the blog will not have to be transmitted from the blog server to the browser every time a given blog is viewed. Each blog entry would be stored as a separate static HTML file which could be cached by browsers and/or caching proxy servers anywhere in the Internet.

There are some issues that would have to be addressed, like what happens if a browser without AJAX capabilities (such as a mobile phone browser) accesses a composite page. This question and several others are dealt with in the Problems page of the project's website.

Cattle shown to align north-south

A recent BBC article reported that a group of scientists had studied a large number of Google map images looking for cattle, and had found that most of them adopt a north-south orientation. The resolution of the images wasn't good enough for the researchers to distinguish which way the animals were pointing, north or south. They followed up with field studies of wild deer in the Czech Republic and determined that most of them faced north while about one third faced south. They checked, and the animals didn't track the sun as it moved across the sky, their orientation was consistently north-south regardless of the time of day. Their conclusion was that the animals possess a magnetic sense, and that they line up with the earth's magnetic field. We believe that migratory birds can sense the earth's magnetic field and use it for navigation but no organ has yet been found in large mammals that confers the ability to discern magnetic fields, but perhaps one will be found one day. The researchers do not offer any compelling explanation for why the animals would benefit from lining up with the earth's magnetic field.

It seems to me that there is another, simpler explanation for this behaviour. Each hemisphere of the earth undergoes seasonal variations - summer, autumn, winter, and spring. During winters sunlight is diminished and the growth of vegetation slows down. Animals deplete the available grazing, and in their natural state when unencumbered by fences they migrate in an equatorial direction to reach pastures where the daylight hours are longer and the vegetation more plentiful. During the winter they deplete much of the available vegetation, and when spring and summer returns, they reverse the process, migrating towards the poles to exploit the new vegetation that grows in response to lengthening days. So their orientation may simply reflect the general direction that they're migrating in when observed. If so, one would expect to see a strong correlation between the local seasons and their orientation - they should be pointing predominantly towards the poles in spring and summer, and predominantly towards the equator during autumn and winter.

It is not necessary to suppose that the animals have magnetic senses in order to point north or south. Bees navigate by the orientation of the sun in the sky, automatically making allowance for the apparent movement of the sun across the sky. If they can manage this calculation with their small brains then mammals should be able to do so as well.

Another Colour Riddle

Just to complement my earlier colour riddle concerning pale people in Europe (link), here's another one, this time set in Africa.

A poacher cuts through the fence of a game reserve at dusk and enters. He travels one mile south, looking for rhino tracks. He sees a few tracks that run down a game path, and follows them one mile west. There the tracks split up, and the poacher must choose which set to follow. He follows one that turns north, and after a while he finds a pile of rhino dung. He picks through it with a twig, noting the grassy content. He tracks the rhino further north, completing a mile from where the tracks diverged, and sees the rhino. He closes in on it quietly, mows it down with an AK47, cuts off its horn, and heads back to his point of entry, one mile to the east.

What is the colour of the rhino?

The answer is "white", but there's a twist in the tail (or tale, if you prefer). In this riddle, the carefully reported distances are irrelevant. The key fact is that the rhino's dung contained grass. White rhinos eat grass. They have long, downward-sloping heads that position their muzzles close to the ground, handily placed for grazing grass. Black rhinos on the other hand eat trees. They have short necks and heads, and are well able to crane their muzzles upwards to reach the twigs on which they browse. The diet of the rhinos can easily be determined from their dung.

Why did the poacher care if he was following a white rhino rather than a black one? White rhinos are about 50% heavier than black rhinos, and generally have bigger horns. The poacher gets more money for a heavier horn.

And the twist in the tail? "White" rhinos aren't actually white, they're a dull gray, much the same colour as the "black" rhino. It is believed that the white rhino was originally named "weit" (wide) by German settlers in East Africa. The white rhino's muzzle is broad and squared-off, which helps it in grazing grass. The black rhino on the other hand has a more pointy muzzle with a long, prehensile upper lip that it uses to grab twigs and direct them into its mouth.

And the moral of the story? Get to see some rhinos (see link) before they're all extinct.

Desktop, schmesktop meets Android

Seven years ago the IT tabloids were full of the question, "when will Linux take over the desktop?" I wrote in Linux Today:

Why the desktop fixation? Soon desktops will be history. The concept that you might need to go to a machine on a special desk in a particular room of your house whenever you need computer power will be as obsolete as the notion of going to the handpump over the well in the yard any time you need water. Houses have been "wired" for water and electricity for 100 years. It's everywhere you need it, just turn the tap or flick the switch. The same will happen with computer power this decade. Significant appliances will have all the computer power they need built-in, and connect to the web to make themselves accessible. people will wear whatever computer power they need while they're on the move. Linux is already shaping well in this world of "imbedded logic".

When news of Alexander Bell's wonderful new invention, the telephone, was telegraphed around the world over 100 years ago, the mayor of Chicago was particularly enthusiastic. He said that he could foresee the day when every city would have one of these devices. As it turned out, telephones played in a different market to telegraph offices. And Linux will play in a different market to desktops.

On the surface, things haven't changed that much. The IT tabloids are still pounding the same question. It is becoming less relevant with every passing day.

By comparison, when last did you last hear someone ask:

"When will Windows take over from MVS/zOS on the mainframe?"

The answer is, never. zOS is doing just fine on the mainframe, but the mainframe has become a niche market. A pretty capacious niche too be sure; IBM sells more mainframe MIPs than ever before, but the market place has moved on. There are far more MIPS on desktops than there ever were in data centers.

The turn of the desktop will come too. It has served us well, and old-timers like me will remember it with fond affection for many years to come, but we will have our work cut out trying to explain to the next generation why anyone would bother to go to a clunky box on a desk in a particular room room whenever they needed computer power. That's just about as absurd as suggesting that whenever you want to make a phone call, you have to go to a clunky black Bakelite contraption attached to a long wire in a particular room. Actually, I still have one of those, but it isn't Bakelite. Maybe I could find a retro model in a novelty store? Nowadays most folk, even in developing nations, carry slim portable wireless phones around with them. There are more mobile phones than fixed line phones in the world today, and mobile adoption is accelerating. They are typically powered with 200MHz or better processors. That's 5 times more processor power than NASA used to put the first men on the Moon in 1969 (they used three IBM System/370 model 168 systems, each sporting 12.5MHz processors).

So great, we have access to these powerful portable processors, and each has instant access to the Internet. But what can we do with them? Well, make calls to be sure. But modern cell phones can do a lot more than just that. The biggest inhibitor to their full exploitation is, as is ever the case with computers, the effort required to develop a full software stack, from the kernel all the way up to the glitzy user interface.

Until now.

Google has changed all of that. They have announced and made available a full open source stack for mobile phones. It covers everything from the bare metal upwards, and it's extensible. It's called Android. Check the link for an overview of what's in it, plus some videos. The cost of entry for would-be mobile phone manufacturers has suddenly plummeted. PC sales have already started declining in the East, where gizmo adoption rates are high. Stand by for descent, the West. And stop worrying about when Linux will take over the desktop:

1. It never will
   
2. No one will notice, or care (except for the Smithsonian)

Colour Riddles

There's an old riddle that runs like this:

A hunter sets off from his base camp. He travels one mile south, sees bear tracks, follows them for one mile east, finds and kills the bear. He skins it and carries the skin back to his camp, which is one mile to the north.

What is the colour of the bear's fur?

You are supposed to reason that if a trip of one mile south, one mile east, and then one mile north brings the hunter back to his base camp then he must have started from the North Pole, and so the bear must be a Polar bear, and hence its fur white. Of course Polar bears don't range that far north , but global warming is rapidly depleting the Arctic ice pack, and perhaps one day there will be Polar bears near the North Pole, searching for enough ice to stand on.

Nit-pickers have pointed out that you can also choose a starting point near the South Pole such that after you have travelled one mile further south, a trip of one mile east will take you in a complete circle round the South Pole (or two or more whole circuits if you started closer) back to where you started circling, and the one mile leg back north would then take you back to where you started. But there are no bears at all in the Antarctic, so that doesn't fit the riddle's premises. Perhaps, if the Arctic ice pack thaws out completely, we will transport some Polar bears to the Antarctic; it's solid ground near the pole, with a high enough altitude that it will stay cold far longer than the North Pole will.

I have devised another "colour" riddle for readers to ponder:

The scene is a valley in what is now western Germany, 10,000 years ago. An ice age has the northern hemisphere in its grip; although the season is late spring, snow still blankets the land. A curl of smoke issues from the mouth of a cave, which is largely closed off by hanging hides. A weak sun shines through thin cloud. A group of men straggle out of the cave. They wear thick furs; only their faces are exposed to the cold weather. They carry bows, arrows, and spears. They cast about, looking for animal tracks in the snow. Soon they find some deer tracks, and set off in pursuit of their prey. Almost all the men walk with some difficulty; their legs are badly bowed, they walk with bandy gaits. But the leader of the group has straight legs, and walks easily and quickly. After a while they close in on their prey carefully; they unleash a volley of arrows, and the deer is fatally wounded. It swiftly bounds away, but quite soon weakens and slows. The group leader runs fast and closes in on the deer. He uses his spear to kill it. The rest of the group join him. They exult in their success. They quickly butcher the deer into sections with their sharp flint knives so that they can transport the meat back to their cave. The leader has done more than the others to secure their quarry, and so he takes a larger share than the others. He and his family will eat well.

What is the colour of the leader's eyes?

Rickets is a human disease that causes softening of the bones in children, resulting in bowed legs in severe cases. It is caused primarily by a vitamin D deficiency. Our bodies will create all the vitamin D that they need if exposed to sunlight. Early humans who had to live through ice ages spent much of their time in the shelter of caves, and they wore comprehensive fur clothing to keep warm. Archaeological remains tell us that many of them suffered from rickets, often severely.

What has this got to do with the colour of the leader's eyes, you may ask? I am blessed (or cursed) with a fair complexion – a pale pink skin, blue eyes, blond hair when young that darkened somewhat with age, before it fell out. There are hundreds of millions of other humans with my general colouration; we are called Caucasians, which tells you something of our origin, the Caucasus in Russia. While we are many, we are greatly outnumbered by other varieties of human, almost all of which have darker skins, dark hair, and brown eyes.

As a fair person living in Africa, I am vividly aware of how poorly equipped I am to cope with the hot sun. The melanin in my skin, which should protect the deeper layers of skin from the sun's ultraviolet radiation, is dysfunctional; it is clumped into small circular disks called "freckles", and all of the skin between is unprotected. I have often felt the sharp sting of sunburn, and had to deal with the outer layers of my skin peeling off. My blue irises do a poor job of keeping strong sunlight out of my eyes. When the light is bright, much gets through my blue irises, suffusing the scene with glare that washes out the details that I need to see. Hair on heads may benefit us in many different ways; the way that I miss most is to protect my pate from sunburn. Dark hair does a better job of this than does blond hair.

Given my dysfunctional complexion, I have often wondered why it ever persisted when by chance it first arose amongst my forebears. They didn't have the benefit of sun tan lotions, lip balms, dark glasses, and fedora hats. How did they manage to out-compete their darker kin? The riddle that I posed above may give us an explanation. Pale people arose only in cold climes; there were ice ages that made these climes colder still; rickets was a major problem under these conditions; pale people absorb what little sunlight there is more than do darker people, and hence would suffer less from rickets.

Interestingly, it was recently determined that red hair was common amongst the Neanderthal people (see here). They too developed a form of albinism, as did the Caucasians. They too lived in Europe through a number of ice ages. They too suffered from rickets. So far as we can tell, Homo Sapiens did not descend from the Neanderthals, so this is probably an example of parallel evolution; a common solution to a common problem.