Trevor's Trinkets

Smart phone chorded keyboard

2011-07-03T09:06:00.005+02:00

In 2007 I blogged about five finger keyboards, also known as chorded keyboards, as an alternative to the dinky and hard-to-use physical keyboards on mobile phones. Five buttons located on the edge of the phone, under the fingertips in normal use, could be used to enter text whenever needed, without having to waste time moving fingers from one key to another. In 2008 I built a simple demo of the idea in JavaScript.

Over the past few years smart phones have become common, and their preferred entry method is via a pop-up "soft" keyboard on a touch sensitive screen. Mostly we use one finger to hunt-and-peck text into these, but some interesting alternatives have been developed, like sliding your finger across the keyboard from key to key, or using gestures. Gizmode has a nice article on some interesting new ways to capture text into an Android phone.

Android 2.3 Soft Keyboard

If we implemented a chording keyboard on a touch-sensitive screen, how would it compare with what's out there already? Let's start with the default soft keyboard that ships with Android 2.3. It has 34 soft keys, of which 29 yield characters and 5 functions (e.g. backspace). You tap a key briefly to get its character or function. 28 of the character-bearing soft keys have a extra character value that you access by pressing and holding the soft key, providing a total of 62 characters and functions. These extra values include the numerals and special characters used for punctuation and maths. As on PC keyboards, the extra values appear above base values on each soft key, so it's easy to work out which soft key to press in order to get an extra character. For example:

3
e

15 of the Android 2.3 keyboard's soft keys carry additional letter variants, that is, the base letter plus a diacritic of óne type or ànother, 57 variants in all. They aren't visible on the soft keyboard, but each relates to a base letter on the soft keyboard. For example, you access the diacritic variants of the letter "e" by pressing and holding the base letter "e". A pop-up appears with its diacritic variants following the default "3" which appears in the upper half of the soft "e" key:

3èéêëęě

You choose between these options by sliding your finger sideways. The focus moves across the list of options in sympathy. The character that has focus when you lift your finger from the screen is taken.

Soft Chording Keyboard

Let's design a phone app to capture text using chording key principles, but with the fingers touching soft keys on the phone's screen. Ideally, the fingers should only have to move up and down in one place. With my adult male fingers, I can only fit three fingertips across the width of my HTC Wildfire's touch screen. A classic chording keyboard with three buttons would give us 2^3-1 = 7 different characters, way too few (I'm using the notation A^B to mean A to the power of B). If we take into account the sequence of touches and allow for sequences of 1, 2 or 3 touches then we get 3*2*1 combinations that include all three fingers plus 3*2 combinations that include only two fingers plus 3 combinations that use only one finger = 15 different characters, still way too few. If we take into account the sequence in which the finger tips touch the screen and also the sequence in which they leave the screen then we can encode 3*2*1 * 3*2*1 + 3*2*2 + 3*1 = 51 combinations. Better, but we're still short of the 62 characters that the Android 2.3 keyboard can encode, ignoring case shift and diacritics. We will have to allow individual fingers to press and release their assigned soft key more than once in a coding sequence. Then the sky is the limit.

While we're about it, let's note that most pop-up keyboards take up a lot of real estate. If the user needs to fill in a multiple field form then the pop-up keyboard will likely cover most of the form. It would be nice if our chording keyboard used up less screen space, but still guide the novice user through character selection. Here's a sample "home page" for our pop-up chording keyboard:

Backspace	Blank	Shift: Aa	New line
Put three fingertips on this line to enter text

We have a list of four common functions/characters across the top. A production version would use symbols rather than words. The lower half is where the user's fingertips will rest. We could predefine three blocks and require the user to put a fingertip in each, but lining up fingertips with blocks would take time. Instead, the user is free to place three fingertips anywhere on the line of text, and the keyboard app will find them. When this is done, the top line is replaced by cues that lead the user through the process of selecting a character. The user then lifts and lowers fingers in various sequences to choose characters. At any time the user can lift all fingers simultaneously to cancel the process and return to the home page. Once selection starts, here's how the first set of cues might look:

a-m	n-z	12#
(==)	(==)	(==)

a-m gives access to the first half of the alphabet, n-z to the second. 12# gives access to numbers and special symbols. (==) marks where the user's fingertip happens to lie. The app writes the cues above each fingertip. If the user lifts finger 1, selection of a character in the range a-m will start. The cues will change, and the keyboard pop-up might then look like this:

a-d	e-h	i-m
(---)	(==)	(==)

(---) marks where fingertip 1 was before it was lifted; the other two are still down. The keyboard app wouldn't actually display the (==) or (---), they're only there to assist with the explanation. The first cue a-d is italicised to indicate that the user should lower that finger to choose that option. If the user lifts finger 2 then the pop-up might change to:

e	f-g	h
(---)	(---)	(==)

Bold cues show tell the user that if that soft key is chosen, the selection process will arrive at a character. In this case, lowering finger 1 will select the letter e. After that, the user could lower finger 2 so all three fingertips are on the glass. The keyboard would return to the top character selection page, ready to start the next character.

Accessing Diacritic Variants

We can allow users to access diacritic variants of letters in a similar way to the Android soft keyboard. Once you complete a sequence that yields the letter "e" for example, then the selected letter, and the secondary character that shares its soft key, and the diacritic variants of the selected letter could appear in the cue area like this:

e3èéêëęě
(==)	(---)	(==)

The selected character "e" has the focus. If you ignore this cue and lower finger 2 to start selecting the next character, "e" will be chosen as the next input character. If you pause, focus will move to "3", the secondary character that shares the "e" soft key. If you pause longer, focus will move cyclically further to the right, wrapping back to the left once it runs off the right edge, so you can select any variant just by waiting. Alternatively, you can slide your fingers to the left or right, and the focus will move left or right in sympathy. Whatever character has focus when you lower finger 2 will be selected. Fingertips that slide off the side of the screen will be ignored. The base letter and the focus will be located at the right of the string of options if fingertip 3 is the only one in contact with the screen, so that it has room to slide left. After a sideways slide you might have to lift all fingertips and re-centre them, which is undesirable, but you won't have to be too fussy about where you locate them, so it should be quick.

Character Code Points

Starting from the home position with three fingertips on the screen, you have a choice of three movements – raise any one of your three fingers. After any of these movements, you have another three choices, and so on. You are navigating a trinary tree, with a single character at each leaf position and cues at non-terminal nodes. A five level balanced trinary tree would have 1 root with 3 child nodes, 9 grandchild nodes, 27 great-grandchild nodes and 81 great-great-grandchild nodes. Each successive level is reached by raising or lowering one fingertip. Of the 27 great-grandchild nodes, one will have all fingers raised, which position we reserve to end the current character selection sequence as an "Oops!" gesture. So we will lose three potential code points, leaving 78. Of these 78 leaf nodes, some have all three fingers down. We discard these because that configuration is required to start the next character. All fingers down can be reached after four moves by raising two fingers and then lowering two fingers, 3*2*2*1 possible sequences = 12 code points; or by raising and lowering a finger and then doing the same again, 3*1*3*1 = 9 code points (we accept "all fingers down" as valid if this happens at level 2). That leaves 57 code points, five less than the number of characters and functions that appear on the Android soft keyboard. But one of the Android keys is not needed (12# selects special characters), and our home page includes four common functions to compensate. All of the code points can be accessed with only four finger up-down movements.

In practice, we may choose to follow the example set by Samuel Morse and allocated shorter code sequences to more common characters, and longer code sequences to less common sequences, to reduce the average number of finger movements needed to code a character. The sample screens above allocate 2/3 of the code space to the alphas, although they form only 42% of the Android character set. We also need to group the special characters so that we can give cues that will guide the novice user to find each character required. They can be divided into the 10 numerals, characters that can be drawn with one continuous line stroke @&()_-,./ and characters that require more than one line stroke !#$%+=:;"?

If we introduce longer and shorter code sequences then we will also be able to include other characters that are present on the PC keyboard but not the Android 2.3 keyboard, such as `~[]{}<>

How Quick Would It Be?

Could a chording keyboard achieve reasonable text entry rates in the hands of a practised use? You can benchmark your own digital dexterity by resting your wrist on a flat surface and drumming three fingertips on it, as if you were impatiently waiting for someone. Try different patterns. It's an easy action for the fingers. Many folk can clock 10 taps per second; that's 20 finger movements per second. If we use a balanced trinary tree for our coding and include an extra movement to get all fingers on the screen to start each character, we need 5 finger movements per character entered (ignoring capitals and diacritics), so our fingers are theoretically capable of entering 20/5 = 4 characters per second or 48 words per minute, faster than the average typer on a full-sized PC keyboard. How fast we would be when typing a given text is another question which only experimentation can answer. Of course you can type letters at a furious pace if you simply drum your fingertips on a PC keyboard, but you will get the same three or four letters repeated over and over again. Once you start moving your fingers around the keyboard to find particular letters, the process slows down greatly.

Rhino Poaching

2011-03-02T08:23:00.000+02:00

A tide of rhino poaching has swept down through Africa, and has reached South Africa where the largest pool of surviving rhinos are to be found. Rhinos are killed for their horns, which are believed to have medicinal properties by many in South-East Asia. The horns are also used to make ceremonial dagger handles. Armed conflicts have arisen between gangs of poachers seeking rhinos and the rangers whose job it is to protect wildlife. Some poachers have lost their lives in these exchanges. Many people have responded to these developments with great anger, and some have expressed the opinion that the poachers got no more than they deserved. It seems that to them that the lives of rhinos are more important than those of the humans that hunt them. Others have suggested that the South African army should be deployed to protect the rhinos from further poaching.

Before one picks a fight, or accepts an invitation to fight from another, it is prudent to size up the opposition. There are about 2.5 billion people in South-East Asia, while there are only 45 million residents in South Africa. If it comes to outright conflict or even just economic jostling, there is no way that South Africans, or the rhinos, can win this fight. Enlisting the army won't solve the problem either. Rhino horn fetches a higher price than its weight in gold. For that kind of money, the cooperation of an army can be bought, and they come with their own weapons.

After centuries of hunting and poaching, rhinos are quite rare, and they have been accorded the status of being an officially endangered species. It is illegal to trade in rhino products. That has had little effect, other than to drive the price of rhino horn up. Rhinos have become an icon of endangered animals in Africa, a subject of passionate discussion. It has become difficult to engage in rational debate on this subject.

Suppose, just for argument's sake, that it was cattle horn that was so sought after by people in South-East Asia, that for some reason they were unable to raise enough cattle to meet their demand for cattle horn, and that they approached Africa willing to pay high prices for it. How would we respond? Would we build high electrified fences around our cattle and deploy armed guards to protect them? Probably not. It's more likely that a wave of entrepreneurs would buy land and start more cattle farms, steadily raising production until the demand was met. The law of supply and demand tells us that as the production of cattle horn increased, so its price would drop. The incentive for buyers to seek out poachers, or for poachers to engage in the risky business poaching, would be much reduced. As the supply of cattle horn became more plentiful and its price dropped, more and more people would be able to buy it and to test its supposed medicinal properties for themselves. Health researchers would be able to scientifically test its efficacy against various diseases, and to report these results. If it turned out that cattle horn was indeed a cure for a wide range of diseases then large numbers of humans would be able to benefit from this discovery, and cattle production would increase further. If on the other hand it turned out that cattle horn was no better for curing diseases than toe-nail clippings (which it chemically resembles) then demand for cattle horn would diminish, although still continue at a lower level since some folk would remain convinced of its curative powers regardless of what the researchers said.

Let us now return to reality, where it is rhino horn that is in such demand that rhinos are being driven rapidly towards extinction. If we really want to avoid this outcome, and indeed to see an increase in the number of rhinos as time goes by, we should legalise rhino farming and the sale of rhino horn produced on these farms. This would enable a far greater number of people in South-East Asia to enjoy the supposed benefits of rhino horn, and also provide new employment opportunities and prosperity for many in Africa, where rhinos flourish when given the opportunity.

Some readers may respond with horror to the suggestion that rhinos be bred on farms for the production of rhino horn. But we breed cows, pigs, sheep, goats and chickens for food, hide, and many other uses, so why not other species? Rhino meat is perfectly edible, so it's not that the rest of the beast need be discarded. Rhinos are better adapted to the semi-arid areas of Africa than are domesticated animals, and are not susceptible to many of the diseases that afflict cattle. If rhinos are farmed for their horns and meat then those rhinos that remain in game reserves are far less likely to be massacred for their horns. Let us not allow our emotions to cloud our judgment to the point that we inadvertently drive free-range rhinos to extinction, and deny future generations the joy of seeing these wonderful animals roaming free in the wild.

Backup and Recovery Considerations

2010-02-07T16:26:00.000+02:00

Or – Why Murphy was an Optimist

Every IT system is subject to failure. When things go wrong, we try to recover, usually by using a backup copy that we took just in case. This post deals with some of the considerations that one should keep in mind when planning for backup and recovery. It doesn't attempt to cover the details of the backups that are required for any specific IT technology, they are so diverse, and in many cases so complex, that it's impossible for a blog post to cover the ground. Instead, it deals with questions like how should you plan for backup and recovery.

Along the way, we also take a sidelong look at Murphy's Law, which as we all know states:

Anything that can go wrong, will go wrong.

You may also have heard the claim that Murphy was an optimist. That sounds absurd, given the totally bleak outlook of Murphy's law, but in fact it's perfectly true, for reasons that everyone who is responsible for planning backup and recovery needs to know. We'll deal with that issue too.

Planning Backup and Recovery

Backup and recovery doesn't just happen – especially the recovery part. If you want to be able to recover from your backups when things go wrong, you have to plan carefully. There are three major variations in the way that you can do the planning. I call them foresight, on-demand, and hindsight.

Foresight Planning

This is the hard way of doing the job. Before anything goes wrong, you sit down and think carefully about all the things that could possibly go wrong, and then you apply Murphy's rule and assume that if something can go wrong, it will go wrong sooner or later. You then work out what backups and other evasive action you will need to take in order to recover from each scenario when it goes wrong. This is doing the hard yards in advance, and there are several reasons why it's difficult and rather unrewarding to do:

Few other people on your organization will agree with you that your worst-case scenarios could come about. Ah, don't worry about it, it will never happen! is a phrase that you will get sick of hearing. It's hard to get the cooperation that you need from other folk with this mode of preparation.
If you do a really good job and handle the catastrophe when it happens without even breaking a sweat, no one will even know what a good job you did. They'll all think that your particular patch is a really easy one to handle.
In particular, your boss won't know what a valuable contribution you make to his peaceful sleep till you move on to better things and someone less well prepared than you takes over.

On-Demand Planning

This is a more risky way of handling the problem. You wait for something to go wrong, and when it does, you size up the situation and then make a brilliant, intuitive move to fix it all in one stroke. Well, maybe not one stroke – would you believe two strokes, or maybe three? This approach has a number of advantages, but also some potential downside:

You don't have to spend so much time persuading other folks to cooperate with you on solving the problem, or even persuading them that it can happen, because you only spring into action once it has happened.
You don't have to worry that your hard work will go unappreciated. Your boss will probably arrive at your desk at some time, and maybe his boss too if you can't fix the problem quickly. They may offer you a lot of helpful suggestions on approaches that you could take, and perhaps a few precautions that you could and should have taken beforehand. They may also suggest a lot of interesting new career moves that you could make if things don't work out, quickly.

Hindsight Planning

Hindsight planning is popularly carried out by folk between about 1AM and 4AM in a darkened room, staring at the ceiling. There are no distractions, so you can really focus on the job and do it right.

The big advantage of hindsight planning is the most accurate and economic way of getting the job done. With the benefit of hindsight, you know exactly what backups you should have taken when, and exactly what recovery procedures you should have undertaken when things went wrong. Next time, you'll know exactly how to handle the situation. If there is a next time.

The big drawback of hindsight planning is that you may never get an opportunity to put your hard-won wisdom into practice. You may be following a new career, one of those recommended to you by your manager.

Which Planning Strategy is Right for You?

We all know one size doesn't really fit all, so which of the three strategies is right for you? Probably a mixture of all three. There is a limit to the patience and cooperation that you can expect from your co-workers, so if you go flat out for the foresight model, you will irritate your colleagues. They will warn you off, and if you persist and really get up their noses, they'll go out of their way to sabotage you.

The on-demand model is risky, but also offers rich returns if you pull it off. Everyone knows that you pulled it off, and they will all think that you're a hero. Well, most of them. There's always a few malcontents. Because of its high risk elements, you may want to mix in a large element of the foresight method. Prepare for the problem, but when it happens, be sure to spend some time telling everyone how unlikely it was that this problem should ever occur. Let the tension build, and once you have your audience, tell them that there is one maneuver that might just save their bacon, then pull your ace from the bottom of the pack.

No matter how smart you are, there are going to be times when something slips through the gaps, and you will find yourself in hindsight mode. Providing you have more successes than failures, you will likely get away with it. Your management will know that no one else is going to achieve a 100% success rate, especially no one who's prepared to work for peanuts like you earn. Cover your back in advance by always asking (in writing) for a few more backups that you actually need, and keep track of the ones that don't get taken. Don't complain about the missed backups, just keep a note of them in your back pocket. Then when you find that you have totally run out of options and your back is to the wall, announce that you can save the situation by using one of the backups that you know isn't there. With horror, and an audience, you discover that it isn't there, and you pass the buck to some other poor sucker. Just don't pull this stunt too often, or your colleagues will get wise to you, and sabotage you out of a job in self defense.

Murphy Revisited

So getting back to Murphy, just why was he an optimist, and what does it have to do with you? Look carefully at his law:

Anything that can go wrong, will go wrong.

There's an unspoken assumption in this statement – that if something can't go wrong, it won't. This turns out to be a really dangerous assumption, as anyone who has scars on their back from years of backup and attempted recovery will know. I would like to offer you my rule, that is particularly relevant for folk who are responsible for planning and executing backup and recovery:

The most difficult problems to recover from are those that you knew could never happen.

There are two reasons why this is true.

Firstly, when they do happen, you simply can't bring yourself to accept that they really did happen You waste precious minutes or hours staring at the forensic evidence in stunned disbelief. While you do, the few precious opportunities that you might have had to escape from your doom slip by, unnoticed

Secondly, once you are able to accept the fact that the unthinkable has happened, you have absolutely no plan in mind to recover from it. You sit there staring at a blank page, with a ring of anxious faces surrounding you, looking at you with gradually fading hope.

The moral of this story is:

Always Have a Plan

Think through everything that could go wrong in advance. Brainstorm with other people and get their ideas too. If they're too busy and preoccupied with their own problems to help out, invest in a few drinks after work some time and, once they have started to chill, entertain them with a few horror stories about things that went wrong at other places. Chuckle with them over other people's misfortunes. Don't tell them that in some of these stories, you were the fall guy – keep it light, keep them laughing. After a while their creative juices will start to flow, and they'll reward you with a few horror scenarios of their own. Take discreet notes.

Once you have an inventory of potential problems, categorize them by probability and impact, as best you can. Remember, probability is never zero. If you don't believe me, ask Heisenberg. Once you have the list, start working on ways in which you could potentially recover should a given problem arise. You will not be able to develop a 100% recovery strategy for 100% of the problems, life is too short, so you have to prioritize. Develop detailed plans for the more likely, bigger impact disasters on your list, and less detailed plans for the less likely or less impactful items on your list. Your goal is not perfection, it's survival. For the raft of problems that seem too bizarre to happen, you can develop one simple, common solution – like keeping $2,000 and your passport in a plain brown envelope in your bottom drawer.

How Many Backups Do You Need?

Just to pick up on one common problem that you are pretty sure to encounter – you can't do recovery without backups. The more problem scenarios you develop and solutions you create to handle them, the more backups you are likely to need.

Sooner or later a delegation will come to your desk and tell you that you're taking too many backups, burning too much valuable removable media and/or bandwidth to backup sites, and too much processor time taking backups. They may point out for example that you still take daily backups of a database that is no longer needed, because the company sold off the division that used that database two years ago. Don't admit that this small fact had slipped your notice. You need to do some homework in advance. Find out the major laws that apply to the retention of financial records in your country, like the Sarbanes-Oxley act in the USA for example, and quote some of the penalties that may be imposed if these laws are broken. Point out that you wouldn't be backing up the data if it wasn't still online, occupying valuable disk space that is denied to you, and suggest that some guy who used to work with the system, but who has since left the company (and ideally the country too) once told you that other systems still in operation post updates to the supposedly discontinued system, and these have to be journalled to comply with record retention legislation. By now your accusers will be shuffling their feet and glancing sideways at one another. At this stage, throw them a fig-leaf so they can beat a hasty retreat from your desk without feeling that they have come away completely empty-handed. Offer to take the backups only once a week instead of daily.

You must always take complaints of this sort seriously, and make time to have a detailed discussion with your accuser about the various backups that you are taking, why you're taking them (better keep notes, or you'll forget yourself), and whether the need is still as great as it seemed when the backup was first put into the roster.

But as you sit down to have this conversation, be sure to kick off an extra, unscheduled backup to cover the time that you will be busy in the meeting and unable to monitor the systems. Just in case.

Methodologies considered risable

2009-05-04T10:39:00.000+02:00

A recent blog on The Agile Fallacy got me thinking. I've been programming for 45 years and still am, although slowly. I have seen the wheel go round many times, and I have a somewhat different perspective to that expressed in the blog. I won't mention any specific technologies because that will only yank various folk's chains. In the 1960's most big development projects delivered late and lousy. It became a standing joke that programmers used all of the project time and budget to deliver the first 90% of the agreed deliverable, and an equal amount of time and money to deliver the other 90%, assuming they were allowed to carry on that long. Peter Jackson famously remarked that movies don't actually get "released", they escape into the wild, and the same can be said for most large application development efforts. Whatever isn't done right at the time of release is done later, and called maintenance. The programmer's ultimate refuge is to claim "the users' requirements changed", even if the requirements were to implement double-entry bookkeeping which has been around in its present form for about 800 years. I rambled on about this issue in an earlier blog entry.

Fred Brooks, manager of IBM's OS/360 development project, wrote The Mythical Man Month to describe many of the pitfalls found in the 1960's. A lot of DP managers (that's what we used to call them) got fired as a result of non-delivery, and in self-defense they developed "methodology", elaborate systems of checks and balances, reports and reviews, that slowed the whole development process down to a point where there were far fewer ugly surprises. You can tell that "methodology" was a con game from the start by its name – "methodology" should mean the study of methods, not the practice thereof. It should have been called just "methods", but that was far to simple a word for the grand collusion then underway. With the introduction of "methodology" productivity plummeted, but the DP managers got to keep their jobs, and that's what counted most (to them).

The underlying problem with predicting how long development will take is that the productivity of individual programmers varies hugely. A research project at Dartmouth College back in the 1960's showed that the slowest student programmers were over 100 times slower than the fastest. The only reason that the study showed "only" a factor of 100 is that the researchers got tired of waiting for the last few laggards, closed the lab and went home. Sadly, you can't tell which programmers are fast and which are slow just by looking that them, so it's hard to pick a winning team. Ornate methodologies overlay so many admin tasks on all programmers that the difference in productivity between the best and the worst is masked.

While DP managers were happy with "methodologies", the really competent programmers were totally frustrated with them because they knew that they were being hobbled. As they say, "how can you fly like an eagle when you are surrounded by turkeys?" Some of the more cunning ones solved the problem by developing revolutionary new "methodologies" that allowed them to deliver running code dozens of times more quickly than the old, high-bulk "methodologies". On inspection, these new "methodologies" succeeded by jettisoning most of the padding that had been packed into the old ones, thus allowing the good programmers to get the job done fast. In order to allow them to use these new techniques, the practitioners had to dress them up with fancy new names, buzzwords, smoke, and mirrors.

Things went well for a while with the competent programmers delivering code at speed, but after a while the rest of the industry demanded the right to catch up. Snake oil salesmen were quick to "analyse and to codify" the new fast-track methodologies, and to come up with books and courses that would allow any pedestrian programmer to fly like an eagle. Or so they said. Institutions all over the world tried the new panacea, and found out the hard way that they didn't work for them. The new paradigms became discredited, but the fast programmers now had the bit between their teeth, and they realized that the way forward was to coin a new name for low-bulk programming every time the previous name got discredited. To date we have seen a number of iterations of this particular wheel, with the smart programmers soaring like eagles while the rest of the pack follow behind, crashing through the undergrowth and gobbling wildly as they struggle to become airborne.

So what should we do about this problem? The reality is, we can't afford to do without the really competent programmers, and if the new methodology game is the price that we have to pay in order to allow them to get the important stuff done fast, so be it. Just be aware that your mileage may vary.

Five Fingered Keyboards (3)

2008-12-03T19:58:00.003+02:00

I have at long last put together a small demo of a five finger keyboard, as discussed in a previous entry in this blog. I have released it as an open source project on Google Code, here. It's an HTML page with embedded JavaScript that illustrates the concept of a sequence-sensitive chording keyboard. The version that I uploaded, v0.1, is limited and buggy, but it's a beginning.

DIV_SRC could cut news feeds' bandwidth by 90%

2008-11-05T09:49:00.002+02:00

The Internet is taking over from printed newspapers, magazines and journals. It is also enabling ordinary people like me to publish their thoughts, hopes and ideas in blogs such as this one, almost free of charge. Major news feed sites such as as BBC News, CNN and Slashdot.org have millions of readers word-wide, and the amount of bandwidth consumed by people browsing news feeds is large and growing fast.

News feeds change their front pages often, and for this reason they make them uncacheable, or if they can be cached then it's only for a minute or so. Pretty much every time a person browses a news feed, they get a complete copy of the page from one of the news feed's servers. But if you look at a news feed you will find that it is made up of a number of short stories, often with links to fuller versions of the story. As time goes by, new stories are added and the older ones get pushed down the page, eventually falling off the end into an archive. If you look at a given news feed twice in ten minutes then the chances are good that while the content may have moved around a bit, most of it will be unchanged. And while you probably wouldn't look at the same news feed that frequently, if you consider all the people who share your ISP, and hence your ISP's caching proxy server, the ISP may have to fetch a given news feed page in its entirety thousands of times a day, even if 90% of it is unchanged from one fetch to the next. This places a large and growing load on the news feed's servers and bandwidth, and on the ISP's upstream bandwidth.

I have written an open source program called DIV_SRC, the package can be downloaded from Google Code. The tool allows large, composite HTML pages like news feeds to be broken into sections. The base page references these sections with <div src="URI"> tags (or any other tag types that the caller chooses). DIV_SRC.js is a client-side JavaScript program that finds all of the section references of this sort in the base page and fetches the referenced content using AJAX. A demo news feed shows that if news feeds used DIV_SRC then their main pages, which are necessarily dynamic, would shrink to about 5% of their current size. The individual stories that they carry could then be cached by browsers and caching proxy servers, reducing the news feeds' bandwidth bill by 90% and giving better response times to users.

No server-side coding is required to implement DIV_SRC, it can run off a vanilla web server that delivers nothing but the usual static files of HTML, images, style sheets and JavaScript. The DIV_SRC.js JavaScript code is compact, less than 5KB in terse form.

DIV_SRC could also be used by blogging engines to reduce bandwidth consumption and to improve response times. Blog pages delivered from blog engines such as Blogspot.com come with the HTTP header Cache-control: max-age=0, which prevents browsers and caching proxy servers from caching blog pages. This ensures that every viewer always gets to see the latest version of the page, which is good, but at a cost. If the blog is popular then browsers all over the Internet will be getting fresh copies from the blog server every time the blog is viewed. By using DIV_SRC, blogging engines could ensure that browser users continue to see the latest version of each blog, but the unchanged sections of the blog will not have to be transmitted from the blog server to the browser every time a given blog is viewed. Each blog entry would be stored as a separate static HTML file which could be cached by browsers and/or caching proxy servers anywhere in the Internet.

There are some issues that would have to be addressed, like what happens if a browser without AJAX capabilities (such as a mobile phone browser) accesses a composite page. This question and several others are dealt with in the Problems page of the project's website.

Cattle shown to align north-south

2008-09-07T05:49:00.001+02:00

A recent BBC article reported that a group of scientists had studied a large number of Google map images looking for cattle, and had found that most of them adopt a north-south orientation. The resolution of the images wasn't good enough for the researchers to distinguish which way the animals were pointing, north or south. They followed up with field studies of wild deer in the Czech Republic and determined that most of them faced north while about one third faced south. They checked, and the animals didn't track the sun as it moved across the sky, their orientation was consistently north-south regardless of the time of day. Their conclusion was that the animals possess a magnetic sense, and that they line up with the earth's magnetic field. We believe that migratory birds can sense the earth's magnetic field and use it for navigation but no organ has yet been found in large mammals that confers the ability to discern magnetic fields, but perhaps one will be found one day. The researchers do not offer any compelling explanation for why the animals would benefit from lining up with the earth's magnetic field.

It seems to me that there is another, simpler explanation for this behaviour. Each hemisphere of the earth undergoes seasonal variations - summer, autumn, winter, and spring. During winters sunlight is diminished and the growth of vegetation slows down. Animals deplete the available grazing, and in their natural state when unencumbered by fences they migrate in an equatorial direction to reach pastures where the daylight hours are longer and the vegetation more plentiful. During the winter they deplete much of the available vegetation, and when spring and summer returns, they reverse the process, migrating towards the poles to exploit the new vegetation that grows in response to lengthening days. So their orientation may simply reflect the general direction that they're migrating in when observed. If so, one would expect to see a strong correlation between the local seasons and their orientation - they should be pointing predominantly towards the poles in spring and summer, and predominantly towards the equator during autumn and winter.

It is not necessary to suppose that the animals have magnetic senses in order to point north or south. Bees navigate by the orientation of the sun in the sky, automatically making allowance for the apparent movement of the sun across the sky. If they can manage this calculation with their small brains then mammals should be able to do so as well.

Another Colour Riddle

2008-01-28T04:33:00.000+02:00

Just to complement my earlier colour riddle concerning pale people in Europe (link), here's another one, this time set in Africa.

A poacher cuts through the fence of a game reserve at dusk and enters. He travels one mile south, looking for rhino tracks. He sees a few tracks that run down a game path, and follows them one mile west. There the tracks split up, and the poacher must choose which set to follow. He follows one that turns north, and after a while he finds a pile of rhino dung. He picks through it with a twig, noting the grassy content. He tracks the rhino further north, completing a mile from where the tracks diverged, and sees the rhino. He closes in on it quietly, mows it down with an AK47, cuts off its horn, and heads back to his point of entry, one mile to the east.

What is the colour of the rhino?

The answer is "white", but there's a twist in the tail (or tale, if you prefer). In this riddle, the carefully reported distances are irrelevant. The key fact is that the rhino's dung contained grass. White rhinos eat grass. They have long, downward-sloping heads that position their muzzles close to the ground, handily placed for grazing grass. Black rhinos on the other hand eat trees. They have short necks and heads, and are well able to crane their muzzles upwards to reach the twigs on which they browse. The diet of the rhinos can easily be determined from their dung.

Why did the poacher care if he was following a white rhino rather than a black one? White rhinos are about 50% heavier than black rhinos, and generally have bigger horns. The poacher gets more money for a heavier horn.

And the twist in the tail? "White" rhinos aren't actually white, they're a dull gray, much the same colour as the "black" rhino. It is believed that the white rhino was originally named "weit" (wide) by German settlers in East Africa. The white rhino's muzzle is broad and squared-off, which helps it in grazing grass. The black rhino on the other hand has a more pointy muzzle with a long, prehensile upper lip that it uses to grab twigs and direct them into its mouth.

And the moral of the story? Get to see some rhinos (see link) before they're all extinct.

Desktop, schmesktop meets Android

2007-11-14T05:53:00.000+02:00

Seven years ago the IT tabloids were full of the question, "when will Linux take over the desktop?" I wrote in Linux Today:

Why the desktop fixation? Soon desktops will be history. The concept that you might need to go to a machine on a special desk in a particular room of your house whenever you need computer power will be as obsolete as the notion of going to the handpump over the well in the yard any time you need water. Houses have been "wired" for water and electricity for 100 years. It's everywhere you need it, just turn the tap or flick the switch. The same will happen with computer power this decade. Significant appliances will have all the computer power they need built-in, and connect to the web to make themselves accessible. people will wear whatever computer power they need while they're on the move. Linux is already shaping well in this world of "imbedded logic".
When news of Alexander Bell's wonderful new invention, the telephone, was telegraphed around the world over 100 years ago, the mayor of Chicago was particularly enthusiastic. He said that he could foresee the day when every city would have one of these devices. As it turned out, telephones played in a different market to telegraph offices. And Linux will play in a different market to desktops.

On the surface, things haven't changed that much. The IT tabloids are still pounding the same question. It is becoming less relevant with every passing day.

By comparison, when last did you last hear someone ask:

"When will Windows take over from MVS/zOS on the mainframe?"

The answer is, never. zOS is doing just fine on the mainframe, but the mainframe has become a niche market. A pretty capacious niche too be sure; IBM sells more mainframe MIPs than ever before, but the market place has moved on. There are far more MIPS on desktops than there ever were in data centers.

The turn of the desktop will come too. It has served us well, and old-timers like me will remember it with fond affection for many years to come, but we will have our work cut out trying to explain to the next generation why anyone would bother to go to a clunky box on a desk in a particular room room whenever they needed computer power. That's just about as absurd as suggesting that whenever you want to make a phone call, you have to go to a clunky black Bakelite contraption attached to a long wire in a particular room. Actually, I still have one of those, but it isn't Bakelite. Maybe I could find a retro model in a novelty store? Nowadays most folk, even in developing nations, carry slim portable wireless phones around with them. There are more mobile phones than fixed line phones in the world today, and mobile adoption is accelerating. They are typically powered with 200MHz or better processors. That's 5 times more processor power than NASA used to put the first men on the Moon in 1969 (they used three IBM System/370 model 168 systems, each sporting 12.5MHz processors).

So great, we have access to these powerful portable processors, and each has instant access to the Internet. But what can we do with them? Well, make calls to be sure. But modern cell phones can do a lot more than just that. The biggest inhibitor to their full exploitation is, as is ever the case with computers, the effort required to develop a full software stack, from the kernel all the way up to the glitzy user interface.

Until now.

Google has changed all of that. They have announced and made available a full open source stack for mobile phones. It covers everything from the bare metal upwards, and it's extensible. It's called Android. Check the link for an overview of what's in it, plus some videos. The cost of entry for would-be mobile phone manufacturers has suddenly plummeted. PC sales have already started declining in the East, where gizmo adoption rates are high. Stand by for descent, the West. And stop worrying about when Linux will take over the desktop:

It never will
No one will notice, or care (except for the Smithsonian)

Colour Riddles

2007-10-26T01:29:00.000+02:00

There's an old riddle that runs like this:

A hunter sets off from his base camp. He travels one mile south, sees bear tracks, follows them for one mile east, finds and kills the bear. He skins it and carries the skin back to his camp, which is one mile to the north.

What is the colour of the bear's fur?

You are supposed to reason that if a trip of one mile south, one mile east, and then one mile north brings the hunter back to his base camp then he must have started from the North Pole, and so the bear must be a Polar bear, and hence its fur white. Of course Polar bears don't range that far north , but global warming is rapidly depleting the Arctic ice pack, and perhaps one day there will be Polar bears near the North Pole, searching for enough ice to stand on.

Nit-pickers have pointed out that you can also choose a starting point near the South Pole such that after you have travelled one mile further south, a trip of one mile east will take you in a complete circle round the South Pole (or two or more whole circuits if you started closer) back to where you started circling, and the one mile leg back north would then take you back to where you started. But there are no bears at all in the Antarctic, so that doesn't fit the riddle's premises. Perhaps, if the Arctic ice pack thaws out completely, we will transport some Polar bears to the Antarctic; it's solid ground near the pole, with a high enough altitude that it will stay cold far longer than the North Pole will.

I have devised another "colour" riddle for readers to ponder:

The scene is a valley in what is now western Germany, 10,000 years ago. An ice age has the northern hemisphere in its grip; although the season is late spring, snow still blankets the land. A curl of smoke issues from the mouth of a cave, which is largely closed off by hanging hides. A weak sun shines through thin cloud. A group of men straggle out of the cave. They wear thick furs; only their faces are exposed to the cold weather. They carry bows, arrows, and spears. They cast about, looking for animal tracks in the snow. Soon they find some deer tracks, and set off in pursuit of their prey. Almost all the men walk with some difficulty; their legs are badly bowed, they walk with bandy gaits. But the leader of the group has straight legs, and walks easily and quickly. After a while they close in on their prey carefully; they unleash a volley of arrows, and the deer is fatally wounded. It swiftly bounds away, but quite soon weakens and slows. The group leader runs fast and closes in on the deer. He uses his spear to kill it. The rest of the group join him. They exult in their success. They quickly butcher the deer into sections with their sharp flint knives so that they can transport the meat back to their cave. The leader has done more than the others to secure their quarry, and so he takes a larger share than the others. He and his family will eat well.

What is the colour of the leader's eyes?

Rickets is a human disease that causes softening of the bones in children, resulting in bowed legs in severe cases. It is caused primarily by a vitamin D deficiency. Our bodies will create all the vitamin D that they need if exposed to sunlight. Early humans who had to live through ice ages spent much of their time in the shelter of caves, and they wore comprehensive fur clothing to keep warm. Archaeological remains tell us that many of them suffered from rickets, often severely.

What has this got to do with the colour of the leader's eyes, you may ask? I am blessed (or cursed) with a fair complexion – a pale pink skin, blue eyes, blond hair when young that darkened somewhat with age, before it fell out. There are hundreds of millions of other humans with my general colouration; we are called Caucasians, which tells you something of our origin, the Caucasus in Russia. While we are many, we are greatly outnumbered by other varieties of human, almost all of which have darker skins, dark hair, and brown eyes.

As a fair person living in Africa, I am vividly aware of how poorly equipped I am to cope with the hot sun. The melanin in my skin, which should protect the deeper layers of skin from the sun's ultraviolet radiation, is dysfunctional; it is clumped into small circular disks called "freckles", and all of the skin between is unprotected. I have often felt the sharp sting of sunburn, and had to deal with the outer layers of my skin peeling off. My blue irises do a poor job of keeping strong sunlight out of my eyes. When the light is bright, much gets through my blue irises, suffusing the scene with glare that washes out the details that I need to see. Hair on heads may benefit us in many different ways; the way that I miss most is to protect my pate from sunburn. Dark hair does a better job of this than does blond hair.

Given my dysfunctional complexion, I have often wondered why it ever persisted when by chance it first arose amongst my forebears. They didn't have the benefit of sun tan lotions, lip balms, dark glasses, and fedora hats. How did they manage to out-compete their darker kin? The riddle that I posed above may give us an explanation. Pale people arose only in cold climes; there were ice ages that made these climes colder still; rickets was a major problem under these conditions; pale people absorb what little sunlight there is more than do darker people, and hence would suffer less from rickets.

Interestingly, it was recently determined that red hair was common amongst the Neanderthal people (see here). They too developed a form of albinism, as did the Caucasians. They too lived in Europe through a number of ice ages. They too suffered from rickets. So far as we can tell, Homo Sapiens did not descend from the Neanderthals, so this is probably an example of parallel evolution; a common solution to a common problem.

Dark Energy "explained"

2007-09-03T08:18:00.000+02:00

Until 1929 practically everyone thought that the stars in the skies are fixed in space, that the Universe was pretty much constant and unchanging. Then the astronomer Edwin Hubble published the results of 10 year's observation that showed that the Universe is expanding. Working backwards from his observations, it turns out that the Universe began about 12 billion years ago with an almighty explosion called the Big Bang. Stephen Hawking and Roger Penrose showed mathematically that it all started from an almost dimensionless point, and has been expanding ever since.

The big question was, would it keep expanding forever, or would the force of gravity eventually suck it all back into a dimensionless dot again?

Then in 1998 more precise measurements showed that the Universe is expanding at an ever-accelerating rate. This was a big surprise, because no one knew of any force that could cause this acceleration. It was soon called Dark Energy. Since then, theoretical physicists have had a field day speculating about what Dark Energy might be, and where it came from.

Not to be left out of the fun, I have developed my own theories. They start out with some rather unimportant and boring thoughts about the behaviour of ionized gases and free electrons in the Earth's atmospheres, and that of other planets, and the atmospheres of the sun and other stars. Then they move onto more exotic settings, like the accretion disks around neutron stars and black holes. Up to 10% of the matter falling into a black hole is converted into energy through friction, releasing huge amounts of electromagnetic radiation. This would ionise much of the gas in the accretion disk, freeing electrons. The equipartition of energy principle, an outworking of the second law of thermodynamics, dictates that those electrons that remain free will attain the same mean temperature (i.e. kinetic energy) as has the surrounding gas. Since protons weigh 1836 times more than do electrons, the free electrons will gain much higher velocities than the surrounding gas. Many will achieve escape velocity, and leave the environs of the black hole, while the heavier protons will fall into the black hole and in due course into the singularity at its centre.

In the singularity at the centre of the black hole, the curvature of space, density of matter, and dilation of time are infinite. Cause and effect break down, since it would take an infinite amount of time for a cause to have an effect. It is reasonable to hypothesize that the net positive charge that enters the singularity will become sequestered, and its charge will be completely invisible to the surrounding cosmos, leaving a net and growing surplus of negative charge in the form of the free electrons still rattling around outside. These would flee the environs of the black hole where an apparent excess negative charge holds sway, eventually percolating out of the containing galaxy, but their flight would be impeded by the gas, dust, and magnetic fields within the galaxy. This flight would take a long time, especially for larger galaxies. Those galaxies with central black holes would therefore have an apparent net negative charge; the larger the galaxy, the larger the charge.

If the free electron hypothesis is true then it may account for the Dark Energy that is held to be responsible for the accelerating expansion of the cosmos. The growing numbers of free electrons will repel one another, and this repulsive force is much stronger than the force of gravity, pound for pound.

If all big galaxies have big black holes at their centres, helping to give them gravitational cohesion, then many of the earliest massive galaxies must have been largely consumed by their central black holes by now. If the free electron hypothesis is true then these galaxies would have released prodigious numbers of free electrons by now. These would expand like gigantic, invisible bubbles in the fabric of the cosmos, repelling other (negatively charged) galaxies from their vicinity, and perhaps giving rise to the observed "foam-like" structure of the cosmos. Observation shows that the cosmos is mainly composed of vast empty bubbles of space, with gas, dust, and the visible galaxies clumped into skeins and membranes between these voids. The BBC recently reported the discovery of a huge void in the cosmos.

You can find all of these wild conjectures, and several more, in a paper that I put online recently at http://turton.co.za/pubs/electrongas4.html

I have also started another blog just to talk about this topic. It's kind of esoteric.

My blog got mothballed!

2007-09-03T08:09:00.001+02:00

Hello, World. Again.

My blog is back from the near-dead. It was mothballed by a Blogspot bot, trawling for spam links in blogs. There have been a rash of spam links in blogs, sucking gullible viewers to sites that plant Trojans on the PCs – at least those that run vulnerable operating systems. Here's a link (dare I do this?) to a BBC article that talks about the problem: http://news.bbc.co.uk/1/hi/technology/6970368.stm

Anyhow, I'm back. Sorry for the short interruption in service.

Five Finger Keyboards (2)

2007-07-26T02:17:00.000+02:00

So "chording keyboards" are what I "invented" in my blog entry on five finger keyboards. My thanks to all you folk who responded with so many excellent comments. I exposed my ignorance, you diluted it with great feedback. It looks like simple binary encoding is the norm for chording keyboards, giving only 31 combinations with 5 keys. Sequenced chords give 324, and Alex is already exploring that idea. Doubly sequenced chording (the order that you lift your fingers counts) and extended sequences didn't ring any bells, so maybe that small part of my idea was novel.

Encouraged by your response, I have decided to share some of the more trivial details that occurred to me on this topic. Hopefully someone will build one of these gizmos and try them out, it's the only way to tell how well they will work.

Balance

Commenters pointed out that it could be hard to keep the mobile phone steady if you need to press all the finger buttons on one side, but not the thumb button on the other side. One edge of the phone could rest against the base of the thumb, which would help. If any buttons got clicked by the thumb base, they would be ignored since they would not have been enrolled. The two buttons top on either side of the phone could be replaced by a rocker switch for the thumb tip to operate. Many phones already have a rocker switch to control volume. The rocker switch would allow the thumb to express three values (up, down, off) instead of just two (on, off), considerably increasing our code space of 324 combinations. If all 4 fingers are pressing on one edge, the thumb tip could press evenly on both halves of the rocker switch so that it doesn't rock up or down.

Similarly, the thumb might need to press its switch while all 4 fingers don't press their buttons, leading to an imbalance. The finger buttons could be recessed in small saucer-shaped depressions. When the finger tips are straight, they would exert pressure on the rear edges of these depressions rather than the buttons, and could thus counter-balance the pressure of the thumb. When users need to press the finger buttons, they would flex their fingertips inwards.

Repeating keys

There are some buttons, like space, tab, and the cursor movement keys, that we need often need to press repeatedly. If we had to go through a multi-finger sequence each time, we would lose patience very quickly. We could cheat our way round this limitation in the following ways:

Suppose I must key 5 and 1 (thumb) to get to the cursor movement keys. Once I do, the soft labels next to fingers 2 through 5 all change to the familiar arrow up, down, left and right symbols. I keep key 1 pressed (that's why I assign it to the thumb, it's strong) and release finger 5, its selection job is done. I can then press and release any of keys 2 through 5 any number of times, keeping 1 down, and I get the corresponding cursor movement each time. If I press and hold any cursor key, I start to get automatic repeats, maybe 2 per second to start, speeding up to 8 per second if I persist. Once I release key 1, the soft labels next to keys 2 through 5 revert to their initial values.
Keys 4 and 1 might give me the 4 tab keys (up, down, left, right). What, you didn't know about tab up and tab down? You must be using a keyboard with a small number of combinations!
Similarly, any terminal key (i.e. one that finally selects a character) could give automatic repeats if held down long enough, maybe one second. This delay would have to be customisable by the user, since while we're learning how to use these things we'll be slow. Or maybe the phone could measure the speed with which we press key combos and adapt automatically. Why not, it has nothing better to do while waiting for keyboard input.

Remembering key sequences

We'll probably end up with learned committees meeting in Geneva to argue about which key sequences should lead to which characters. As qwerty as shown, first horse past the post gets the prize. We need to think about this problem before some arbitrary and clunky scheme gets adopted by default. Here are some rough ideas. I hope that they will provoke lots of thought and discussion, and get improved out of sight.

We need to group keys into related sets so that people find it easy to remember where they are in our code space. The alphabet and numerals are easy because they have a natural sequence. Accented alphas like à,á,â,ã,ä, and å could be located just a bit deeper in the code tree than their unaccented counterparts. Once you reach a, for example, the accented variations could appear on other keys. Since you could use up a few keys and fingers reaching the letter you want, we could adopt the same cheat that we did for the cursor keys; once you reach the bare letter, you only have to keep the last key that you clicked down, and the others keys all become available to select accented variations. If you release the last clicked key without clicking any other key after depressing it, you get the bare letter.

Here are some other possible logical groupings of special characters:

Punctuation: . , ; : ? ! " '
Arithmetic: + - * / ^ ( ) = != < > =< >= !
Brackets: [ ] { } ( ) < >

Yes, some characters like ! and ( ) appear in more than one logical group. Why not? We have a nice big code space, unlike those clunky old qwerty keyboards

Modal spaces

I have suggested a couple of times that you may arrive at a place in the code tree where you can release all the keys that you pressed, except for the last one, and all the other keys get new functions and new soft labels. The cursor movement keys could fall in one such space, and the tab keys another. Paging the display up / down / left / right is another obvious candidate. This mode is different from usual, where you keep all keys that you have pressed down till you get to your final destination. Having different modes could lead to confusion. At the very least, we could use clues like different soft label background colours to indicate the mode that each key is currently in, e.g.

a-i may denote a soft key that we have yet to click
a-i may denote a soft key that we have already clicked
á-å may denote a soft key that we have already clicked, but which now has a new function

Even today's mobile phones have input modes, so the users shouldn't find the concept too alarming. When you're capturing a new contact and you move the cursor to a phone number input field, the phone usually switches to numeric keyboard mode, while when you're entering a name, it's in alpha mode.

Happy imagineering, folks, we may be inventing some small part of the future. Which, as Alan Kay remarked, is a lot easier than trying to predict it.

Five Finger Keyboards

2007-07-23T02:15:00.000+02:00

In previous blog entries I have talked about mobile phones replacing desktop computers and the ways in which we could run applications on them. The biggest inhibitors that phones present are their dinky screens and keyboards. Screen resolutions are rapidly getting better, and iPhone has shown that if you can use the real estate that is usually hogged by buttons, you can get a good-sized screen onto a mobile phone. But without buttons, how can you enter text, or operate the phone's menus? iPhone offers you a virtual keyboard. When you need to enter text, an image of a keyboard pops up. You activate the keys by poking them with your fingertips. But there isn't much space for the keyboard. The keys are so close together that it's hard to touch one without touching others next to it. The iPhone makes it easier for you by using a form of predictive text, similar to what normal mobiles do; it checks the spelling of each word that you type, tries to work out what keys you meant to touch, and fixes the spelling on the fly. This approach works, but you will probably use only one fingertip to key in text, and you still need to aim quite carefully, so it isn't going to be fast. Some mobile devices like the Blackberry provide a full qwerty keyboard, but space limitations are such that you have to use a stylus, or point and aim with one finger very carefully, to use such a small keyboard. There is clearly still room for improvement.

Five Key Keyboards

Let's think of another approach - the five key keyboard, one for each finger and the thumb. Not an entirely new idea - there are about 2,000 Google hits for "five [or 5] key keyboards". They have been used to control a number of simple devices. But can we scale this up to the big time, to handle general text entry of the sort that we would usually use a full-size desktop keyboard to do? Let's try.

Simple Binary Combinations

A general desktop keyboard has about 120 keys, a lot more than the five we're looking at. But suppose we allow the user to press combinations of keys on our five key keyboard? We use multi-key combinations on desktop keyboards, after all: Shift-a for A, Ctrl-c to copy, Ctrl-Alt-Del to freshen up our desktop software. Five binary (on/off) keys allow for 2*2*2*2*2 = 32 unique combinations. We need to assign one of these combinations to the gaps between keys, that leaves 31 that we can use. Enough for the Latin alphabet and a few other letters, such as a numeric shift, Alt shift, Ctrl shift.

Sequenced Combinations

That's interesting, but can we do better? Yes we can, if we take the order in which the keys are depressed into account. Not such a strange idea, since Shift-a gives a different result to a-Shift, after all. That would give us:

5 + 5*4 + 5*4*3 + 5*4*3*2 + 5*4*3*2*1 = 325 combinations.

Even less one for the gap between characters, that's quite impressive, even more than the number of combinations that we can get on a desktop keyboard using two-key combinations (a, Shift-a, Alt-a, Ctrl-a for example).

Doubly Sequenced Combinations

Can we do better? Yes we can, because once we have depressed a sequence of keys, we need to release them, and if we take the order in which we release the keys into account then we have:

5 + 5*4*2*1 + 5*4*3*3*2*1 + 5*4*3*2*4*3*2*1 + 5*4*3*2*1*5*4*3*2*1 = 15,685 combinations.

Now we're cooking on gas! That's enough combinations to handle the characters commonly encountered in Kanji and traditional Chinese writing, all with just five keys.

Extended Sequenced Combinations

Can we do better? Yes we can, because so far we have only considered the cases where we press various keys in various sequences, and then release them in various sequences. What happens if we allow sequences in which keys may be pressed, released, pressed, released, and so forth, the only limitation being that at least one key must be pressed at any time, otherwise the sequence ends? Then the sky is the limit - you would have an arbitrarily large number of different sequences at your disposal. In fact this would hold true even if you have only two keys on the keyboard. You could press one, then the other (2 combinations) and then release and repress either one or the other, gaining one net bit each time, until you have chosen the character that you want. Interesting, but tedious.

In practice, 324 combinations would be enough to satisfy most needs and to confuse most users, so we can forget about the extended sequences for now. We will return to them when we discuss the needs of people with disabilities.

Arranging the Keys

How would we arrange five keys on a mobile phone? Simplistically, we could locate four down one edge for the fingertips to operate, and one on the opposite edge for the thumb to operate. But which edge? Left-handed folk might like the opposite edge to right-handed folk. And even right-handed folk will likely switch hands from time to time; like when their one thumb is cut and bandaged (as mine happens to be right now); or when they're actually listening to a call - right-handed folk are mostly left-eared, and tend to hold their mobiles in their left hands while they're talking. They might still want to be able to operate the phone in this position to do things like:

increase or decrease the earpiece volume
put the current call on hold and answer a second call
terminate a call
initiate a conference call

So in practice, we would be better off with a row of buttons down both edges of the mobile phone. The first time a particular person picks up a mobile, the mobile wouldn't know how it is being held. It could measure the electrical resistance between the buttons and try to work out which finger is on which button. But most likely the phone would have to go through a finger enrolment procedure. It could show a picture of a hand and then highlight each finger/thumb tip in sequence. The user would respond by depressing the button that happened to be under the finger/thumb in question. This would quickly sort out whether the phone was held in a right hand or a left hand. The user may well have more flesh than just finger/thumb tips in contact with the mobile's buttons. It's common to have the base of the thumb pressing against one edge of the mobile to stabilise it. The mobile could detect this by measuring the electrical resistance between buttons, even those that aren't normally depressed. Chance depressions of buttons that hadn't been enrolled would be ignored.

This Sounds Hard!

So technically, it's doable. But how easy would it be for a phone user to memorise up to 324 key depression combinations? Or even just about 100 if the user is happy with the Latin alphabet? Well, we can make it easier by using the concept of soft keys, which are already used by Java midlets when they run on mobile devices. Most modern mobiles have at least two keys just under the bottom edge of the screen with no labels permanently assigned to them. As you work through the mobile's menus, or run Java midlets, different labels are assigned to these keys. The labels appear on the screen just above their respective buttons. They may change as you move from one screen to another. Once the user has registered the buttons that they are going to use to operate the mobile, it can pop up little labels next to each of these buttons. As the user starts depressing buttons, the labels next to the remaining buttons (and even next to the depressed buttons) could change. Assuming that the user has registered a full five buttons, they might navigate through the following sequence of button depressions to choose a character. The sequence is shown left to right. In each panel, the number indicates the button next to which the label will appear, the buttons already depressed by the user in previous panels are shown in italics and the one that they choose to depress in that panel is shown in bold. If a menu option appears underlined then that option will be taken if the user releases all of the keys, completing the character selection process.

a-i
j-r
s-z
0-9
other...

a-i
b-d
e-g
h-i
other...

a-i
b-d
c
d
other...

In this sequence, the user chooses successively a-i, b-d, then c to get the single character "c". Three key depressions are required in the sequence 1, 2, then 3. The keys are then released, and the mobile ignores the release sequence.

With this scheme, the novice user is guided with a series of soft key popups as to which key to depress next. With practice, users will quickly learn the sequence of depressions that give them the characters that they need most often. Users won't have to move their fingers from one button to another as they do with conventional mobile devices (or even desktop keyboards). They should be able to enter text quicker than folk do on current mobile keypads. Maybe with practice we will even be able to match our desktop keyboard skills. It sounds unlikely, but bear in mind that 93 year old Gordon Hill, a former telegraph operator, beat 13 year old Brittany Devlin in a texting speed test where Gordon transmitted the test message verbatim using a Morse code key while Brittany texted the message through a mobile keyboard, using common texting shortcuts (see here). Hopefully we could get international standards on the key sequences that give us at least the commonly-used characters and controls, so that users can migrate from one manufacturer's mobile s to another with relative ease.

Oops!

One thing we all know is that as we key characters, we make mistakes. Thank heavens for the backspace and delete keys! So if we implement the five key keyboard, how will we allow users to correct mistakes? The character set should certainly include sequences for backspace and delete, and also cursor movement keys (up, down, left, right, tab, back-tab, home, end, and so forth). If you happen to be part-way through a long sequence and depress the wrong button, the mobile may allow you to let go that button and press another instead, ignoring your first choice, if it does not implement the extended sequence scheme described above.

People with Disabilities

People with various disabilities also need to be accommodated in our digital world. When a person picks up a particular mobile for the first time, they will have to go through an enrolment process so that the mobile can discover which fingers they have positioned on which key, as described above. If the user happened to be missing one or more fingers or a thumb, they would not respond when those particular digits were highlighted on the mobile's screen. The phone would discover this, and would use a different character encoding scheme that falls within the capabilities of its user. Hopefully we can develop international standards for less than five key sequences for the commonly used characters as well. If the disabled person can operate only a few keys, we could still define extended key sequences that will allow them to access all the characters that they need to operate a mobile.

Patents Pending?

I have done absolutely no searches to find out if any of the ideas described in this article have been patented or not. If you're interested in developing any solutions that incorporate some of these ideas, maybe you should take a snapshot of the page to defend yourself against patent claims in the future. If it appeared here before it reached the patent office, it's prior art, and can't be patented. Let's all do some "open source" invention to try to improve the way things work for one another rather than always trying to make money from one another.

Use Cases On Steroids

2007-06-21T00:29:00.000+02:00

Computer software development projects still often run late and over budget, and the people who commission them are still often surprised and disappointed by what they get at the end of the development process. Software development has been around for over 60 years now, and it should be a mature, reliable process, but some big gaps remain. I've been designing and writing software for over 40 years, and I have scars to prove that I blew it often enough myself. I have been trying for a long time to find a way to make the development process more visible and easy to understand for the people who will eventually use what we build, so they get advanced warning when we're going wrong, and can help us to sort out our mistakes before they get cast in code, because that's even worse than being cast in concrete.

This paper discusses the biggest problems that happen time and again in the software development process:

We don't fully understand the users' requirements up front.
The users don't really understand the design that we put together.
It's only when we deliver code to the user that we all find out how much trouble we're in.
The users don't have sufficiently detailed plans to test the system before it goes live.
When the system does go live, we get into even more trouble.

I show ways in which the humble use case can be extended to better solve these problems, and to improve the quality of the solutions that we deliver.

Why do things keep going wrong?

If we have been developing computer software for so long, and if we know so much about how it should be done that our universities offer graduate courses in computer science and software engineering, how come we keep getting it wrong? In his book "Great Software Debates", Alan M. Davis states that "Requirements" are "The Missing Piece of Software Development". Usually the people developing software are not experts in the business that they're trying to automate, and the people in the business, who know it backwards, are not experts in software development. Both groups use their own language to talk about their area of expertise. Neither group understands the other particularly well. Communication is poor. Both groups hope that the problem will go away while they're developing the system. It does, but too late. By then the damage is done!

"The user requirements have changed"

First, let me dispose of the oldest, lamest excuse in the industry (I should know, I have used it often enough): "The user requirements have changed". Maybe we have to build an accounting system, or an order entry system, or whatever. People have been doing these things for centuries. Double-entry bookkeeping, for example, goes back to 12th century Italy, and hasn't changed much since. The people doing it get trained in schools, colleges, universities, and then get extensive on-the-job training before they start to practise their trade. So we build a system to meet what we think their needs are. We have many meetings with them, where we talk computer jargon and they try to pretend they understand what we're saying. But the first time they really understand how little we know about their business is when they try out the system that we have built for them. Then, suddenly, they communicate, long and loud. But we are smart. We have been burned before, so we got them to sign a contract in advance that says if they want anything different from what we build them, they pay. We say, "The user requirements have changed". That's usually not true. It's their understanding of what we're doing to them that has changed, and, after a lot of loud shouting, our understanding of their business processes and needs changes too. But then we get deployed on a new project in a different business where we're just as green as before, and the cycle repeats itself.

How can we discover the real requirements?

Lots of good ways have been developed to help computer people to talk with business people to discover what it is they do, and what the proposed new software should do in order to help them get their job done. These are generally called "methodologies" by the people who develop them. As the Wikipedia article on this points out, it would be more accurate to call them "methods". There used to be a whole lot of competing methods around, each with their own jargon. Mercifully, most have converged on a common jargon called the Unified modelling Language together with common techniques and diagrams. The UML components that deal most directly with capturing and documenting user requirements are:

The Actors, who use a computer system or trigger actions within it.
Actions, which are the things that Actors and computer systems do together to achieve something useful.
Use Cases, which are short stories written in business language that describe what Actors do to perform Actions.
Use Case Diagrams, which are Use Cases written as diagrams.

In short, these UML elements deal with the question of who is going to do what, how, in order to get what done. This is the most important part of the project, because if we get it wrong, we will build the wrong solution. Many early methods used complex jargon and diagrams that made this part of the process very difficult for business people to understand and fully participate in. Long and hard experience has taught us the importance of keeping it simple, and current UML methods are good in this area. If we follow the UML guidelines, the users will generally be able to understand what's going on, participate fully, and we should end up with a set of use cases that an accurately describe what's needed.

So why does development still go wrong?

In my experience, there are two major reasons why software development often delivers the wrong solution (or the right solution to the wrong problem):

The user requirements aren't detailed enough to fully specify the target system
The development process is invisible to the users; they can't identify errors as they arise

It's tricky to get the right level of detail in use cases. Here's the problem, and it afflicts all aspects of UML, not just use cases: in the books and training materials, the methods are used to describe activities at a trivially simple level. Here for example is a use case diagram that says that a patron orders food from a waiter in a restaurant; that the chef cooks it; that the waiter delivers it together with a drink; and that the patron pays for it. A nice, simple view of how things work. But we all know that reality is a lot more complex. The patron may have to be shown a seat, the waiter may want to talk about the specials, the patron may want to ask how various dishes are prepared. The chef has to plan the day's food, get appropriate recipes, obtain the required ingredients, peel, chop up, and otherwise prepare the food, cook various components in various ways, and combine the finished product on a plate. Eating itself is quite a procedure, which is maybe why it got left out of the use case diagram. Payment may be through cash, cheque, or credit or debit card, each of which have their own procedure. A credit card may be refused by a bank, and then the patron may pay with cash instead. And so forth.

If the use case is made detailed enough to be a useful and reliable guide to the programmer who needs to write the code, it will end up very big and bulky. It will take a lot of time and effort for the users to specify the use case at this level of detail, and they're "too busy". But this raises the ugly question, "if you're too busy to do it right, when will you find time to do it over?" When the project is planned, commitment must be made by suitably qualified users to spend the time needed to get the requirements right, and documented.

The bulk problem

The use case method, like most of the other methods in UML and all earlier modelling disciplines, were largely defined back in the days when modelling was done on large sheets of paper in a meeting room. Completed sheets of paper were stuck to the walls. Use cases with realistic levels of detail get so big and bulky that they won't fit onto a single page. They will spill over dozens, maybe hundereds of pages, and there's no easy way to navigate reliably from one page to another.

While we're on this subject, let's note that use case diagrams are much more bulky that plain text use cases. If you open a text editor and type in the text that appears in the action diagram referenced here, in the same sized font, you will find that the diagram occupies ten times more space than the text version. If real-life use cases don't fit on a single page in text form, they won't fit on ten pages in diagram form.

Drill-down detail

Nowadays programmers mainly use computer based modelling tools to create and edit the various diagrams that they use in support of the design effort. Some of these tools like ArgoUML are open source and free. They offer an elegant solution to the dilemma of simple versus detailed use cases. The tool user can specify a process as a set of simple steps, so that they fit on a single screen. Those steps that need more explanation can be given "children" steps, as many as are needed. The presence of these children can be signalled by prefixing the step with the familiar + icon that we see in file explorers. This tells us that there's more detail to see. If we click on the icon it becomes a – and its children appear below it. If any child action needs further explanation, we can repeat this procedure. Realistically, the requirements gathering phase needs to be supported by a tool like this. That way sufficient detail can be gathered over time without overwhelming the audience with unwanted detail. So requirements gathering should be based on a computer-based modelling tool, and the display projected onto a big screen so the participants can see what's happening.

The devil is in the details

To be really unambiguous for programmers, use cases would have to describe every single field of data to be captured on every single form or panel, and the validation to be carried out upon these inputs, and the complete list of all outputs that will be displayed on each form or panel. If this were done, the labour required to produce and maintain the use cases would be almost as much as that needed to write the programs that do the work. Programmers claim, and often with justice, that they hardly have enough time allocated to them to write the programs once, and that they don't have the time to produce really detailed use cases. If they did so, they would end up writing every program twice, in two very different formats. On the other hand, the end users who must help to develop and then validate the use cases would find it difficult to understand how the finished software would look and behave if all they see is textual use cases statements. But if use cases are not detailed down to the level of identifying the individual fields in each form or panel, they will be too high-level for the end users to assess their accuracy and relevance.

In my experience, the gap between use cases and running code is so big that users usually don't know enough about what is being developed to judge the proposed design before it has been turned into running code. By then it is too late to easily fix the problems that become visible.

Adding value to the use case

Perhaps the best way to make use cases detailed enough to keep the programmers honest, but at the same time meaningful to the users so that they can understand them, is to link each step in the use case to a separate mock-up panel which shows all the input fields required and all the output fields returned. Users can compare mock-up panels to the existing paper forms or legacy screen panels that they currently use to get the job done. They can compare the two field by field, and ensure that each field input into the paper form can be captured in the mock-up panel; or if not, ask why not. Creating the mock-up panel will not impose an unreasonable and irrelevant burden on the programmer providing each such panel is subsequently used as part of the system being developed. The mock-up panel can be refined through successive iterations to become the production panel.

In the case of a web-based software, which is the popular paradigm today, the mock-up panels can be developed as HTML pages, because ultimately this is what they will have to be. The use cases can also be developed as HTML documents. An index can be developed in HTML to list the use cases in the same hierarchical structure that is used in the UML model. Each mock-up HTML panel can be given a short, unique ID (this is ultimately required if users are going to have useful conversations with help desk personnel over telephones). An extra column can be appended to the use case scripts to carry the ID of the mock-up panel to which each paragraph of the use case script refers. Each such ID can be made a hyperlink to the mock-up panel, pointing to a different target window, so that when the reader of the use case clicks on a panel ID, it appears in a different window, and the user does not lose their place in the use case script. If a user has to perform a large number of use case validations, they could be given access to two physical screens, with the use case script window positioned on one screen and the target window that displays the mock-up panels on the other screen. With this sort of setup, the user can read the script in one window and swiftly see a mock-up of the panel that will be displayed when the function has been programmed. The user can then compare each mock-up to existing paper forms or legacy system screens, and ensure that it is complete and consistent. A given panel will often appear more than once in a given use case script, and across different use case scripts. The user will be able to validate the mock-up panel once in exhaustive detail when it first appears, and to devote less attention to the panel on each subsequent reappearance. This approach requires a lot less writing of tedious detail than would a use case that contains a field-by-field narrative for every panel every time it is referenced. Much less work for the person producing the use case, and similarly for the user who is validating the use case.

Here is a simple example of some steps in a use case that follow the method described above:

5.	The searcher enters search criteria that identify the documents of interest.	ggls01
6.	The system presents a list of the titles of documents that meet the search criteria, ordered with those that best meet the criteria first.	gglr01
7.	The searcher is able to click on the title of any of the documents listed to view its contents.	gglr02

In a functional computer system, values entered as inputs in one panel will often appear as outputs in subsequent panels. A mock-up interface built of separate static HTML files will not behave in this way. It is possible to get some of this behaviour without having to write specific logic for each mock-up panel. Each panel can at some stage in the development cycle be morphed from a static HTML page to an active page such as a JSP or PHP page, which it will ultimately have to be. This can be done by adding some fixed wrapper lines to the file and renaming it. The fixed wrapper lines can include logic to harvest all of the inputs captured by the user in prior HTML forms, and to store them in a hashmap in the user's session, without any regard to the names or values of the various fields. When the next page is presented, each output field can contain a method invocation that passes the name of the output field to a standard output method. This method could check if a value has been associated with the name passed, and if so return the value, else a question mark. The hashmap could be primed initially from a simple ascii file that contains name/value pairs, in order to prime the pump.

The UML community feel most comfortable with modelling when it is diagram-based. In order to gain their acceptance of mock-up panels, it may be best to give them a diagram-like name such as UIGrams.

Measuring the scope of work, and progress

One of the most vexing issues that face the owners of systems under development, and the developers of such systems, is the big disconnect between the specification of the system's requirements, in which the owners participate, and the production of a working, testable system. The owners have almost no way of knowing how much of the required work has been done, and whether the work is of adequate quality, until they see running code. Much the same dilemma may afflict the development project manager, unless he or she is an accomplished programmer as well as a project manager, a rare combination. By the time the code runs well enough to test, so much time and money have flowed that the owners may find themselves committed to using the final product even if it is not to their liking.

Many different approaches have been tried to provide system owners with an objective measurement of progress, which they and the development manager can use to check whether the project is on track. One of the earliest was to estimate the number of lines of program code that would be required to complete the project, and to count the number of lines coded on a regular basis. This turned out to be a poor metric for several reasons:

No one can accurately estimate how many lines of code will be required to complete a system, especially if other coders are involved.
Programmers typically write many lines of code quite quickly, but then spend a lot of time correcting errors, which may not result in much net growth in the number of lines of code.
Hard experience has shown that if programmers know that their progress is being measured in terms of the number of lines of code that they write, they will write more lines of code. They will tend to clone more sections of code rather than writing a function or subroutine or method which they invoke from different places. This proliferation of code will eventually make the system more difficult to fix and enhance once it has gone into production.

System developers do develop a lot of artefacts in the process of systems specification and development. These should ideally be captured into a formal computer-based development model. Many different models have been developed over the years, but fortunately there has been a convergence on a single group of standards called UML (Universal modelling Language, see http://www.uml.org/). Unfortunately, one has to be an expert in the various models available within UML in order to assess the scope, completeness, and quality of the models developed by the system developers, so this isn't of much use to system owners in determining progress.

Function Point Analysis (see http://www.ifpug.org/) was then developed, and turned out to provide a far more reliable measure of the work to be done. It's probably the best system that we have, but has the drawback that it requires a lot of work from both the developers and the system owners to determine in advance how many function points of what complexity a new system will entail, and to measure progress against plan. And as classically conducted, function point analysis tends to be all overhead in the sense that it does not contribute directly to the design, development, or testing of the system.

For an online interactive transaction-based business or administrative transaction processing system, it turns out that the function point metrics are largely determined by the number of panels and the number of input and output fields on the panels that the users will interact with. This would make no provision for batch, but batch programs typically constitute a small fraction of the overall system development effort. The amount of work required to develop a system (excluding batch) can therefore be based on a count of the number of panels, together with their input and output fields, that will be needed to implement the required functionality; and that progress should then be measured against the agreed set of panels. This is a more rough-and-ready approach than function point analysis, but it has the great advantage that it does not require either the developers of the system owners to do work that does not contribute directly to the final system. These are the steps that are required:

During the requirements gathering phase, developers and owners work together to identify the elements listed below. Developers will capture them into the agreed modelling tool, and the owners will be asked to verify that this has been done accurately (this part of the model is easy for non-specialists to check):

The actors that will play a role in the system.
A hierarchical list of the major actions that these actors will perform.
Models of each panel that will be required to support the actions identified, in HTML if it is a browser-based system.
Use cases that describe in words the various actions identified, in HTML if it is a browser-based system, with links to the model panels.
A data model that contains all of the data items identified in the actions and panels.

During the design development phase:

The users can work their way through the use cases, viewing the model panels at each step of the process, and validate or correct them.
The developers refine the data model into normal form, then produce a database design.
The designers populate the model with the classes, attributes, and methods that will be required to implement the system.

During the code development phase:

The developers flesh out the model classes with the code required to implement the system.
The developers create the database and the classes required to manage the data in them.
Simple, standard logic can be added to the model panels to propagate inputs entered by users onto subsequent panels.
Other team development members refine the look and feel of the model panels until the users are comfortable with them.
Snapshots of key panels are taken and are signed off by the system owners as being the look and feel that they require.
The development team ensure that the agreed look and feel is applied uniformly across all panels, preferable via style sheets.
The system owners test and sign off (or critique) the modified panels to assert that they have the required function and appearance.

During the testing phase:

As the various parts of the system are implemented, the corresponding model panels are fleshed out with embedded logic as required.
The use cases now become the test scripts. The users use them as their guide for testing the system methodically, but now panel-to-panel navigation is achieved through software logic in the test system rather than by clicking links in the use case (although use case links may still be used to navigate to the appropriate software panels where this makes sense, i.e. input from a prior panel is not required).
Navigation across those sections of the system that have not yet been developed may still be done via the use cases so that users can assess the components under test in a plausible context rather than in isolation.
A systematic colour scheme convention should be implemented through style sheets to distinguish model panels from working panels.

With this approach to the development process, the users start to get exposure to a "straw man" model of the final system during the system specification phase. The model system gradually evolves and improves throughout the design and development phases, and becomes real during the testing phase. Almost from day one, the users have something tangible and comprehensible to work with. They can see and assess how well the finished system will meet their needs, and provide meaningful feedback to the developers while development is still underway, rather than only after the project has supposedly completed and the developers have been assigned to different projects.

Dated snapshots of the model and developed system should be taken weekly and archived by both the developers and the system owners, so that when (not if) disputes arise as to what was previously done and agreed or not agreed, evidence will be available to help resolve the disputes.

Linking use cases to Java code and documentation

If the software development takes place in Java then a further refinement is possible – use cases can be cross-linked to the source code once written, and to the Javadoc once generated (Javadoc is a set of HTML documents that list all of the classes, and for each class all of its methods and attributes in a Java system).

Once the requirements gathering phase is complete, the text of the use cases is supposedly fixed. Analysts should then study the use cases and from them work out what objects are needed to represent the objects that appear in the use cases. The objects should correspond to the nouns that appear in the use case. The possessive form (e.g. the dog's bone) suggests that the class bone is an attribute of the class dog. Verbs should suggest the methods that the various objects (nouns) will need to implement. Adjectives qualify nouns, and may suggest subclasses.

It would be nice if the modelling tool allowed the developer to highlight nouns, verbs, and adjectives found in the use case, and to indicate which objects, attributes, methods, and subclasses they correspond to. The text of the use case could be colour-coded to show these classifications. As the analysis proceeds, the system could recognise nouns, adjectives, and verbs that the analyst has previously categorised, and offer the link previously made by the analyst as the default interpretation of the new occurrence of that word. The analyst could accept the default, or create a new object or method.

Once this analysis is completed, simple source code skeletons could be created automatically from it, and the use case linked to the source, so that clicking on a noun takes the viewer to the corresponding source class or attribute definition. Once the programmers have fleshed out the generated code stubs with working code, they will normally generate Javadoc documentation from it. The use cases could also link to the places within the generated Javadoc where the corresponding classes, method and attribute definitions are defined. Missing links (e.g. a noun that doesn't link to a class, or a verb that doesn't link to a method) would suggest areas of the use case that have not yet been fleshed out with source code, and hence parts of the software that require further attention.

Here is a simple example of how a marked-up step in a use case might appear. Classes have pink backgrounds, methods blue, both are underlined (would be hyperlinked in the real system), and tooltips may be added for extra information.

3. The quick brown fox jumps over the lazy dog.

References

Alan M. Davis. Great Software Debates (October 8, 2004), pp:125-128 Wiley-IEEE Computer Society Press

The Virtual Call Centre

2007-03-16T09:08:00.000+02:00

The rate at which electronics technology is improving remains amazing. Electronic products are getting smaller and smarter, and old products that never had electronics in them are getting infiltrated by electronics, or completely redefined as electronic appliances. The camera, for example. Given the dizzying rate at which old products improve and new ones emerge, the staff of the stores that sell this stuff have no chance of providing comprehensive after-sales service, as stores used to do in the good old days. Increasingly, the manufacturers of these gizmo's have to implement call centres in order to provide after-sales service to their customers. Call centres are springing up all over the place. Given the high cost of establishing them, many companies choose to set them up in second and third world countries where costs are lower. Thanks to constantly dropping communications costs, and new paradigms like Voice over IP, it's affordable to support US customers with a call centre in Ireland, India, or South Africa. And with a suitable geographic distribution, call centres can service customers any time that customers want service, at times that are also convenient for the call centre staff.

But building conventional call centres is still an expensive undertaking, even in third world countries. You have to rent a building, or buy land with a building, or build your own. You need aircon, heating and cooling, power, plumbing, parking lots. You need lots of furniture, desktop computers, and specialised telephone equipment. You need computer servers to handle and route incoming calls. You have to hire and train staff. Having made a big investment in infrastructure, you need to keep every desk manned for as many hours of the day as possible.

This model isn't all that convenient for call centre staff either. They are tied to their desks for 8 hours a day or longer. They must commute to the call centre before they can start work, and commute home afterwards. It may not be easy for them to pick up the kids after school, or to take the baby for her immunity shots, or the dog to the vet.

The Virtual Call Centre can change all of that. And the technology required to implement it is available right now. We would need some new software to make it work smoothly, but hey, that's a Simple Matter of Programming (SMOP - an acronym popular with computer salesmen).

Imagine we wanted to open a virtual call centre (VCC for short) for business. We would need only a server in a server farm, and another server in a different farm to act as a backup. The server would take incoming customer calls, perform the usual metrics, and then distribute them across the available call centre agents (CCAs for short). Oops! Where are our CCAs? We don't have a building. We advertise on the web for CCAs who can work from home. In order to act as a CCA, a person would need a desktop computer, telephone equipment of sorts, and a broadband connection to the Internet. Ideally, they should have these in their own home, but other arrangements could be made (the cottage call centre - but that's another blog). Their telephony equipment could consist of a headset with attached microphone that plugs into their desktop computer. It would link to the VCC's server through Voice over IP (VoIP). When the agent is ready to take calls, he or she would fire up on their desktop a program that they have downloaded from the VCC's website. This would allow them to log on and indicate that they're ready to take calls. The VCC's server would check what product types they're licensed to handle, and start routing calls to them. Since all calls pass through the VCC's servers, they can still be recorded, and monitored for quality (the monitoring staff could also be working from home). As soon as the call is finished, the VCC's server would know that the CCA is available for another call, and could route one through as needed.

Any time CCAs need to leave their workstations, they could click a Hold button on the VCC program on their desktop. If they need to go collect kids or run some other errand, they would terminate the VCC program and just walk away, any time they wanted to. The VCC spends nothing on accommodating staff, and can afford to have a larger number of CCAs so that between them they can take off time when they need to, or work when they want to. They would need to be paid per call handled as well as a retainer for just being online. The VCC client program could ask them a question after every minute of inactivity just to make sure that they're still there and ready to handle calls. With a beep, in case they're crusing the web while waiting. In fact the VCC could beep several candidate CCA's, and the first one to accept the call would get it, plus the bounty for handling it.

Of course to handle calls, CCAs need access to customer and product information. The VCC might handle products for many different manufacturers, and needs to deliver this product information to CCAs on demand. This problem isn't unique to VCCs, it applies to normal ones as well. CCAs could access the required product information through a browser connected to the VCC's server, which could in turn get the required information from the manufacturer's servers via SOAP or some such technology.

The cost savings that could be achieved by VCCs would be huge, but how would the VCCs enlist, train, and evaluate staff?

The VCC could invite potential staff members to enroll on their website by advertising on the Internet or through conventional media. The enrollment software could automatically check the network turnaround time and bandwidth between the enrollee's workstation and the server to see if the enrollee has a suitable broadband connection. If not, the server could ask the enrollee what location they are in, and hand the conversation off to a geographically closer server, if they have one. If the network connection isn't adequate then the enrollee could be informed of this, and invited to try again with better equipment. Even if an enrollee passes the connection test and gets approved as a CCA, each time they log on the client app would have to validate their network connection to make sure that it's adequate. Otherwise the CCA could qualify in a cyber café and then go home to their 14,400bps diallup modem

Enrollee skills assessment could be carried out over the Internet, with the enrollee browsing a multiple choice questionnaire. Language and product area skills would have to be assessed. Just to make sure that enrollees don't qualify by paying a friend to help them through this process, staff could be given short snap tests at random times whenever they are logged on, in between calls. Staff training could be handled in a similar fashion, with online web-based training interspersed with frequent online questionnaires. Staff could be shown on request what skills are in short supply so that they could plan their future training accordingly.

Quality control could be implemented as it is today in conventional call centres, by recording and optionally monitoring calls. When customers first call the VCC, they could be played a message telling them to hit the hash key if they want to break out of a call at any time - if, for example, the CCA gets abusive, or walks away in the middle of a conversation, or simply doesn't know anything about the product (which may happen if the CCA's kid sister is manning the station). Calling customers could also be invited to hit the hash key at the end of the call to give feedback on the quality of the service that they have received. Hitting the hash key would connect the customer to a voice response unit which would walk them through the various options such as providing feedback on a completed call, or complaining about the quality of service that they are receiving.

While I have used the word "staff" to describe the folk that act as CCAs, they need not be full time employees. They can be part-time folk working when it's convenient for them; mothers with a few hours available in the morning while their children are at schools, students with a few hours available in the evenings after lectures, or garrulous senior citizens like me, in between naps. Providing the workforce spans several suitable countries and continents, sufficient numbers will be available at the right times to meet the demands of customers calling in.

The Virtual Call Centre is just a specific example of a much bigger economic trend that I'm sure will start to emerge - the Virtual Office. I wrote about this shortly after the 9/11 incident. If we can get folk to stay at their homes, distributed across wide continents, instead of huddling together in vast high-rise buildings, or buses, or aircraft, we will remove many of the targets that terrorists love to hit. And at the same time liberate people from wasting what adds up to years of their lives commuting.

Mobilizing Mobiles

2007-03-03T00:19:00.000+02:00

Ah, the power of blog! You publish a simple idea, and back comes real gold. My thanks to eric and xiris for their comments, and for pointing out that much of what I "predicted" in my last post about mobiles replacing desktops has already happened! Nokia have already ported the Apache web server to run under the Symbian operating system which many of their phones use, and they have built "cgi scripts" that expose much of the phone's data (messages, address lists, images captured) to any browser that connects to the phone's IP address and (I guess) authenticates itself. So if you leave your phone at home by mistake, you can browse it from the office, check for messages and reply to them, check for missed calls, and so forth. Very clever! Of course this requires each phone to have a distinct IP address, and there aren't enough to go around, but IP Version 6 is gradually getting implemented and that will over time give us more than enough addresses.

Eric points out that much of what I "predicted" happening on mobiles is already available on PDAs such as the Nokia Tablets.

Xiris is concerned about the lack of storage space on mobiles. Storage size has been growing a lot faster than Moore's law, but so has our appetite for using it. Three years from now, mobiles might offer 20Gbytes of storage, and we'll be demanding terabytes. The good news is, we don't need all of our data all of the time – our attention span is simply too limited. Our mobiles' storage can operate as a level one cache, with the bulk of our files residing on a file server somewhere in cyberspace. We'll be able to search an index of all of our files, and choose the one we want. If it isn't already in our mobile, it will fetch it for us. And once the mobile's storage starts getting full, it will upload some files that we haven't used for a while and free up the space that they were using.

Worried about how long the download will take? HSUPA will soon offer better than 4Mbps downloads to our mobiles, and of that isn't fast enough, HSOPA will offer us up to 100Mbps.

Worried about who's going to store all your files and maintain an index across them, and deliver them to you when you need them, and how much it will cost? At the rate they're going, Google will probably do all this for free, unless you beat them off with a stick. Other folk will surely be happy to do it for money.

Xiris is also concerned about the lack of processor power in mobiles. They typically have a 200MHz 32 bit processor today, some have 400MHz, and they're "only" improving at the rate of Moore's law. If you want to to do serious spreadsheeting, for example, how responsive will your mobile be? And how much power would it gobble, running the battery down? Xiris also suggests the answer – to embed a utility processor in the docking station, which is a fixture and which can draw line power, and to offload some of the processing workload onto this processor while you're docked. This is a very realistic approach. The dinky screen size of the mobile is going to limit how much serious processing you do with it. If you need to crunch a serious spreadsheet, or update a large document, you will probably seek out a docking station for its big screen and keyboard if nothing else, and get the use of its processor at the same time.

So how can we offload processing onto the docking station? Let's get back to the way in which we deliver the mobile's applications to the docking station. I suggested that we should web-enable them, so that the user can use a browser on the docking station to view and drive the mobile's applications. But think about it – do we really want to code every mobile application twice, once to run on the mobile itself and a second time to run in a browser in a docking station? Life is too short, programmers too scarce (I know, I am one, and I can't get around to programming all the stuff that I want for myself). So let's assume that the mobile's applications are delivered in the same way to either its own screen or to a docking station – as a web app. Most modern mobiles have browsers built into them. Many only support the WAP or WML subsets of HTML, which doesn't include JavaScript, but a fair number of mobile browsers can handle JavaScript, and this number is increasing as the amount of memory available for phone apps grows.

So let's assume that the mobile of the future delivers its applications to a browser in much the same way that Google delivers its Docs and Spreadsheets applications, whether to a docking station or to its own screen. A lot of the processing now takes place in the browser instead of on the web server, courtesy of some clever JavaScript code. And the JavaScript code in Google's spreadsheet web app isn't nearly as smart as it could get. Currently, it sends every cell that you change to the web server via AJAX so that the server can decide how to format the cell's content, and update any formulae that might depend on that cell's value. Most times we change a cell, it's to write in a simple number or character string. It would not be hard to write JavaScript to check whether this is the case, and to format the cell accordingly. Nor would it be hard to check whether any formula includes the changed cell in its scope; if not, there's no need to recalculate the formula. Nor would it be hard to check if the user had previously specified a particular format for the cell that has changed, or a range of cells that includes the changed cell, and to apply the appropriate format. Nor would it be very hard to implement many of the functions that spreadsheets offer, and especially the ones most often used, in JavaScript code, so that any changes to the cells in their scope could be calculated in JavaScript in the browser. Only the way-out and weird cases would have to be sent up to the server, and that would be rare. The browser would handle the regular stuff on its own, offloading the server.

I have had a look at the HTML page that implements Google spreadsheets. It includes about 380KB of JavaScript code, most of which has had the comments and white space stripped out. When JavaScript is downloaded to a browser, it will be zip-compressed if both web server and browser support zip, which most do. That would bring the JavaScript download down to about 125KB. On a dedicated 1Mbps link, this would take about a second. That's about as long as it takes to open a spreadsheet package on my current laptop. And almost all browsers cache the files that they download, so if you get into the spreadsheet application several times in a day, your browser would only need to download the JavaScript code once – unless it ran short of buffer space and needed to prune its buffer.

Likewise when users of http://blogger.com like me :) use its posting page to write up and format a new post, clever JavaScript code in the posting page handles all our text input, editing, and formatting instructions in the browser, only involving the server once we have completed the document preparation. I have just had a look at the posting page. It includes about 23 JavaScript files and has a small amount of JavaScript embedded within it. In total, it uses less than 234KB of JavaScript code to implement what is a pretty handy document composition and editing facility. Most of the JavaScript files still have their comments and white space intact, for readability. There are utilities that can strip out comments and white space and spit out just the code that does the useful work. With zip-compression, the JavaScript download would be less than 60KB. On a dedicated 1Mbps link, this would take half a second. That' a lot quicker than opening your average word processor. And JavaScript file buffering wold help here too.

In short, we could offload most of the spreadsheet and document processing overheads onto the browser, reducing the load on the web server and the network at the same time. If the web server is running on your mobile and the spreadsheet or document processing is taking place in a browser on a docking station, then the processor in the docking station will be doing most of the work. So you wouldn't need a very fast processor in your mobile, nor would you burn up its battery while you do these tasks.

On the other hand, if you're on the road and only have your mobile to access one of the documents or spreadsheets on it and maybe make a few changes, then the browser and JavaScript will run on your mobile. Your processing rates will not be great, and you will burn your battery while you're busy, but at least it's possible. The constraints of your mobile's screen and keyboard will probably keep this mode of operation to a minimum, but it would be nice to know that when you really need to do it, you can.

The JavaScript language was developed originally by one man, Brendan Eich, rather than a committee. It was loosely based on the Java computer language. It's a much smaller, simpler language, and is not gaining features and fat as quickly as is Java, which has locked horns with .NET's C# in a sequel to the browser wars movie. Folk who know JavaScript well find it really elegant and powerful. It's not the language that anyone would make their first choice for coding a spreadsheet or document processor, but it's there in almost all browsers. As Woody Allen once said, 80 percent of success is just showing up, and JavaScript is showing up much more than any other in-browser language. Its biggest current drawback is that it's an interpretive language, and runs slower than a compiled language. But then, Java and C# are also inherently interpretive languages. Maybe it's time the open source community started working on a Just In Time compiler for JavaScript. Because of the small size of the language, it wouldn't be very hard to do.

By the way, I don't hold any stock in Google, but after researching for this post ... hmm.

After the Desktop – What?

2007-02-27T11:41:00.000+02:00

For decades the desktop PC has been the public face of computing to most folk. It has delivered a dazzling array of applications to us, but at a cost – it has tied us down to desks. Unless you are chained to your desk by your job, this can be pretty inconvenient. The workforce is becoming increasingly mobile, courtesy of the Internet which connects most places, and the mobile phone. Desktop PCs are anything but mobile.

The IT industry responded to mobility needs with the laptop, which was a big improvement over the desktop, but even so there are lots of places we go where laptops aren't convenient. Think of a ball game, bar, bicycle, or bathroom. So industry built palm tops or PDAs. Small enough to fit in a pocket or purse, and with enough smarts to handle many of our common computer needs – email, calendaring, address books, some calculation. But communication was a problem, until PDAs started incorporating mobile phones. Now mobile phones have returned the compliment, and incorporate the functions previously found in PDAs. The relentless miniaturization of electronics has enabled manufacturers to cram more and more function into mobile phones. They now offer fairly good web browsers, email, organizers, instant messaging, plus built-in cameras, videos, sound recording, music downloads and playing, and a few TV channels. Growing amounts of local storage allow users to take their favourite images and music with them.

Mobile phones are small and light enough to take with you almost any place with little inconvenience. They have a huge advantage over desktops, laptops, and the classic PDA – they have communication built in. You don't have to find land lines, modems, LANS, or cyber cafés to access the web. Almost anywhere you go, your mobile can connect. These factors are going to make mobiles the most widespread, heavily used human-computer interface.

Mobiles will replace the desktop.

"But wait, this cannot be!", I hear you say. "The mobile's screen and keyboard are too dinky, and its pointing device is too clunky! It cannot compete with the convenience of a big screen and keyboard."

Suppose you could have the best of both worlds. That when you're on the road, you could carry most of the files that you're currently working on in your mobile; that if you really need to access them and you don't have a big screen handy, you could still do so, slowly; but when you do get to your office, or your home, or to a cyber café, you could access the files and application on your mobile through a big screen and keyboard, with a proper pointing device. It wouldn't be that hard to do. When laptops first came out, they were challenged for storage space, modems, and other conveniences that were standard on desktops, so manufactures built docking stations for laptops. The laptops were as small and light as possible, the docking stations were bigger and heavier, but they held the extra goodies that couldn't fit into early laptops. Since then, shrinking electronics have enabled manufacturers to put pretty much everything we need into the compact form of the laptop.

So what would a docking station for a mobile phone look like, ideally?

It must have a fair-sized screen and keyboard, and a good pointing device
It must talk to the mobile wirelessly, like Bluetooth but much faster, and encrypted
It should be interchangeable:

any mobile should talk to any docking station
any docking station should allow access to any application on any mobile

Of all the requirements, interoperability is the most important – otherwise we'll all end up dragging our own docking stations behind us on airport trolleys, which would defeat the purpose. But how can one docking station talk to any application on any mobile? Surely that can't be done, it would require us to develop a universal terminal, which is clearly impossible – no, wait, we've done that already – it's called the browser.

So the mobile phone of the future will incorporate a small web server! No big deal, most modern electronic appliances like network switches already do so. The mobile user will be able to walk up to any available docking station and click a menu item on his or her mobile phone. It will connect securely via local radio to the (nearest) docking station(s). A prompt will appear on the docking station screen, the mobile user will type in credentials to authorise access, and the job will be done. The user can then interact with applications on his or her phone through a big screen and keyboard, and a proper pointing device. Apart from the applications that we have on our mobiles today, we would also get many of the applications that we need PCs to access today, such as word processing, spreadsheets, and other office automation software. Google Apps has already shown us that it's possible to deliver pretty good office automation apps through a vanilla web browser. And the response times would be great, we would be on a local wireless link.

Mobile users would be able to access files on their phones, update them, and save them plus new ones. These would be the files that they are currently working with. They could keep their other files on file servers that their phones could access securely through the Internet. Docking stations may offer fast broadband Internet connections, but if they didn't, the mobile phone would revert to its own network operator.

So where could we hope to find docking stations of this type in the future?

On desks at home, and in offices for those folk who are tied to offices
In cyber cafés, hotel lobbies and rooms, and airport lounges
In kiosks on street corners
In aircraft and buses, in the back of the seat in front of you
In cars

What software will the docking stations be running? They will be pretty simple appliances, and everything that they need to do can be done today with open source solutions like Linux and Firefox. If some manufacturers want to compete in this market space with software they pay for, that's their funeral. And what software will the mobile phones be running? That's up to the manufacturers. There will probably be a diversity for some time to come, but free open source options now exist that provide most of the function required by a mobile phone, and it's hard to beat free. Most of the new development on mobile phones is adding applications rather than re-inventing how phone calls are made. It's getting increasingly more difficult for manufacturers who use proprietary operating systems on their mobiles to source or develop all of the applications that are already available free in the open source space. Their problem will only get bigger as time goes by.

Once we have access to most of the PC applications that we use today in this convenient format, the need for most desktop and laptops PCs will fall away. No doubt there will still be power users who will need fairly powerful computers, lots of storage, and big screens, and who find it most convenient to get this in the shape of a desktop PC, but they will become the exception rather than the rule.

I would suggest that the open source community should stop obsessing with the battle for the desktop, and start focusing on the battle for the platform that will replace it. As I have said before, the day will soon come when the notion of having to go to a particular machine on a particular room every time you need access to information or computer power will be as obsolete as the notion of having to go to the hand pump over the well in the front yard every time you need water.

Open for Business

2005-07-15T05:59:00.000+02:00

"I think that I shall never see
A thing as lovely as a tree."

How nice then to be able to share ideas with other folk without causing the death of innocent trees to make paper. Once we have read what it carries, paper ends up in landfills, and these are eating up the limited space that we have on this precious earth.