This is not another Microsoft bash.

Really.

More of a Celebration of the Positive : it is, after all, nearly Christmas.

Always look on the Bright Side of Life and all that jazz…

So, I thought you, Dear Reader, might like to hear about some of the things Microsoft has got RIGHT recently.. sort of restore your Faith in Human Nature, the Wisdom of Crowds and the Divine Right of Corporates.

FTP FIXED !

It now WORKS … mostly (well, if’n you all go right out directly and get that patch for your IE7; and be sure’n tell all those good customers of yours to go get it right now… shoot, it’s a FREE FIX! )

Which is really nice to know, particularly for those of us who have apps out there which depend upon the FTP PROTOCOL for data transfer being implemented as per the RFC and working as it has always worked, even if our customers and clients have installed IE7 (or had it stealth installed) .

First off .. an article from Webmaster World http://www.webmasterworld.com/microsoft_windows_os/3521267.htm

which starts out

Whatever you do right now DO NOT upgrade to IE 7. The most recent patch/upgrade will stop you from being able to ftp to your websites.

This is happening to both XP and Vista users and was part of an automated update that happened last week and now this week with Vista.

and then from Microsoft

http://support.microsoft.com/kb/934376
You may be unable to use an FTP application to upload a file to a remote server on a computer that has Internet Explorer 7 installed

Now I must admit that I stood a while in uffish thought on this one. Quite a while.

Resting by the Tumtum tree, in uffish thought, I wondered .. why is it that a BROWSER upgrade affects my system (worse, those of my customers : I have VMWARE and can isolate such stuff if I am really careful, so I am safe-ish, maybe)?

Why is it that I can upgrade Firefox or Opera willy-nilly and not have collateral effects. Why is it that the greatest danger to the stability and integrity of my OS comes from the OS vendors when they background upgrade a piece of software that sits on top of the OS?

Well, this is good news day .. no need to spend more thought on it, uffish or otherwise. It has BEEN FIXED for us, calloo callay frabjous day and all that, and we are all appropriately grateful.

It seems that a background upgrade to WinInet.dll was responsible; there has been some whiffling and burbling about it being a “security risk” .. but WHY? Is FTP itself a security risk – in which case why did we not know about it long long long ago, and why does it not affect *NIX systems?

There is a LOT of data going around for which FTP is the transport vehicle.

But .. we sho’nuff are glad that it has been fixed, and that we know that when customers ring up saying “it’s broke” we can say .. well, maybe you installed IE7 or it was installed for you and there is an easy (and free) hotfix right here. Snicker-snack : as easy as that.

HUNGARIAN IS DEAD! long live …. ?

Well, I always thought that “Hungarian Notation” was someone’s idea of a cosmic prank (hey, let’s see how many zillion human hours can we waste by promoting this ‘convention’ ?) and tried to avoid it as I would the Jubjub bird and the frumious Bandersnatch.

And it seems that Microsoft has tired of the joke too. And, they have officially knocked Hungarian on the head

This news via the lutrov.com blog.

‘nuff said. Except maybe you too wasted a bunch of time on this, going with the flow, assuming they must be smarter than you and even though it seemed just downright silly and tedious and timewasting, there must be a there there, right?

And now it seems like there wasn’t, after all.

AND EXCEL CAN NOW MULTIPLY! MORE GOOD NEWS

(well, if’n you all go get dat fix – but shoot, who’s gwine notice anyoldhow?)

Quote of the day: “Nothing is more dangerous than a boss with a spreadsheet.” Scott Adams

Or maybe the spreadsheet itself?

We have been around this issue on Chris LLoyd’s excellent statistics Blog “Fishing in the Bay “ .. all kudos to Chris, good issues, good comments. http://blogs.mbs.edu/fishing-in-the-bay

That’s just to let you know that we are not on the same side of the fence when it comes to Excel, which is now of course synonymous with spreadsheets.

So, here’s the drum on the multiply problem (now fixed, we think)

Well, it seems Excel 2007 cannot/could not multiply correctly

http://www.appscout.com/2007/09/excel_cant_multiply.php
http://www.appscout.com/2007/09/microsoft_to_swat_excel_bug.php
http://blogs.msdn.com/excel/archive/2007/09/25/calculation-issue-update.aspx

Kind of a worry that, imho. But it does not seem to have worried anyone too much .. not programmers, not statisticians, not financial engineering hot shots (maybe because they don’t really use Excel anyway), but when I notified an accountant/actuary client of mine they immediately notified all of their clients and put their own stress testing in place.

It appears that it may not affect earlier versions, but if that is the case then someone must have rewritten the core calculation engine (which would be curious), or there is a deeper problem still. It is peculiar that what looks like a 16 bit to 32 bit conversion error is showing up now. Or wait, does Excel 2007 support 64 bit calculations? Did someone get their assembler instructions wrong?

The bug is bad enough. The implications are horrendous. ….

Once upon a time, statisticians would have been horrified to learn that one of their core tools could screw up like so, people used to care about numerical instabilities and so on.

But now it appears that we should not kick up a stink about an error like 65000 being close enough to 100,000.

And people run major businesses with these “tools”. Should someone make a stand somewhere and say, no, these things are not good enough, they are not only a ridiculous travesty of a calculation engine (that is, they get things wrong that the calculator embedded in my son’s mobile phone gets right) but they are going to lead you into making more and more dubious calculations, a house of cards.

On one of the links above is an “explanation” for what went wrong:

“It all boils down to the fact that you can’t represent an infinite group of non-integer numbers using a finite number of bits. In fact, Excel can store about nine quintillion distinct values. The numbers going into your calculations may be infinitesimally different from the number displayed, and for two calculations that nominally have the same answer the result may be infinitesimally different. Excel generally manages just fine in dealing with these tiny differences, but in exactly 12 instances out of the nine quintillion possibilities it goes completely bonkers.”

This was written by someone who should know better, and is an attempt to put a positive spin on incompetence. ROUNDING errors , and loss of precision, are of course intrinsic to the representation of a real number in a digital fashion .. but of course this has been well known since the dawn of programming and indeed since the birth of any sort of mechanical calculating device.

The fact that Excel can store “nine quintillion distinct values” is an utter irrelevance, and is a statement designed to mislead and divert attention from the fact that simple arithmetic, requiring only very low precision, is just plain WRONG in Excel. Not imprecise, but WRONG. No one else gets it WRONG.

So the argument that this is just one of those things, somehow a result of the fact that those silly computers can’t really store everything properly, golly gosh they can’t be expected to get it all right all the time and what is the problem with just 12 or so “numbers” being just that teensy weensy bit off when there are NINE QUINTILLION that the poor thing has to think about .. aw shucks, give those hardworking “engineers” at M$ a break, willya? ..

And a little more background

… it has always been a mystery to me why anyone numerate would use such an error prone and obscurantist approach (as Excel)

A long time ago a colleague attempted to introduce me to the joys of Excel .. a language in which I could apparently summate across a range of cells which contained mixed alpha and numeric values, and get an answer (and no warning). I was underwhelmed. I just checked, and the situation remains the same.

Now, no doubt there are tricks and arcana which will help protect you against such stupidity, but out of the box it is certainly EASY to do what I just said. If one has to rely on arcana and tips and special knowledge, then this is WITCHCRAFT we are talking about, not science.

I cannot envisage RELYING on a spreadsheet prepared by someone else, I cannot envisage employing someone to build complex spreadsheets (an oxymoron?) .. how could you POSSIBLY know if the “answer” is correct? If someone applied to me, for a job, with “advanced spreadsheet skills” on their CV then I would think that they had been left behind in the race rather (putting it kindly), and they should maybe go get a job in a business where it did not matter if the answer was right or wrong. And in the long run, Darwin rules, OK?

(fwiw, I have personal knowledge of a banking application which was converted from - as in completely rewritten - Excel to SAS : it cost them a lot of money partly because the original spreadsheet was complex aka messy, but apparently it was worth it.)

Computation is what proper programming languages were invented for. Strong typing (so the compiler tells you that you cannot add an alpha and a numeric) protects you against first level idiocies .. so don’t use variants OK (and if anyone knows what I am on about, use early binding rather than late .. same issue). The language should have strong support for sophisticated and readable flow of control constructs. And sensible meaningful variable names (not B1 and B2) .. Of course the logic should be transparent and clearly readable .. it should be “literate” (as in the literate programming concept of Knuth). And testable and stressable. And sealable.

Separation of data and algorithms .. we have spent years learning about the benefits of encapsulation, of designing software such that it has known and publishable and testable capabilities .. why muddy the waters by mixing up formulae and data in the one thingummy?

No doubt I will be asked what I would do instead. Well, as a data entry mechanism (ie no calcs) Excel is occasionally adequate, although every time I have taken this “easy” route I have had occasion to regret it (because the data entry operators have too much freedom). If the data is serious, get serious about a mechanism that will allow you to enter and check it properly (a database and form would do, even Access). But there are specialized data entry apps around.

So, use Excel, if you must, to enter data. Use a separate app to analyze it. Better still, use a language that supports OOP concepts.

Perhaps statisticians need to be better software “engineers” (although I dislike the usage) in order to encourage better data management and data analysis practices and not endorse sloppiness and tolerance of likely human error.

I think it is worth remembering that, although Bill Gates was supposedly “technical” he cut his teeth on quick basic, an interpreted “script kiddie” “language”. Some of that early influence still persists, and I guess we should call a spade a spade, and not endorse outmoded and potentially dangerous practices.

Andrey Kostenko commented:

For the sake of objectivity, one may be also interested in a special
feature of FORESIGHT, the International Journal of Applied
Forecasting, published February 2006, titled “Software: Spotlight
on Excel for Data Analysis and Forecasting.” The section includes:

INCORRECT NONLINEAR TREND CURVES IN EXCEL by Rick Hesse

THE UNRELIABILITY OF EXCEL’S STATISTICAL PROCEDURES by Bruce D. McCullough

ON THE USE AND ABUSE OF MICROSOFT EXCEL by Paul J. Fields