Why Must Software be Rewritten for Multi-Core Processors?

Perm url with updates: http://xahlee.org/UnixResource_dir/writ/multi-core_software.html

Why Must Software Be Rewritten For Multi-Core Processors?

Xah Lee, 2009-06-04, 2011-03-02

I had a revelation today, namely, that it is necessary to rewrite software to use multi-processor in order to benefit from it.

This may sound stupid, but is a revelation to me. For the past decade, the question has been on my mind, about why should software needs to be rewritten to take advantage of multi-processors. Because, in my mind, i thought that software are at some fundamental level just algorithms, and algorithms, have nothing to do with hardware implementation aspects such as number of processors. I always felt, that those talks about the need or difficulty of rewriting software for multi-processor (or multi-core these days) must be the product of idiocy of industrial imperative coding monkeys. In particular, some languages such as java, the way they deal with it, seems to me extremely stupid. e.g. the concept of threads. In my mind, there should be a layer between the software and the hardware, such as the operating system, or the compiler, that should be able to automatically handle or compile the software so that it FULLY use the multi-processors when present. In short, i thought that a algorithm really has nothing to do with hardware details.

I never really thought hard about this issue, but today, since i got a quad-core PC, so i looked into the issue, and thought about it, and i realized the answer. The gist is that, algorithm, fundamentally means manipulating some hardware, in fact, algorithm is a step by step instruction about some form of hardware, even the hardware may be abstract or virtual. For example, let's say the simplest case of 1+1. It is a algorithm, but where is the hardware? You may say it's abstract concept, or it being a mathematical model. What you call 1+1 depends on the context, but in our context, those numbers are the hardware. To see this, lets say more complex example of listing primes by sieve. Again, it comes down to “what is a number”? Here, numbers can be stones, or arrangement of beads on abacus, it's hardware! As another example, say sorting. To begin with, you have to have something to sort, that's hardware.

Another way to see this is that, for a given computing problem, there are infinite number of algorithms to achieve the same thing. Some, will be better ones, requiring less steps, or requiring less storage. All these are concrete manipulation issues, and the thing being manipulated, ultimately we have to call it hardware. So, when hardware changes, say from one cpu to multi-cpu, there's no way for the algorithm to magically change and adopt the changed hardware. If you need a algorithm that is catered to the new hardware, you need a new algorithm.

One might think that there might be algorithm Omega, such that it takes input of old hardware O, new hardware N, and a algorithm A, and output a algorithm B, such that B has the same behavior as A, but N+B performs better than O+A. This is like asking for Strong AI.

One thing we often forgot is that algorithms ultimately translates to manipulating hardware. In a modern digital computer, that means software algorithms ultimately becomes machine instructions in CPU, which manipulate the 1s and 0s in register, or electricity voltage in transisters.

In a more mundane point of view, a automatic system for software to work on multi-processors is a problem of breaking a given algorithm into independent units (so that they can be computed in parallel). The problem of finding a algorithm is entirely different from the problem of breaking a algorithm into independent units. The problem of dissecting a given algorithm into independent units is a entire new branch of mathematics. The problem of writing a unitized algorithm is also relatively new, with the need to analyze a problem by independent units. For example, let's say factoring. Factoring is a well defined mathematical problem. There are millions algorithms to do it, each class has different properties such as number of steps or storage units. However, if we look at these algorithms from the point of view of independent units, it's a new perspective on classification of algorithms. Software are in general too ill-defined and fuzzy and complex, the software we use on personal computers such as web browsers, games, video players, don't even have mathematical models. They don't even have mathematical models of their inputs and outputs. To talk about automatic system of unitizing a given software, would be more like a AI fantasy. Roughly, a term that describes this aspect of research is Parallel computing.

In the Wikipedia article, it talks about types of parallelism: Bit-level parallelism, Instruction-level parallelism, Data parallelism, Task parallelism. Then it also discusses types of hardware related issues: multicore, symmetric multiprocessing, distributed computing, cluster, grid. The subjects mentioned there more close to this essay are “data parallelism” and “task parallelism”. However, neither are high level enough as i discussed here. The issue discussed here would be like “algorithmic parallelism”. Indeed, Wikipedia mentioned “Automatic parallelization”, which is exactly what i'm talking about here. Quote:

Automatic parallelization of a sequential program by a compiler is the holy grail of parallel computing. Despite decades of work by compiler researchers, automatic parallelization has had only limited success.[40]

Mainstream parallel programming languages remain either explicitly parallel or (at best) partially implicit, in which a programmer gives the compiler directives for parallelization. A few fully implicit parallel programming languages exist — SISAL, Parallel Haskell, and (for FPGAs) Mitrion-C.

It says “A few fully implicit parallel programming languages exist”. If you are a starry-eyed comp lang tech geeker, don't get carried away by what those words might seem to mean.

(Note: Wikipedia has a dedicated article on the subject: Automatic parallelization)

Reading those Wikipedia articles, it solidifies the popular thought of why functional languages have a great time with the multi-processor future. Because in functional languages, you write software by synthesizing functions, and functions are by definition, independent units. This leaves the compiler much more able to analyzes and dissect to produce efficient byte code for multi-processor machines.

More Wikipedia readings: Multi-core (computing), 64-bit, X86 architecture, Instruction set architecture.

For a excellent discussion of this topic, with actual code examples, see:Guy Steele on Parallel Programing.

✻ ✻ ✻

I've been studying math and computing fanatically since about 1991. Sometimes, i suddenly had a insight on questions that has bugged me for years. I the past few years, i have written out some of these. Here they are: Multiplication and Multiplicative IdentityThe Geometric Significance of Complex ConjugateWhat is Technical Drawing, Descriptive Geometry, Projective Geometry, Linear Algebra.


The Complexity And Tedium of Software Engineering

Perm url with updates: http://xahlee.org/UnixResource_dir/writ/programer_frustration.html.

The Complexity And Tedium of Software Engineering

Xah Lee, 2009-06-02

This page is a blog of a experience of few examples that illustrates some seemingly trivial task can become quite tedius and complicated in the software industry.

A Complexity with Emacs

Discovered a emacs problem.


  • Seems that emacs 23 will have a problem loading a css-mode written by Stefan Monnier
  • The css-mode.el file does not contain any sort of version number, which makes the problem worse.

Detail: I have a Mac running emacs 22 with OS X, and i have PC running emacs 23 and Windows Vista. When i use the emacs 23 to load css mode, it gives this error: “if: Wrong type argument: integerp, (0 . 8)”.

The problem seems simple in retrospect, but wasn't simple at all when you trying to get things done and things don't work as expected. Here's the story.

Emacs 22 does not have a css mode, emacs 23 does. There's one css mode written by Stefan. I've been using it on the Mac for the past couple of years. The same package is now bundled into emacs 23, which i'm using on PC. However, the code in the 2 files are not identical. I have my emacs setup to load css mode. Since i am migrating to PC, alone with my whole emacs system, and am setting up a Mac and PC and network that'd let me work with either machine harmoniously. On the PC, when i start css mode, it gives error, but not always. You don't know what's wrong. It could be a fuckup in the emacs distro i'm using on PC (which is emacsW32), or it could be emacs 23 problem (emacs 23 is still in beta), or it could be something in my emacs customization that works perfectly well on the Mac but not on the PC with whole new environment. Eventually, i realized that's because sometimes i started plain emacs 23 without loading my own setup, it was using the bundled css mode, so no error, but when i loaded my own emacs setup, it gives error. This seems still simple in retrospect, but wasn't then. I added a version check to my emacs init file(s), so that if emacs is 23, don't load my css mode. The next day, same symptom occurs. Eventually I realized that's because the load path is set so that my own version of css mode comes before the bundled css mode, so that, when i load css, again my old version is loaded. Eventually, the solution is to check if css mode bundled with emacsW32 runs ok on emacs 22 on my Mac; if so, simply use that as the single source for my Mac and PC. When doing this work on checking what versions are the files, i realized that the 2 css mode's files don't contain version number at all. All this takes 5 minutes to read, but was one of the gazillion seemingly trivial issues and problems when setting my Mac/PC networking environment with cygwin and all. This took me one week, and haven't gotten to wholly converting my Mac files to PC. Added the time to google for for the answer, possibly write a report to the emacsers, etc, all in all, i'd say this problem caused me 4 hours.

Here's the emacs version i'm using. On the Mac: “GNU Emacs 22.2.1 (powerpc-apple-darwin8.11.0, Carbon Version 1.6.0) of 2008-04-05 on g5.tokyo.stp.isas.jaxa.jp”. On Windows “GNU Emacs (i386-mingw-nt6.0.6001) of 2009-05-28 on LENNART-69DE564 (patched)”

URL Encoding

Discovered a subtle issue with automating url encoding. In url, if you have the ampersand “&” char, and if this url is to be inside a html doc as a link, then, there's no automated procedure that determines correctly whether the char should be encoded as “%26” or “&”. If the char is used as part of the file name or path, then it should be encoded as “&”, but if it is used as a separator for CGI parameters, then it should be encoded as “%26”.

The rule for Percent encoding the char is that the ampersand is a reserved char, therefore it must be percent encoded if it is used for normal file path names. So, when it is NOT used as part of path names, but used as CGI parameter separaters, with GET method of HTTP request, then it must be left as it is. Now, in html, the ampersand char must be encoded as html entities “&” when adjacent chars are not space (basically). So, it becomes “&”.

In summary, if the url in html link contains “&”, and the char is a CGI param separator, then encode it to “&”, else, encode it as “%26”, and my discovery is that the purpose of the char used in url cannot be syntactically determined with 100% accuracy.

Of course, in practice, all this matters shit. Just use “&” plainly and it all works in 99.999% of situations.

Unicode File Names

Unison and Unicode File Names

Summary: Unison does not support unicode chars.

When using Unison version 2.27, to sync files from Mac to PC, the file name on the mac is “赵薇_flag.jpg”, it is copied to PC using Unison 2.27, and the file name became “èµµè-╪_flag.jpg” when viewed under Windows, but the file name shows up correctly when viewed in EmacsW32.

Mac version: 10.4.11, Windows Vista SP1.

This may indicate that Unison encode file names in utf8, or just ascii. Indeed, it is said on Wikipedia that unicode have problems with non-ascii file names.

When the file is copied from Mac to Windows or Windows to Mac, operating either on Windows or Mac as the local machine, using either OS's file manager, the file name is copied correctly.

Setting up Unison itself is not so trivial. It is trivial in concept, but actually took hours. I have Unison on my Mac installed, and use it few times a year, since about 2006, so i'm familiar with it. On PC, first you have to install cygwin. I know there are Unison binaries for Windows but since i use cygwin, so my first choice is staying with cygwin, since it contains the whole unix environment. Installing cygwin is another story, but once you installed Unison in cygwin, and tried to test sync my Mac and PC, you run into the problem that sshd must be turned on in one of the machines. Namely, sshd should run on the “remote” machine. (setting up a local network among Win and Mac is another complex and tedious story) Then, there's the issue of deciding which machine you want sshd to run or both. On the Mac, i can turn on sshd in a minute. On Windows, i'm not sure. I'm not sure if Windows Vista home edition provide ssh server, and am not sure how to turn it on if so. As far as i know, Windows Vista Home does not come with a ssh client. In the process, also realize that firewall must be turned off for ssh port. So, you spend 30 min or possibly hours (here and there) reading or probing with Windows Firewall control panel and whatnot other admin tools. After a while, i decided it's easier just to turn on sshd on the Mac then Unison from the Windows side to the Mac. At least, have this direction work first, and when that works, i can play with the other direction. After all this done, i tried to Unison, but Unison reports that the Unison version on my Mac and PC is not the same, so it doesn't work. Geeze. The one on my Mac turns out to be Unison 2.13.x, and the one i have in Cygwin is 2.31.x. Now, i figured that with each release of Unison, it probably obsolete some older versions. So, back to digging Unison docs and the web. The simplest solution comes to mind is to update my Unison on my Mac to latest, of which, the unison on fink haven't been updated for a couple of years. I use Fink, and am fairly familiar with using Fink. However, after “fink selfupdate” etc, then “fink desc Unison”, the version there reported seems to be 2.13. Then, searching web on fink home page indicates they have unisone-2.27.57-1007, for my OS 10.4 PowerPC. So, why doesn't it show up in my fink? I remember i had a case last year where fink had obsolete mirror databases, a fault entirely on their end.

After spending maybe some more 30 min, i decided to install Unison from a website, binary download. After that done, i got Unison 2.27.x on the Mac. I tried to sync again, still no go. So, it seems like that the Unison version number must be the same or very close. Checking on Unison website, it looks like the current stable release is 2.27.x, so, my cygwin's 2.31.x is actually a beta. Damn. So, now back to cygwin. Luckily, it appears there are several version of Unison there to be installed, and i easily installed 2.27. Then, finally, test sync is successful. Now, i go back to get my files ready in a state to be synced. (long story there. See: Perl Script for Removing Mac Resource Fork and Switching from Mac/Unix To PC/Windows ) When finally i'm ready to Unison, then there's the Chinese character problem!

Emacs on Windows and Unicode File Names

When using emacsW32, dired has problem dealing with files with chinese char on remote machine. Consequently, when trying to copy, delete, etc, the behavior may be dangerous.

e.g. i have this file 林志玲.jpg on a Mac. I use emacsW32 on Windows to view it thru network. (e.g. the path is // ... ) the file name shows up as “_viu0y~a.jpg” in dired.

“GNU Emacs (i386-mingw-nt6.0.6001) of 2009-05-28 on LENNART-69DE564 (patched)”, Mac version: 10.4.11, Windows Vista SP1.


Perl Script for Removing Mac Resource Fork

Perl Script for Removing Mac Resource Fork

perm url with updates: Perl Script for Removing Mac Resource Fork

Xah Lee, 2009-05-31

This page shows some perl script and tips for preparing Mac files to be used on Windows or Linux.

Resource Forks and File Type Code

Before Mac OS X, Mac files heavily relies on Resource fork. With OS X, it is decided in the early 2000s that resource fork is going the ways of dinosaur.

Another confusing thing is that Mac files often has file Type code. Its purpose is similar to Filename extension used by Windows and Internet media type for indicating which app can be used to open the file. The Type Code is not stored in Resource Fork, but is part of the HFS+ file system.

Resource fork for data files is discourage by Apple since early 2000s, and i think vast majority of modern apps does not create files with resource fork. However, Mac applications (those in “/Applications/” folder, may still rely on resource fork to function.

File type code are still used in OS X. However, it can be deleted without creating much problem, since all it does is associating files with applications, and this mechanism is largely replaced by filename extension.

For converting your old Mac files, you cannot simply delete resource fork, because for some files, such as “unflattened” QuickTime movie files, the main data is in the resource fork.

Here's a perl script that removes resource fork of image files. It can be easily modified to report files that contains resource fork. delete_image_file_resource_fork.pl. (resource forks in image files are almost always just thumbnails, created by application such as GraphicConverter. The thumbnail of image files created by Finder are not stored as resource forks.)

For some more details about how to use OS X's command line to check resource fork or file type, see: Mac OS X Command Line Tools Tips.

“.DS_Store” and “Thumbs.db” files

Mac also creates a .DS_Store file in each folder. ( Windows creates Thumbs.db) You'd want to remove them if you are copying them to Windows. You can run the following command in bash to remove them:

find . -name ".DS_Store"
find . -name ".DS_Store" -exec rm {} \; 

Here's a perl script that does the job in more flexible ways: delete_DS_Store.pl

“Icon^M” files

Another Mac specific file are those files named as “Icon^M”, where the “^M” is the Return character (ASCII 13). These are folder icon files, but am not sure they are necessarily in Apple Icon Image format. I'm not sure what's their official status with OS X. However, you can still find them in OS X. For example, you'll find it in “/Applications/Adobe Reader 8/”, as well in StuffIt 10, Mac Pov-Ray 3.6, Adium, and if you use Jamie Zawinski's XScreenSaver for OSX, you'll find a lot “Icon^M” files in your “~/Library/” dir.

Here's a perl script that delete these files: delete_macos9_icon_file.pl.


Switching from Mac/Unix To PC/Windows

Switching from Mac/Unix To PC/Windows

perm link with latest updates: http://xahlee.org/mswin/switch_to_windows.html

Xah Lee, 2009-05-26

I bought a PC with Windows today, and am switching to Windows from my 19 years of Mac experience with 10 years of professional unix computing. This page tells the story.

I have a aging Mac. It is a IMac G5 (iSight) bought in late 2005. It is the last iMac with a PowerPC chip.

The machine is great, but is 4 years old. So, i needed a new machine. I went to local shop Fry's Electronics and was just gonna get the new Mac Mini that Apple just released this month. The cheapest Mac Mini is $600. Looking in the store, you find that a PC for the same price has at least 2 times the hardware power.

Mac vs PC
Mac mini (MB463LL/A) amazon PC (HP Pavilion A6750F)
CPU2.0 GHz Intel Core 2 Duo2.3 GHz AMD Phenom X4 9650 Quad-Core
Hard Disk120 GB750 GB

By the above comparison, you can see that the Mac is fucking shit, and they've been in that state in the past about 6 years. The Mac OS X was beginning to save Mac since 2002, together with the pretty and robust and inexpensive first IMac (“Bondi Blue”) that debuted in 1999. The Mac had a good time from about 1999 to 2003. But since the PowerPC slowness fiasco and hence the Intel "SWITCH" , the Mac has never recovered, getting more and more expensive with its good looks, and far less powerful or flexible. Apple is too busy becoming a fashion and multimedia company with its iPod and iTune successes.

Software has always been the Mac's strong point, however, these days Microsoft's software tech went far beyond Apple, witness the .NET, F# lang, Direct3D, Silverlight, etc, and the operating system itself is no worse than Mac's, starting with Windows NT in the late 1990s and its descendants 2000, XP, Vista. And, almost all of the best so-called Desktop Publishing software that started life on the Mac in the early 1990s have moved to Windows as their main platform in the past decade, in fact, many no longer even produce Mac versions. (Mathematica, Photoshop, Macromedia stuff, 3D modelers ... )

I'm tired of Mac being the second rate citizen for everything. Latest Java version comes 0.5 or 1 year later than Windows, similar for Google Chrome, Google Earth, Adobe Flash, all IM chats from MSN, AOL, Yahoo, Second Life ... all come in crippled version and late. This applies to just about any software. You are lucky if there is a Mac version. In general, not only Mac buys you less hardware muscle, but the Mac version of software is usually slower than Windows version running on similarly powered hardware, because the software gets less development time and less optimization. (e.g. Javascript, Java, Flash, etc.) Commercial websites often have problems when accessed with Mac browsers.

I have owned Macs since 1991, and was a dedicated fan all thru the 1990s. I started to use Windows at work for 8 hours a day from 1999 to 2004. I also own a fancy Windows labtop from about 2003 to 2005. So, i'm pretty familiar with Windows at least as a user. But since about 2006, i haven't touched Windows. In the past 3 days, i'm picking up my Windows knowledge. The experience is great. Of course there are several areas Mac is better. For example, the graphics, anti-aliased font, simply are far superior. Inputting Unicode chars on Windows with its Character Map is fucking idiotic, most Unicode simply doesn't show. User interface of apps in Windows are frequently stupid. Also, lots of fucking pop-up asking you to install this or warning that or support feedback fuck every minute. However, Windows is quite usable, in fact, all things considered, i feel the Windows experience today is better than Mac. Fuck the Mac fanatics fuckheads.

My PC Specification

Here's my PC product page from HP, including hardware spec, software, drivers etc: http://h10025.www1.hp.com/ewfrf/wc/product?product=3872428

For my own records: HP_Pavilion_A6750F_spec.txt, motherboard_Aspen-GL8E.txt.

Also bought: “BFG Tech NVIDIA GeForce 9800 GT 512 MB”, for about $120. http://www.bfgtech.com/BFGR98512GTOCE.aspx. amazon This is for playing Second Life, and my geometry 3D visualization needs. On the Mac, adding a graphics card is impossible unless you buy a Mac Pro, which starts at $2500, and is even less powerful than a sub $1000 PC.

Software I Need

Here's a list of software that i use hourly, daily, or weekly. Many of which are Window's counter-part to similar Mac software. (roughly listed in order of importance to me)

  • emacs editor. On Mac, Carbon Emacs. On Windows, EmacsW32.
  • Unix user level software. On Windows, the answer is Cygwin. In particular, find, grep, xargs, tar, gzip, awk. imagemagick, optipng/pngcrush, ssh, sftp, rsync, unison, curl, wget.
  • Computer languages: Pyhton, php, perl, Ocaml.
  • Browsers: Firefox, Google Chrome, Opera, Safari. (yes, i need them all for web dev.) Firefox add-ons: Web Developer, Html Validator, Flashblock, Undo Closed Tabs Button, Screengrab, StumbleUpon, Cooliris.
  • Math software: GeoGebra, Mathematica, and others.
  • Keyboard macros. On the Mac, this is IntelliType and Quicksilver, along with AppleScript and bash. On Windows, it's IntelliType and Autohotkey and Windows shell and PowerShell.
  • IM software. On the Mac, i use Adium for multi-protocol text chat. On Windows, i'm using Pidgin (software). For voice and video chat, i need Skype, and also occasionally use chat apps from MSN, Yahoo, AOL.
  • Image and drawing apps. Gimp and Inkscape
  • GoogleEarth
  • Movie player VLC and QuickTime. My music player of choice is iTune. For viewing my porn collection of over 30k of images by slide show with random pick, i use GraphicConverter. On Windows, i use Irfanview.
  • Second Life
  • μTorrent or Vuze. Might also try Peer gardian.

All the above are free, except Mathematica.

Discovered MinGW. It is like CygWin, but smaller scale and simpler. It provide you with unix system tools, in particular letting you compile unix source code with GNU C Compiler.

Discovered Inno Setup, a free software that lets you create installers.

I thought i was going to spend a hell a lot of time to configure all things, and re-learn my user-level Windows skills, and in particular getting all my unix utilities to work and get my whole emacs system up and running. But actually, i've spend roughly 3 days on setting up the machine and transferring my files, and it's rather easy. Good thing for me that since about 2005, i've made a habit to maintain all my data in a OS-neutral way, so that they should be easily transferred among Mac, Windows, or Linux if i make the switch.

I'm particular happy that my emacs with few thousand lines of elisp code is easily transferred. (thanks to Lennart Borgman's EmacsW32 and my quality coding with attention to OS dependency) And, am happy how my whole unix need is installed in a breeze, thanks to Cygwin!

Windows Appearance Setup

How to set the front window to have red title bar but still have Windows Aero feature?


Under Windows Aero theme, i find it hard to tell which window is the current. I'd like to set the current active window to have red border. I can do this in Appearance Settings if i set it to Windows Classic, but with Windows Classic theme, you lose many good features of Vista such as live thumbnails when pressing “Alt+Tab” or “Start+Tab”.

I don't find the glass effect productive. Translucency in GUI is first made popular by Apple in OS X's Aqua (user interface) in the early 2000s. (in shows in menu pane, and sometimes can be used for a app's window.) It is pretty, but i find it actually counter-productive. In almost no cases, i find it productive to see parts of what is behind, and if i need to see what's behind, translucency isn't useful. If you set it to too transparant, it inteferes in seeing what's in front, if you set to to too opaque, you lose the productive translucency purpose of needing to peak at what's behind. With default Windows Aero setup, your screen become this blur of panes.

Mac and Windows File Conversion

Mac OS Files With Resource Folk and Type Code

Before OS X (1990s), Mac OS files heavily relies on Resource fork. With OS X, it is decided in the early 2000s that resource fork is going the ways of dinosaur. So, in early 2000s i did some systematic conversion of all my files with resource fork to versions without. You cannot simply delete resource fork, because for some files, such as QuickTime movie files, the meat lies in resource fork. Today, i think that some applications files (those under “/Applications” dir) still require resource fork data to be intact in order to function, but i'm not sure.

Another confusing thing is that many Mac files have a 4-letter Type code, to indicate file type and creator type. (Its purpose is similar to Filename extension used by Windows and Internet media type (aka MIME types used on the web)) This type code is not in the resource fork, it is just part of the HFS+ file system. Today, this info can be safely deleted since OS X has switched to filename extension mechanism when it decided to ditch resource fork.

For tech detail on how to check or remove these data, see Perl Script for Removing Mac Resource Fork, Mac OS X Command Line Tools Tips.

Converting Email

I have about over 15 years worth of email in my current Apple Mail on my Mac. The earliest there dates to 1996. On average, about every 2 or 3 years, i go thru email conversion. It may be from one email application to another, or export/import from a app's upgrade. In the late 1990s, it was several upgrade with Eudora, and Mac version of Outlook Express. In early 2000s, it was from Outlook Express to Apple Mail, and followed by several version upgrades of Apple Mail that came with new versions of OS X in its fickle beginnings. In these years, i've also used quite a variety of email apps in Mac and Windows and Unixes, such as various variants in emacs (rmail, vm, gnus), and several variants of unixes ( mail, pine, mutt), and also have used the email app in Netscape Communicator, and during 2002-2004, used Windows's Outlook Express at work. Have also for some period tried to use Opera Browser's mail, as well as Mozilla Thunderbird. So, all in all, i've had quite some experience of mail conversions.

Generally speaking, you always lose some data. The conversion may be from a upgrade of the same app, or between different apps, or between different OSes. If the apps in question are typical upgrade/conversion path, popular commercial competing ones, or on the same OS (e.g. Eudora, Mac Outlook Express, Netscape Communicator, in late 1990s era, or Apple Mail upgrades in early 2000s), there may be builtin menu command for importing your mail. Otherwise you are out of luck, and have to do manual tech geeking with possibly few day's worth of probing and text processing. After the import, some foreign characters are likely to be fucked up, some header will be fucked such as date or subject, the read/unreaded will be gone, formatting of lines can be fucked up, plain-text vs html vs rich can be fucked, thread info if it exist will certainly be gone, rating/priority/marking or such features will be gone, attachments can be lost.

Although, the most important part, the text themselfs, will largely be preserved.

There are many many reasons of these conversion fucking problems. First of all, it's due to there being no one standardized and robust email format. There is this mbox plain text format, invented by the motherfucking unixers. Almost no 2 email apps that uses seemingly same mbox format are actually the same, and some, such as Mac Microsoft Outlook, uses a database system (which is very good!). Then, it's due to non-precise and badly designed RFC mail protocols invented by the motherfucking unixers. (RFC = Really Fucking Common.) Then, there's several formats for rich text email (i.e. Enriched text, HTML e-mail, MIME.), none of which is near precise. (The Apple motherfucking fanatics, in early 2000s, tried to push its Enriched text format as the standard instead of HTML mail) Then, there's also EOL char disagreement among Unix, Mac, Windows. Then, there's transmission encoding issue, since the unixer fuckfaces's mail protocol only does 7-bits ascii. Then, there's the char set issues (e.g. for different languages), especially in the 1990s or early 2000s, resulting incorrectly encoded charset or incorrectly tagged charset label. Then, each email app may have its own features, such as various markings (e.g. read/unread, replied/unreplied, mail priority), and there's features of link to reply, threading, etc.

See also: Unix and the mbox Email Format, Unix, RFC, and Line Truncation

As to what i'm going to do for my Apple Mail email archive... i'm not sure yet. Apple Mail uses “.emlx”, while Windows Mail uses “.eml”, both are variants of one file per mail plain text format. I could spend a few hours to marshal my apple mails and put them in a dir structure and coax windows mail to import or see it. It will probably take a day, and the result import will probably not be that great. Or, i could just leave my apple mail in plain text as a historical archive, to be searched by grep. We'll see.

Keyboarding & Macros

I'm a operation efficiency nerd. When operating a computer, you have to switch apps, click links/checkbox/buttons, invoke menus, open/close window or tab, adjust volume, change songs, switch input focus, switch fields, input/edit text, or perform pointer oriented input such as in a image editing software. (here, we are excluding other forms of input such as voice) In general, all these are inputs, done by using input systems, a combination of software and hardware. Typically, the hardware is just keyboard and mouse (or touchpad or trackball). The software part is the interfaces used in the OS and various apps.

Operation efficiency means you are able to achieve a input task with the least number of units of physical exertion, basically means less keystrokes, less mouse travel, and most importantly, less brain work to carry the task out. (brain work, such as the need to eyeball a process, is actually the most costy.) As a concrete example, let's say you have a url in a text editor. Say you want to view that url in a browser. There are many ways to achieve that. One way, is to use mouse to select the url, right click to select the copy menu, switch or launch a browser (many ways to do this), put focus on the url field, then right click to paste it, then hit Enter to load the url. This method, is rather most typical. As a example of a less efficient way, is to actually type the url text in the browser field. As a example of more efficient way, is by pressing keys to copy & paste instead of using the mouse. But suppose the task of switch/launching browser, there are different methods of varying efficiency. For example, a typical savvy user will press “Alt+Tab” to switch to it if it already launched. Otherwise, he may pull Start menu, or press Start key, then use mouse or press arrow keys to select the browser. But as a example of a more advanced system, is for someone to have a customized shortcut setup so that he can press one button on his keyboard, and he is launched or switched to the browser, regardless whether the browser is already launched. (in fact, many keyboards have a special key preset to launch/switch browser) But as a example of more advanced operation, for someone using emacs, he may have a scripted customization such that he presses one single button, and a browser is auto launched/switched and the url automatically loaded. This does not imply that such degree of customization is the most efficient, nor that one must use things like emacs, but is here only as a example of a efficiency at one particular common task.

I don't care if you are a linux user, whatever Ratpoison interface you are using, what command-line nutcase motherfuck you are, or how you've customized your keyboard with X11 xmodmap, i assure you, that i'm able to operate my Mac or Windows with my own custom setup more efficient than you, and faster than you too. (efficiency doesn't ensure fast, because speed involves performance aspect too (e.g. if you are 80 years old, doesn't matter how efficient your methods are, you are likely to be slower than a average 18 years old computer user.). But since i'm atheletic and touch type 85 wpm, so, adding my efficiency, i'm certainly faster too.)

I use a Dvorak keyboard layout, and on either Mac or Windows, i am intimately familiar with ALL its keyboard shortcuts. ALL. Not just that. I've also explored to the hilt on how to customize the system, may it be key macros, some system automation scripting, or some level of OS modification, on either OS. But that's not all. Even knowing all of the above, to become most efficient, you must have arrived at a operational method that symphonize each aspect of the system to achive maximum over all efficiency. (input aspects include: keyboard shortcuts, macros, key remappings, and physical keyboard hardware (those modifiers and extra buttons, their physical positions, software keymappings), and the OS behavior and each app's behavior). It is important that one must take all aspects into consideration if the goal is ultimate operational efficiency. For example, suppose you are a keyboard freak like many linux tech geekers who take every opportunity to insist that mouse users are idiots. However, if the task is opening various links in a browser, keyboard operation is certainly a magnitude slower than using a mouse. This is just a obvious example, similar examples are operating image editing or drawing software, including many gaming situations.

Won't go into details of my history and method and software and keyboards etc that i've used in the past 19 years. Just want to say here that in general, Windows Vista offers better operational efficiency than OS X out of the box. In a simplified summary, this basically means there are more keyboard shortcuts on Windows, and the way the Windows shortcut system works lets user achieve a task faster.

Here's some advice for Mac and Windows users for operational efficiency.

See also:

What's window's shell command to launch a app? e.g. on osx, i can launch firefox at a url with “open -a firefox ‹url›”.

e.g. “start firefox yahoo.com”.

Is there a keyboard shortcut to make the task bar on focus, so i can use arrows keys to select a doc? i.e. like “Alt+Tab” but no need to keep holding the Alt.


Is there a way so that i can press a key and switch me to a particular app? The problem with Alt+Tab is that i have to hold the key and eyeball the process. Alt+Shift+Tab is also cumbersome.

I think Autohotkey + IntelliType should do the trick. Will need to find out how.

Is there a way to set a app to launch only a single process? For example, i prefer Firefox to have a single process with many tabs, so i don't have to Alt+Tab among several FireFox instances. Some app, such as emacs, iTune, instant messaging apps, are apparently setup this way.


Fonts and Unicode

I use Unicode a lot, especially on my website, of math symbols, computing symbols, and other misc glyphs. For examples:

In my midst of doing a lot things setting up my PC, somehow i thought that Windows doesn't display most of the Unicode symbols i need out of the box. So, i spend like 4 hours researching and trying to find solutions. The following write up is the result of that. However, in the end, i was mistaken. Windows Vista does support displaying most Unicode chars out of the box, just not as good as OS X.

On the Mac, Unicode chars shows up beautifully out of the box. On Windows, more math symbols shows up as a square, and Chinese are rendered as bitmapped font. Further, to input Unicode in Windows is a pain in the ass, with its ugly and hard-to use Character map app. On the Mac, you can press “Opt+‹key›” for commonly used symbols. There's also Character Palette, similar in functionality to Character Map, but much better user interface.

For inputting math symbols and other special Unicode symbols, i rely on emacs, since i have a system setup in emacs that lets me input my personal set of frequently used Unicode chars.

Brushing up on my font knowledge: Computer font. In particular, the role of TrueType and OpenType in the market, looks like OpenType is almost the universal standard now.

Mac has this technology ATSUI for rending Unicode. That is, a font needs not to contain all Unicode chars. When a font doesn't have some Unicode chars, the glyph from other font that contains it are automatically used. I think Windows must have similar tech. (Note: looks like ATSUI is replaced by Core Text in Mac OS X 1.5.)

On reading Unicode font, i discovered Lucida Sans Unicode. To get browsers or any app to show Unicode, just set it to use that font, which is included in Windows Vista. This discovery is nice, because that means i can also set CSS to use that font on my website, so that my pages looks good for Windows users. (A nearly identical font shipped with OS X is Lucida Grande. These are sans-serif fonts. Haven't looked for Serif ones yet.)

Other Unicode font mentioned in Wikipedia that shipped with Windows are: Unihan font and Everson Mono, but these are not part of Windows Vista.

Other Unicode font mentioned in Wikipedia that seems best quality is Arial Unicode MS, however, it comes with MS Word/Office only. Another one, pay-if-you-like shareware, is Code2000, which is the first i tried to download and use. What a motherfucking ugly garbage. Another one mentioned, that contains a lot glyphs which may be a candidate of what i need, is GNU Unifont. However, it is bitmapped font. Bitmapped font in 2009? Hello? Low life Open Source idiots.

Since i own a Mac, i can take TrueType fonts from OS X to Windows, since i think Windows also read TrueType fonts. This is good for just for displaying chars in my own PC though, such as in emacs. Will have to try this out since fonts shipped with OS X are much better. Will also have to look for a Unicode monospaced font down the road.

See also: Common Fonts on Both Windows and Mac.

Windows Tips and Microsoft Technologies

Learn the windows command line: Command Prompt (Windows)

Windows PowerShell, the next generation of shell, fully integrated with the .NET Framework.

XML Paper Specification, a XML based tech for similar purpose as PostScript.

Old windows init file: INI file, but is now Windows Registry.


Misc Notes and Mac PC Issues

• Emacs. get emacs to load my init file automatically.

• Emacs. In the doc it is mentioned most unix util are bundled one way or another. When i tried M-x grep-find, it seems to give me “The system cannot find the file specified.”. Investigate.

What's the Windows shell equivalent of Bash's “&”? I want to start a process but still have access to the shell window.

Not sure if this is the absolute answer, but a work around is: ex. “start emacs”.

• To network between Mac and PC, you have to turn on file sharing on both. On the Mac, this is done in System Preferences, Sharing, Windows Sharing. On Windows Vista, go to Control Panel, Network and Sharing Center. Then, for the folder you want to share, get the folder properties, Share tab. Once both machines has sharing on, you need to connect them to view. On the Mac, in Finder, under Go, Connect To Server, then Browse button. You can type a server address directly, e.g. “cifs://WORKGROUP;XAH@XAH-PC/USERS”. On Windows Vista, open a explorer, then type e.g. “\\\xah”. When Windows ask you for login name and password, the login name has this syntax: “username@domain” or “DOMAIN\username”.

• The file sharing speed seems to be exceeding slow. On Windows, when you transfer a folder to Mac, where the folder contains some 18k small files totaling 624 MB, the speed reported is 19KB/sec, about 9 fucking hours. If you do the copying of the same folder from Windows to Mac, using the Mac to do the copy, it reports about 1 hour (actualy time may be 40 min). Still seems too slow. There must be something wrong here.

On Windows Vista home edition, how to format a usb drive to NTFS?


• If you have a file on the Mac, who's file name contains Chinese chars. Then, you transfer it to PC thru networked file sharing. The file name will show up on PC something involving the ~ char. If you try to zip that dir on Windows using the bundled zip by right click, it'll report that it cannot due to that particular file name, and suggest you rename the file.

• Emacs issue. If you rename a file in emacsW32 dired, so that the new file name contains chinese. e.g. “林志玲.jpg”. Then, open the file in explorer, it will show up as “e978~1.jpg”.

• non-ascii char file name and networking issues. As far as i know, Mac uses utf-16 to encode file names (HFS+ file system), Windows also uses utf-16 ( NTFS), however, the encoding differ in minor details. The issue of file transfer of files with non-ascii names also depends on the transfer method. When shared thru Windows file sharing , it is done thru SMB/CIFS. Am not sure how that protocol handles file name transfer, but i do know, that as late as 2006, the open source Samba software used by the Mac for SMB/CIFS, have problems with unicode characters.

• Another method to transfer file is to zip it first. ZIP itself has problems with non-ascii characters. (i.e. as far as i know there are few variants of zip, some don't handle unicode well. As late as OS X 10.4.x, when you have a downloaded zip file containing chinese names, and you unzip thru Finder's BOMArchiveHelper, the Chinese names will become gibberish. The solution is to use The Unarchiver, which is used in OS X 10.5.x to replace BOMArchiveHelper.)

• Need to investigate how tar gzip handers files with unicode chars.

• Have not yet determined whether it is faster to transfer by by zip it first than transfer thru network, or directly thru network. Or, transfer thru usb drive, zipped or unzipped.

• If you have a Mac and PC, and without internet connection, you can network the 2 computers by just plug in a Ethernet crossover cable between them. (See: twisted pair Ethernet cable quote: “To connect two computers directly together without a switch, an Ethernet crossover cable is often used. Although many modern Ethernet host adapters can automatically detect another computer connected with a straight-through cable and then automatically introduce the required crossover, if needed; if neither of the computers have this capability, then a crossover cable is required. If both devices being connected support 1000BASE-T according to the standards, they will connect regardless of the cable being used or how it is wired.”)

I have a wireless local network. When transfering files say from Mac and Windows, it is faster if the network is connected by wired network such as ethernet. i.e. just plug in the cable between the 2 computers. However, if you already have the wireless network setup, and if you plug in a ethernet cable between the 2 machines, does the 2 machines actually use the physical cable to talk?

hi len. sometimes i find bugs, most prob just my unfamilarity. should i just chat to you or send a bug report to the emacs win list?

• Sometimes in Pidgin chat app, the friend list window will become blank. Probably whenever there's a network disruption (happens often with wireless). There does not seems to have a reconnect command. Restarting the app seems to work. Also, according to Wikipedia, Pidgin stores all your passwords in a plain text in your disk.

PC Problems

Some Strange Microphone Noise Problems

There's this strange audio problem. When i voice chat on Skype or in Second Life, there's this strange noise heard by those i'm chatting to. The noise gets worse whenever i move the mouse to drag a window or scroll page.

After few days of frustration and random guessing, i found these page on the web of similar problem:

Their problem is worse, because the noise came out from their speakers, even if they are not using the microphone. In my case, i'm using microphone to voice chat using Skype or Second Life to other people, and the noise is heard from their end.

Some suggest updating the sound card driver or mouse driver, some say use a usb mouse instead, most are random uninformed guesses.

The problem really have nothing to do with the mouse. If you move windows with keyboard. e.g. “Alt+Space m”, then hold down arrow keys to move the window, you still hear the noise.

It appears, that it is some sort of interference with the graphics card. No satisfactory solution i found. The most common work around is simply to mute the audio line-in (the microphone), which would not be useful for me. Other helpful work around is to turn the microphone boost off. For me, turning Noise Suppression on in the driver controller also helped. However, the problem is simply still there.

The problem with PC is that, you have all these random problems. They drain your time and energy.

The solution i found is that to plug my microphone in the rear, instead of front. Once i did that, the problem went away. The cause of the problem is possibly incorrect pin connection of the front mic port.

Graphics Card Problem

After i bought the PC for about 5 days, i was playing Second Life, all of a sudden, the system froze. Ctrl+Alt+Del has no effect. So, i hold down the Power key on the PC to force shut it down. Then, after 10 seconds, pressed the power key to start up again. I heard the Windows start up sound, but nothing on my screen. Apparently, the screen is not receiving any signal. I unplugged my DVI to the other port, no help. I plugged in the DVI into the motherboard, no help either. I restart, the machine several times, with the DVI plugged into either the motherboard or one of the 2 female DVI ports on the graphics card, and still nothing shows on the screen.

I was quite frustrated. I thought something is broken, but didn't know what. Friends suggest that return the machine. if i have to return the PC, that means one week of lost work. After about a hour of frustration and panic, i retried of plugging in the DVI to the motherboard, and started the machine. Lucky me, the screen showed up, using Windows's builtin graphics driver. For some unknown reason, my previous try of this didn't work.

So, finally at least i was able to access the system. I went to Device Manager, trying to find out what's going on, thinking that it is some kind of driver problem. After some more uninformed probe, i opened my PC, disconnected my graphics card, rebooted the PC, then, shut it down again, plug in the graphics card, connected the DVI to the card, and powered up the PC. Miraculously, it worked. I do not know what the fucking problem it was. At this point, i went and downloaded the latest Nvidia driver for my card, installed it, and rebooted ok.

(Note: The PC has the on-board graphics chip “AMD 780G Chipset with ATI Radeon™ HD 3200 Graphics”, bundled with ATI Catalyst Drivers. My graphics card is “NVIDIA GeForce 9800 GT”, bought separately. The driver i was using for my graphics card was dated Jan 2009, which came on a CD bundled with the card. The latest driver is dated 2009-04-30, version )

I don't know what is the exact cause of this problem. If it is a software problem of perhaps faulty Second Life software plus older version of graphics card driver, they shouldn't cause the graphics card to be disabled.

This caused me about 2 hours of trouble and panic. PC is full of these kind of problems in comparison to Apple. With Apple products, it works out of the box. You don't have to worry about some fucking “device drivers” or strange beeps. Part of the PC problem is of course due to less quality design, software or hardware, but part of it is due to its mass market with diverse components made by diverse companies. That is the reason, after all, it is much cheaper.