function manualConvertUnicodeToUTF8 should do proper conversion to UTF8

[MartinBudden]

Currently the function just replaces non-ascii characters with their HTML entity encoding (eg \u0150 -> –).

[EricShulman]

I just tried this patch in a copy of TW2.3.0 (by manually editing the core source to replace the previous function definition). I then opened the modified file in IE7, and clicked on 'save changes'. After the displayMessage() that reports "backup file saved", IE7 shows a browser error box:

"Invalid procedure call or argument"

pressing OK then shows an alert() box:

"Failed to save main TiddlyWiki file. Your changes have not been saved"

Then, upon checking the files in the directory, I note that, while the *backup* file is OK, the main file has been truncated to 0 bytes!!!! Uh oh! :-(

Note that manualConvertUnicodeToUTF8() will be invoked by *any* browser in which "window.netscape" is undefined. Of course, this is intended to identify when IE is in use, but may affect other browsers as well and should therefore be tested in as many browsers and OS combinations as possible, including:

IE6, IE7, IE8(beta), Safari, Opera, Opera9.5 (beta)

Because this change has a pervasive impact on the encoding and saving of tiddler content (as well as the core code), and can result in a corrupted (or zeroed) file, it is vitally important that it be thoroughly tested prior to the release of TW2.3.1. Nonetheless, despite the very serious problem noted above, I think that this change should still be included in TW2.3.1.

[EricShulman]

After tracing the code via hand-inserted alerts(), it appears that the failure point is in ieSaveFile().

Specifically, this line:

file.Write(content);

fails with the reported "invalid procedure call or argument".

I think this is because the IE file I/O can only write "text" files, which appears to be limited to "low ASCII" codes only. (i.e., <= 0x7F). This may be more of a problem than previously suspected... and may mean holding off on this patch until true binary file I/O can be implemented for use with IE :-(

[EricShulman]

Due to the inability of Internet Explorer to write UTF8 to a local file, I've changed the milestone for this ticket from "2.3.1" to "soon" so that we can consider other approaches.