Hey Maria!

I've done this before but I believe it needs repeating so here goes. Please find below the file squirrels.utf8 which is originally was a Fidonet posting in
this echo way back when.

----- squirrels.utf8 starts
UTF-8 messaging can be fun for the whole family, especially Fidonet.

Case in point is when you find squirrels in your pants and lack a suitable search for a solution within the Fidonet family. With UTF-8 enabled then one could send a single message with, "I've got squirrels in my pants", "Jag har ekorrar i mina byxor", "-u -+-|-+-A -# -e-e-#-+-#-a -#-|-+-|-#", "+e-c-e -a+|+>++-a-U+++| -a-a++ -C+#++-a+|++-i+++| ++++-a" and "Ich habe Eichh||rnchen in meiner Hose", within a single message which covers five languages thus vastly increasing your coverage when searching for a solution to this most perplexing situation.

So, what are you waiting for? Enable UTF-8 Fidonet text messaging today and join in on the fun.
----- squirrels.utf8 ends

As shown by;

----- '<Esc>:read !wc --bytes < squirrels.utf8' starts
721
----- '<Esc>:read !wc --bytes < squirrels.utf8' ends

and,

----- '<Esc>:read !wc --chars < squirrels.utf8' starts
677
----- '<Esc>:read !wc --chars < squirrels.utf8' ends

there are exactly 44 multibyte utf-8 characters within that 'message'. Supposing a CHRS control line of "CHRS: UTF-8 4", which would be an accurate control line for the above text, what possibly would/could/should be the conversion possible? CP437? Let us try that;

----- '<Esc>:read !iconv -f utf8 -t cp437 -c squirrels.utf8 | tr -d '\0-\177' |
wc --bytes' starts
8
----- '<Esc>:read !iconv -f utf8 -t cp437 -c squirrels.utf8 | tr -d '\0-\177' |
wc --bytes' ends

In other words only 8 of the 44 utf-8 characters could be converted to CP437. That is very lossy. Let us try CP866 given that one of the languages is Russian;

----- '<Esc>:read !iconv -f utf8 -t cp866 -c squirrels.utf8 | tr -d '\0-\177' |
wc --bytes' starts
17
----- '<Esc>:read !iconv -f utf8 -t cp866 -c squirrels.utf8 | tr -d '\0-\177' |
wc --bytes' ends

Better but still less than half which is still too lossy.

Do you or anyone else require more proof that converting utf-8 text is a bad idea and thus a CHRS control line is total waste of bytes when considering it as a variable for a conversion routine?

Also, while we're at it, I suggest that we let the heathen bastards eat cake. :::evil grin:::

Life is good,
Maurice

... Don't cry for me I have vi.
--- GNU bash, version 4.4.12(1)-release (x86_64-unknown-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hey Maurice!

----- '<Esc>:read !wc --bytes < squirrels.utf8' starts
----- '<Esc>:read !wc --chars < squirrels.utf8' starts

The above could have been made leaner and meaner using, tr -d '\0-\177' < squirrels.utf8 | wc --chars' which produces the correct answer of 44 multibyte characters.

Life is good,
Maurice

--- BBBS/Li6 v4.10 Toy-3
* Origin: Prism bbs (1:261/38)

Hey Maurice!

tr -d '\0-\177' < squirrels.utf8 | wc --chars

Excellent. Note I dropped the ' character from the end. I suspect that was a typo.

I suspect that this is all a one-shot-deal just to prove a point but we both know that using tr to delete certain characters comes in EXTREMELY handy at times. tr is most definetly a keeper.

Life is good,
Maurice

... Don't cry for me I have vi.
--- GNU bash, version 4.4.12(1)-release (x86_64-unknown-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hello Maurice!

26 May 2017 23:47, Maurice Kinal wrote to Maria Antonia Josepha Johanna:

----- '<Esc>:read !iconv -f utf8 -t cp437 -c squirrels.utf8 | tr -d '\0-\177' | wc --bytes' starts

here you tell iconv to use utf8 source, this can be default in ftn, but it is a
fail to asssume it, if that was easy you would not need -f

without -f iconv only see 7bit


Regards Benny

... there can only be one way of life, and it works :)

--- Msged/LNX 6.2.0 (Linux/4.11.2-gentoo (i686))
* Origin: openvpn on its way here (2:230/0)

Hey Benny!

here you tell iconv to use utf8 source

That is because the source *IS* utf8.

I finetuned that a bit so that wc will count legitimate 8 bit characters that are not utf-8 as characters rather then bytes. It looks like this now;

iconv -f utf8 -t cp437 -c squirrels.utf8 | \
tr -d '\0-\177' | \
LC_ALL=C wc --chars

Note the LC_ALL=C setting will reset wc to count 8 bit characters as characters
on native utf-8 systems and won't hurt anything on actual 8 bit systems so it is perfectly safe to use even in scripts.

BTW if you change the cp437 to latin-1 you should get an error since of course there is no such alias as far as iconv is concerned. In fact you should get this output;

----- output for bogus latin-1 alias starts
iconv: conversion to `latin-1' is not supported
Try `iconv --help' or `iconv --usage' for more information.
0
----- output for bogus latin-1 alias ends

Of course latin1 will work as it should;

----- output for legitimate latin1 alias starts
1
----- output for legitimate latin1 alias ends

Not too many latin1 characters in that document given there are exactly 44 utf-8 characters.

Are you beginning to see the light?

Life is good,
Maurice

... Don't cry for me I have vi.
--- GNU bash, version 4.4.12(1)-release (x86_64-unknown-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)