Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== SED - Special Characters - sed behaving strangely in UTF-8 environment ======

===== Your problem =====

You are using a Linux distribution with UTF-8 encoding.

You are using sed to operate on files containing German Umlauts or other non-Ascii characters.

sed is behaving quite strangly.  An expression like the following normally should replace an arbitrary string by a single x.  The dot, however, does not match non-Ascii characters any more!

<code bash>
sed 's/.*/x/'
</code>

----

===== The reason =====

The problem occurs if you operate on ISO-8859 (Latin) encoded files.

A non-ascii character is misinterpreted in UTF-8 as a sequence of characters or - even worse - as an invalid UTF-8 string.

So sed classifies the character as something not being matched by a dot.  Strange and dangerous...

----

===== Solution =====

Converting your system back from UTF-8 to ISO-8859 seems not to be a good solution.

A problem similar to the upper one would occur then when you operate with UTF-8 files.

Better use **iconv** to convert the data on the fly:


<code bash>
iconv -f latin1 -t utf-8 sourcefile | sed 's/.*/x/' | iconv -f utf-8 -t latin1

iconv -f ISO-8859-15 -t utf-8 espanol.txt | sed -n '/[[:graph:][:space:]]\{5\}/!p' | iconv -f utf-8 -t ISO-8859-15
</code>