Hi there,
is it deliberate that a reply to a message with the subject header ...
Subject: Re: [TICKET #38797] Ankündigung Kündigung
... leaves the subject entirely empty??
Of course the subject does contain 8-bit chars and is therefore not encoded correctly (or to be more precise, it's not encoded at all), but is this a good reason to delete it completely upon replying?
In the message view pane, the subject is displayed as:
Re: [TICKET #38797] Ankndigung Kndigung
The 8-bit chars are deleted there. Wouldn't it be better to replace them with "?"...? And the same could/should be done upon replying, IMO.
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
Am 19.02.2013 21:37, schrieb Michael Heydekamp:
Hi there,
is it deliberate that a reply to a message with the subject header ...
Subject: Re: [TICKET #38797] Ankündigung Kündigung
... leaves the subject entirely empty??
Of course the subject does contain 8-bit chars and is therefore not encoded correctly (or to be more precise, it's not encoded at all), but is this a good reason to delete it completely upon replying?
PHP 5.4?
default behavior starting with 5.4 is idiotically return an empty string if the inout is not a valid UTF8 string and do not log any warning, well PHP upstream acts like idiots here since the core is still not unicode capable but thats how it works now
http://php.net/manual/en/function.htmlentities.php
Like htmlspecialchars(), htmlentities() takes an optional third argument encoding which defines encoding used in conversion. If omitted, the default value for this argument is ISO-8859-1 in versions of PHP prior to 5.4.0, and UTF-8 from PHP 5.4.0 onwards. Although this argument is technically optional, you are highly encouraged to specify the correct value for your code.
Am 19.02.2013 22:24, schrieb Reindl Harald:
Am 19.02.2013 21:37, schrieb Michael Heydekamp:
Hi there,
is it deliberate that a reply to a message with the subject header ...
Subject: Re: [TICKET #38797] Ankündigung Kündigung
... leaves the subject entirely empty??
Of course the subject does contain 8-bit chars and is therefore not encoded correctly (or to be more precise, it's not encoded at all), but is this a good reason to delete it completely upon replying?
PHP 5.4?
No, 5.3.3-7+squeeze14 with Suhosin-Patch (cli)
default behavior starting with 5.4 is idiotically return an empty string if the inout is not a valid UTF8 string and do not log any warning [...]
Well, but then this can't be the cause, right?
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
On Mon, Mar 25, 2013 at 11:14 PM, Michael Heydekamp listuser@freexp.de wrote:
Am 19.02.2013 22:24, schrieb Reindl Harald:
Am 19.02.2013 21:37, schrieb Michael Heydekamp:
Hi there,
is it deliberate that a reply to a message with the subject header ...
Subject: Re: [TICKET #38797] Ankündigung Kündigung
... leaves the subject entirely empty??
Of course the subject does contain 8-bit chars and is therefore not encoded correctly (or to be more precise, it's not encoded at all), but is this a good reason to delete it completely upon replying?
PHP 5.4?
No, 5.3.3-7+squeeze14 with Suhosin-Patch (cli)
Works for me with PHP 5.4.
default behavior starting with 5.4 is idiotically return an empty string if the inout is not a valid UTF8 string and do not log any warning [...]
Well, but then this can't be the cause, right?
It could, because the Subject indeed contains latin1 characters which are not correctly encoded.
It's a typical example of crappy email messages sent around from PHP scripts tackled together by stupid programmers. The message doesn't specify a Content-Type and no charset at all. And it nonetheless contains plain Latin1 characters in both the body and the subject.
In this case, Roundcube falls back to default_charset config option. The message renders correctly for me with $rcmail_config['default_charset'] = 'ISO-8859-1'; and also replying works fine in that case.
~Thomas
Am 26.03.2013 08:50, schrieb Thomas Bruederli:
On Mon, Mar 25, 2013 at 11:14 PM, Michael Heydekamp listuser@freexp.de wrote:
Am 19.02.2013 22:24, schrieb Reindl Harald:
PHP 5.4?
No, 5.3.3-7+squeeze14 with Suhosin-Patch (cli)
Works for me with PHP 5.4.
default behavior starting with 5.4 is idiotically return an empty string if the inout is not a valid UTF8 string and do not log any warning [...]
Well, but then this can't be the cause, right?
It could, because the Subject indeed contains latin1 characters which are not correctly encoded.
True (except that they're not encoded at all ather than being incorrectly encoded), but: If Harald is correct that PHP starting with 5.4 is returning an empty string, then PHP 5.3.3-7 can't be the cause, that's what I mean. And 5.3.3-7 can even be less the cause, if it works for you even with PHP 5.4.
It's a typical example of crappy email messages sent around from PHP scripts tackled together by stupid programmers.
Full ACK. The question still is how Roundcube should handle such broken messages.
The message doesn't specify a Content-Type and no charset at all. And it nonetheless contains plain Latin1 characters in both the body and the subject.
In this case, Roundcube falls back to default_charset config option. The message renders correctly for me with $rcmail_config['default_charset'] = 'ISO-8859-1'; and also replying works fine in that case.
Hmm, funny! Here it does still behave different, although 'default_charset' is set to 'ISO-8859-1' as well.
8bit chars are still deleted in the display, subject keeps empty upon replying.
Needs investigation... What can I do?
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
On 02/19/2013 09:37 PM, Michael Heydekamp wrote:
The 8-bit chars are deleted there. Wouldn't it be better to replace them with "?"...? And the same could/should be done upon replying, IMO.
What Roundcube version? What PHP version? What IMAP server? What OS? Provide a sample message source.
Am 20.02.2013 08:14, schrieb A.L.E.C:
On 02/19/2013 09:37 PM, Michael Heydekamp wrote:
The 8-bit chars are deleted there. Wouldn't it be better to replace them with "?"...? And the same could/should be done upon replying, IMO.
The main issue is that the subject is empty. Deletion of 8bit chars is just a side issue.
What Roundcube version?
0.9-git (as can be seen in the header of my message)
What PHP version?
5.3.3-7+squeeze14 with Suhosin-Patch (cli)
What IMAP server?
dovecot 1.2.15-7
What OS?
Win7/64 (if you mean the local OS)
Provide a sample message source.
Attached.
(BTW: All of a sudden I have an English keyboard while typing this message
- can Roundcube be the cause of that? I still have a German keyboard on OS
level and in Win apps such as Notepad...)
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
Am 21.02.2013 22:55, schrieb Michael Heydekamp:
(BTW: All of a sudden I have an English keyboard while typing this message
- can Roundcube be the cause of that? I still have a German keyboard on OS
level and in Win apps such as Notepad...)
Now I have a German keyboard again even within Roundcube - without having done anything. Strange...
I was watching football in another browser session while I was typing my previous message. Probably resources went way too low...?
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
As there was no response anymore to the mail below and as the issue does still persist with RC 1.0-git, may I ask: Is there any information that is still missing (and which I would be happy to provide)?
A reply to the message that I attached to my mail quoted below does still lead to an empty subject. Plus that the 8bit chars are deleted upon displaying the subject.
And please ignore the keyboard issue mentioned below, that was apparently only a local problem.
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
Am 21.02.2013 22:55, schrieb Michael Heydekamp:
Am 20.02.2013 08:14, schrieb A.L.E.C:
On 02/19/2013 09:37 PM, Michael Heydekamp wrote:
The 8-bit chars are deleted there. Wouldn't it be better to replace them with "?"...? And the same could/should be done upon replying, IMO.
The main issue is that the subject is empty. Deletion of 8bit chars is just a side issue.
What Roundcube version?
0.9-git (as can be seen in the header of my message)
What PHP version?
5.3.3-7+squeeze14 with Suhosin-Patch (cli)
What IMAP server?
dovecot 1.2.15-7
What OS?
Win7/64 (if you mean the local OS)
Provide a sample message source.
Attached.
(BTW: All of a sudden I have an English keyboard while typing this message
- can Roundcube be the cause of that? I still have a German keyboard on OS
level and in Win apps such as Notepad...)
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany _______________________________________________ Roundcube Development discussion mailing list dev@lists.roundcube.net http://lists.roundcube.net/mailman/listinfo/dev
On 03/25/2013 11:10 PM, Michael Heydekamp wrote:
A reply to the message that I attached to my mail quoted below does still lead to an empty subject. Plus that the 8bit chars are deleted upon displaying the subject.
I reproduced the "empty subject on reply" issue. I'll investigate more.
Am 26.03.2013 20:48, schrieb A.L.E.C:
On 03/25/2013 11:10 PM, Michael Heydekamp wrote:
A reply to the message that I attached to my mail quoted below does still lead to an empty subject. Plus that the 8bit chars are deleted upon displaying the subject.
I reproduced the "empty subject on reply" issue. I'll investigate more.
Thanks!
Are you also able to reproduce the display problem?
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
On Tue, Mar 26, 2013 at 8:48 PM, A.L.E.C alec@alec.pl wrote:
On 03/25/2013 11:10 PM, Michael Heydekamp wrote:
A reply to the message that I attached to my mail quoted below does still lead to an empty subject. Plus that the 8bit chars are deleted upon displaying the subject.
I reproduced the "empty subject on reply" issue. I'll investigate more.
I tracked it down to htmlspecialchars() in html::quote() used to quote the value attribute of the subject input field that returns an empty string if invalid characters are in the input.
The best (but very expensive) solution is to run every string through a charset validation function (whatever that might be) in order to verify that it doesn't contain invalid chars. Or maybe we can just do more sanity checks only for messages that do not specify charset information.
~Thomas
On 03/27/2013 08:42 AM, Thomas Bruederli wrote:
I tracked it down to htmlspecialchars() in html::quote() used to quote the value attribute of the subject input field that returns an empty string if invalid characters are in the input.
Confirmed, and this is not PHP 5.4 only.
The best (but very expensive) solution is to run every string through a charset validation function (whatever that might be) in order to verify that it doesn't contain invalid chars. Or maybe we can just do more sanity checks only for messages that do not specify charset information.
We can also:
- Use ENT_SUBSTITUTE when PHP 5.4 is detected.
- Use rcube_charset::clean() on headers input. This shouldn't be very
expensive.
Am 27.03.2013 08:42, schrieb Thomas Bruederli:
On Tue, Mar 26, 2013 at 8:48 PM, A.L.E.C alec@alec.pl wrote:
I reproduced the "empty subject on reply" issue. I'll investigate more.
I tracked it down to htmlspecialchars() in html::quote() used to quote the value attribute of the subject input field that returns an empty string if invalid characters are in the input.
Thanks for investigating! But hmm ... yesterday, you wrote:
In this case, Roundcube falls back to default_charset config option. The message renders correctly for me with $rcmail_config['default_charset'] = 'ISO-8859-1'; and also replying works fine in that case.
How did you manage that...?
As I already said, the default charset here is also ISO-8859-1, but neither is the subject "correctly" rendered nor does replying "work fine".
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
On 03/27/2013 11:20 PM, Michael Heydekamp wrote:
As I already said, the default charset here is also ISO-8859-1, but neither is the subject "correctly" rendered nor does replying "work fine".
Preferences > Displaying messages > Default Character Set > ISO-8859-1
Works for me too.
Am 28.03.2013 07:53, schrieb A.L.E.C:
On 03/27/2013 11:20 PM, Michael Heydekamp wrote:
As I already said, the default charset here is also ISO-8859-1, but neither is the subject "correctly" rendered nor does replying "work fine".
Preferences > Displaying messages > Default Character Set > ISO-8859-1
Works for me too.
Ah, that's it. ;) I defined it in main.inc.php, but wasn't aware that this is just a default for the user-configurable setting above. I changed this from UTF-8 to Windows-1252, and now it works here too.
But nonetheless, this issue still needs to be handled, right? Especially if a string can't by definition be a valid UTF-8 string, and if the setting above is set to UTF-8. In this case RC should fall back to the local 8bit charset (if RC can determine it). This would solve the problem of the empty subject automatically.
When does the setting above apply at all, BTW? Whenever a subject/body is lacking of a charset declaration...?
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
On 04/01/2013 09:45 PM, Michael Heydekamp wrote:
But nonetheless, this issue still needs to be handled, right?
Ticket created, http://trac.roundcube.net/ticket/1489032
When does the setting above apply at all, BTW? Whenever a subject/body is lacking of a charset declaration...?
Yes.
-
A.L.E.C -
Michael Heydekamp -
Reindl Harald -
Thomas Bruederli