Hi all,
today I met this problem:
Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="UTF-8"; name*="UTF-8''Rendel%C3%A9sek da Vincire legfrissebb.xls"; Content-Disposition: attachment; filename*="UTF-8''Rendel%C3%A9sek da Vincire legfrissebb.xls";
Do you know about this one?
There was a conversation about MIME problems around september. Thomas Bruederli mentioned switching to SwiftMail in this conversation, and also earlier when there were problems about SMTP handling.
Do you plan to switch to SwiftMail or did anybody take a deep look into it?
Bye, Balazs
List info: http://lists.roundcube.net/dev/
On Tue, Jan 13, 2009 at 7:00 PM, Balazs Horvath balazs.horvath@weboor.net wrote:
Hi all,
today I met this problem:
Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="UTF-8"; name*="UTF-8''Rendel%C3%A9sek da Vincire legfrissebb.xls"; Content-Disposition: attachment; filename*="UTF-8''Rendel%C3%A9sek da Vincire legfrissebb.xls";
Do you know about this one?
There was a conversation about MIME problems around september. Thomas Bruederli mentioned switching to SwiftMail in this conversation, and also earlier when there were problems about SMTP handling.
Do you plan to switch to SwiftMail or did anybody take a deep look into it?
Bye, Balazs
I'd rather not. ;-)
Do you know if you are using the fileinfo extension to "guess" the mimetype? Like, is it installed on your server (check in phpinfo()). Also, can you double-check the message source and let us know what the attachment is sent with?
Till _______________________________________________ List info: http://lists.roundcube.net/dev/
On Tue, Jan 13, 2009 at 19:00, Balazs Horvath balazs.horvath@weboor.net wrote:
Hi all,
today I met this problem:
Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="UTF-8"; name*="UTF-8''Rendel%C3%A9sek da Vincire legfrissebb.xls"; Content-Disposition: attachment; filename*="UTF-8''Rendel%C3%A9sek da Vincire legfrissebb.xls";
Do you know about this one?
There was a conversation about MIME problems around september. Thomas Bruederli mentioned switching to SwiftMail in this conversation, and also earlier when there were problems about SMTP handling.
Well, SwiftMail won't solve this issue for us since the mime headers have to be set manually (AFAIK).
Do you plan to switch to SwiftMail or did anybody take a deep look into it?
That server-side mime-type detection seems to cause more problems than solving them. I tend to switch back and just trust the mime-type submitted by the client (when uploading). It always worked well for me but I'm not using Windows...
~Thomas _______________________________________________ List info: http://lists.roundcube.net/dev/
On Jan 22, 2009, at 10:13 AM, Thomas Bruederli wrote:
I tend to switch back and just trust the mime-type submitted by the client (when uploading).
Hmmm, that made my security radar do a beep. Trusting user input isn't always safe.
An example with non-serious consequences : In a application I built, I had a simple check for a file type, and refused uploads if the file didn't meet our requirements. Users would simply re-name the file with a different extension, which would make the browser report the required mime type.
I had to add a check that used something like ' file -ib ' to make sure something goofy wasn't going on.
It seems fileinfo - libmagic has a bit of a hard time with Microsoft
formats,
which are common e-mail file attachments. This may be why
the comment :
That server-side mime-type detection seems to cause more problems than solving them.
My application referenced above required a file type far different from Word, Excel, etc. so I didn't run into problems.
Thomas Bruederli, 09.01.22 17:13:
(...)
That server-side mime-type detection seems to cause more problems than solving them. I tend to switch back and just trust the mime-type submitted by the client (when uploading). It always worked well for me but I'm not using Windows...
~Thomas
I'm digging more deep and deep in this attachment problem.
fileinfo
it is for detecting mime type by so called magic numbers
on the same server, when calling file -bi and finfo_open on the same
doc file it gives you different results
file Munka1.xls
Munka1.xls: Microsoft Office Document
/usr/php5fcgi/bin/php-cgi fileinfo.php
Content-type: text/html
application/octet-stream
So file gives you right, but finfo_open does not.
Actually for docx files it will always detect ZIP as format, because it's actually a zip, and this will never change!
IMHO we can keep using fileinfo, but we need to combine it with
- extension based method
- using the client submitted mime-type
I like arekm's solution (http://trac.roundcube.net/attachment/ticket/1485311/mime-detect.patch) what if we put the list to a external config file, so this can be an option. If admin wants, he can override the detection of some extensions.
What do you think?
Balazs
List info: http://lists.roundcube.net/dev/
On Wed, Mar 4, 2009 at 1:54 PM, Balazs Horvath balazs.horvath@weboor.net wrote:
Thomas Bruederli, 09.01.22 17:13:
(...)
That server-side mime-type detection seems to cause more problems than solving them. I tend to switch back and just trust the mime-type submitted by the client (when uploading). It always worked well for me but I'm not using Windows...
It has nothing to do with Windows, or Mac, or Linux or OS/2 Warp. It's a matter of security that clients can say whatever they want when they upload a file.
(...) I like arekm's solution (http://trac.roundcube.net/attachment/ticket/1485311/mime-detect.patch) what if we put the list to a external config file, so this can be an option. If admin wants, he can override the detection of some extensions.
I like this patch as well, though security "concious" people might scream. ;-( I wonder if Fileinfo is configured correctly (it takes a magic file) and maybe that's the issue? Otherwise I wouldn't know why it doesn't work while a simple file does. They should use the same magic database in the end.
Till _______________________________________________ List info: http://lists.roundcube.net/dev/
Otherwise I wouldn't know why it doesn't work while a simple file does. They should use the same magic database in the end.
First, trusting the mime-type from the client is very dangerous.
On Fedora, the php-pear-Fileinfo package uses the same magic file
that the command line file command uses.
If you compile the fileinfo extension separately, you might be using
a different magic file, which might provide inconsistent results.
I have seen /usr/share/misc/magic ( default for fileinfo upstream )
and /usr/share/file/magic ( default for Fedora ).
You also need to use the same flags or options.
Using the PHP fileinfo functions, there are options you pass
( FILEINFO_MIME ) in order to get the output you want.
Using the file command, there are flags you pass ( -i ) in order to
get the output you want.
The defaults are different between the two.
For example, these options should give the same between <?php finfo_file() ?> and file from the command line -
file -biLz filename finfo_file($finfo,"filename",1046)
You need to add the integer values of the FILEINFO_* options
-b is the default for finfo_file() -i == FILEINFO_MIME or 1040 -L == FILEINFO_SYMLINK or 2 -z == FILEINFO_COMPRESS or 4
-k == FILEINFO_CONTINUE or 32, which is mutually exclusive of
FILEINFO_MIME
That is, you can't use FILEINFO_CONTINUE and FILEINFO_MIME together,
is is either one of the other.
Sometimes using FILEINFO_CONTINUE and parsing that string gives you
more information than FILEINFO_MIME does, but that parsing is harder
and more complicated.
if finfo_file($finfo,"filename",1040) returns a gzip or zip mime-type, you could then do
finfo_file($finfo,"filename",1044)
to see what type of data is inside that compressed file.
HTH
On Wed, Mar 4, 2009 at 4:56 PM, chasd chasd@silveroaks.com wrote:
Otherwise I wouldn't know why it doesn't work while a simple file does. They should use the same magic database in the end.
First, trusting the mime-type from the client is very dangerous.
On Fedora, the php-pear-Fileinfo package uses the same magic file that the command line file command uses. If you compile the fileinfo extension separately, you might be using a different magic file, which might provide inconsistent results. I have seen /usr/share/misc/magic ( default for fileinfo upstream ) and /usr/share/file/magic ( default for Fedora ).
You also need to use the same flags or options. Using the PHP fileinfo functions, there are options you pass ( FILEINFO_MIME ) in order to get the output you want. Using the file command, there are flags you pass ( -i ) in order to get the output you want.
The defaults are different between the two.
For example, these options should give the same between <?php finfo_file() ?> and file from the command line -
file -biLz filename finfo_file($finfo,"filename",1046)
You need to add the integer values of the FILEINFO_* options
-b is the default for finfo_file() -i == FILEINFO_MIME or 1040 -L == FILEINFO_SYMLINK or 2 -z == FILEINFO_COMPRESS or 4
-k == FILEINFO_CONTINUE or 32, which is mutually exclusive of FILEINFO_MIME That is, you can't use FILEINFO_CONTINUE and FILEINFO_MIME together, is is either one of the other. Sometimes using FILEINFO_CONTINUE and parsing that string gives you more information than FILEINFO_MIME does, but that parsing is harder and more complicated.
if finfo_file($finfo,"filename",1040) returns a gzip or zip mime-type, you could then do
finfo_file($finfo,"filename",1044)
to see what type of data is inside that compressed file.
HTH
Maybe we need a wiki page.
@Charles, would you be so kind and start on one? I'd contribute some findings as well.
This also sounds like a great opportunity for a couple unit tests. I'll make myself a mental note for that.
Till _______________________________________________ List info: http://lists.roundcube.net/dev/
On Mar 4, 2009, at 10:56 AM, till wrote:
Maybe we need a wiki page.
@Charles, would you be so kind and start on one? I'd contribute some findings as well.
Tell me where you'd like it, looking at the existing structure, a
logical place doesn't pop out at me.
chasd, 09.03.04 18:58:
On Mar 4, 2009, at 10:56 AM, till wrote:
Maybe we need a wiki page.
@Charles, would you be so kind and start on one? I'd contribute some findings as well.
Tell me where you'd like it, looking at the existing structure, a
logical place doesn't pop out at me.
till, chasd
To the wiki, and to better understading of the problem I recommend this blog post: http://gaarai.com/2009/02/14/generating-mime-type-in-php-is-not-magic/
The guy is still working on it, I gave hime some hints, and will give more after I experiment with the info on fileinfo provided by chasd. (Hey man, how do you know that? I couldn't find that info in any documentation!)
I mentioned a patch: http://trac.roundcube.net/attachment/ticket/1485311/mime-detect.patch this was the base, but in the meantime I started using Trac more extensively, so:
The same proposal I made, and description of problem: http://trac.roundcube.net/ticket/1485311
And the resolution by thomasb http://trac.roundcube.net/changeset/2313 with a hard-coded map, they way I was thinking.
So as I said I'll start digging deeper and experimenting with fileinfo, to get the best solution we can.
chasd, you wrote "Hmmm, that made my security radar do a beep. Trusting user input isn't always safe." "First, trusting the mime-type from the client is very dangerous. "
But guys, the security part of this is out of the scope of RC I think... If the user sends something bogus by playing with the extension, who cares? There are so many ways to do that without RC. Okay, we should try helping SPAM and VIRUS filters, but this is their task IMHO.
Balazs
List info: http://lists.roundcube.net/dev/
Balazs Horvath wrote:
chasd, you wrote "Hmmm, that made my security radar do a beep. Trusting user input isn't always safe." "First, trusting the mime-type from the client is very dangerous. "
But guys, the security part of this is out of the scope of RC I think... If the user sends something bogus by playing with the extension, who cares? There are so many ways to do that without RC. Okay, we should try helping SPAM and VIRUS filters, but this is their task IMHO.
The problem might just be that the server itself is at risk depending on what happens....
Best regards,
Michael
On Thu, Mar 5, 2009 at 12:46, Michael Baierl mail@mbaierl.com wrote:
But guys, the security part of this is out of the scope of RC I think... If the user sends something bogus by playing with the extension, who cares? There are so many ways to do that without RC. Okay, we should try helping SPAM and VIRUS filters, but this is their task IMHO.
The problem might just be that the server itself is at risk depending on what happens....
What risk do you mean? The doesn't touch these files (except fileinfo).
~Thomas _______________________________________________ List info: http://lists.roundcube.net/dev/
On Thu, Mar 5, 2009 at 14:35, Thomas Bruederli roundcube@gmail.com wrote:
On Thu, Mar 5, 2009 at 12:46, Michael Baierl mail@mbaierl.com wrote:
But guys, the security part of this is out of the scope of RC I think... If the user sends something bogus by playing with the extension, who cares? There are so many ways to do that without RC. Okay, we should try helping SPAM and VIRUS filters, but this is their task IMHO.
The problem might just be that the server itself is at risk depending on what happens....
What risk do you mean? The doesn't touch these files (except fileinfo).
Correction: "The server doesn't touch these files" Sorry!
~Thomas _______________________________________________ List info: http://lists.roundcube.net/dev/
Then it should be fine and RC should be ok with just doing any kind of detection, maybe even just based on the extension... If unknown files are sent as binary it will be offered as download, so it should not be a big deal...
And even using fileinfo might be a security risk for the server, as there have been security issues in that package as well :)
Best regards,
Michael
Thomas Bruederli wrote:
On Thu, Mar 5, 2009 at 14:35, Thomas Bruederli roundcube@gmail.com wrote:
On Thu, Mar 5, 2009 at 12:46, Michael Baierl mail@mbaierl.com wrote:
But guys, the security part of this is out of the scope of RC I think... If the user sends something bogus by playing with the extension, who cares? There are so many ways to do that without RC. Okay, we should try helping SPAM and VIRUS filters, but this is their task IMHO.
The problem might just be that the server itself is at risk depending on what happens....
What risk do you mean? The doesn't touch these files (except fileinfo).
Correction: "The server doesn't touch these files" Sorry!
~Thomas
On Mar 5, 2009, at 4:57 AM, Balazs Horvath wrote:
http://gaarai.com/2009/02/14/generating-mime-type-in-php-is-not-magic/
My summary :
" None of the hosting providers I use has the right fileinfo
software, nor can I install it. "
(Hey man, how do you know that? I couldn't find that info in any documentation!)
The man page for the " file " command has that documentation, I guess
I am quite familiar with that command.
I'd used a PHP exec() to the file command for file type detection
since PHP 3 ( which kinda dates me ).
I simply matched up the file command flags to the predefined
constants for the fileinfo options
http://www.php.net/manual/en/fileinfo.constants.php
A few sample files and some sample PHP code produced the information
I posted earlier.
I just tried .docx, .odt, and .ods -
Using options "1046" : docx : application/xml compressed-encoding=application/zip ods : text/plain charset=us-ascii compressed-encoding=application/ octet-stream odt : text/plain charset=us-ascii compressed-encoding=application/ vnd.oasis.opendocument.text
Using options "38" : docx : XML document text ( Zip archive data, at least v2.0 to extract) ods : ASCII text, with no line terminators (OpenDocument Spreadsheet) odt : ASCII text, with no line terminators (OpenDocument Text)
Those return strings seem to identify the file types fairly
conclusively.
If you find the file type is a zip file using "normal" 1040 options,
poke at it again with different options.
I find that opening the magic file with no options allows you to
probe the file multiple times using different options, but you have
to remember to specify the options at probe time instead of assuming
the options you want have been globally specified.
Maybe because I got burned on file type issues in the past I am
sensitive to it ( and was forced to learn about it in detail ).
Looking at the upstream fileinfo mail list, newer versions might be
able to better determine Office 2007 file types.
http://mx.gw.com/pipermail/file/2009/000311.html
However my test of a Fedora 11 rpm rebuilt on F10 didn't show any
improvement.
Another interesting thread -
http://mx.gw.com/pipermail/file/2008/000283.html
BTW :
OpenOffice.org uses a standard file format :
http://www.idpf.org/ocf/ocf1.0/download/ocf10.htm
This is the same format used by Adobe for Mars files.
If you explode the zip, there is a "mimetype" file at the root level
with the mime-type inside.
The fileinfo library can see that in a odt but for some reason not a
ods ( or a mars ).
Not sure if code to peek inside the zip for that mimetype file is
worthwhile.
MS uses something similar to but not the same, <snarky>typical of MS</ snarky>
http://en.wikipedia.org/wiki/Open_Packaging_Convention
If the user sends something bogus by playing with the extension,
who cares?
I think passing the security buck to some other part of the system
isn't good practice. If you look at the OWASP site at all, the
preferred way is to validate and test all input _and_ output.
-
Balazs Horvath -
chasd -
Michael Baierl -
Thomas Bruederli -
till