Hello list:
After resolving several issues in LDAP with your help (see other threads), I am now confident to move all contacts to a private LDAP address book.
In that process, I realised that the export/import functionality of RC does not seem to work 100%, as exporting works, but re-importing the resulting file into RC does not!
How to reproduce:
single rcube_contacts.vcf file with all contacts in it) 3) Select a new empty (e.g. LDAP) address book 4) "Import Contacts", and select the single VCF file created in step 2) above 5) The import process fails
Expected result: successful re-import of all contacts that we exported before into a new empty address book. Actual result: "Successfully imported 0 contacts."
Importing several VCF files that contain only 1 contact per VCF seem to work fine, but of course manually importing individual files is not practical beyond a handful of contacts...
Best regards, Achim
On 12.04.2012 01:57, Achim wrote:
- Open Address Book
- Select several contacts and export to a VCF file (results in a single
rcube_contacts.vcf file with all contacts in it) 3) Select a new empty (e.g. LDAP) address book 4) "Import Contacts", and select the single VCF file created in step 2) above 5) The import process fails
Expected result: successful re-import of all contacts that we exported before into a new empty address book. Actual result: "Successfully imported 0 contacts."
Importing several VCF files that contain only 1 contact per VCF seem to work fine, but of course manually importing individual files is not practical beyond a handful of contacts...
Any errors in log? Would be good to see this file and your config.
Hello Alec:
On 12.04.2012 08:13, A.L.E.C wrote:
Any errors in log? Would be good to see this file and your config.
No errors, this is from logs/ldap:
[12-Apr-2012 10:34:53 +0200]: C: Connect [ldap.calendar.tld:389] [12-Apr-2012 10:34:53 +0200]: S: OK [12-Apr-2012 10:34:53 +0200]: C: Bind [dn: cn=user@example.com,ou=private,ou=rcabook,dc=domain,dc=tld] [pass: ****************] [12-Apr-2012 10:34:53 +0200]: S: OK [12-Apr-2012 10:34:53 +0200]: C: Search [(&(objectClass=inetOrgPerson)(|(mail=john.doe@example.com)))][dn: cn=user@example.com,ou=private,ou=rcabook,dc=domain,dc=tld] [12-Apr-2012 10:34:53 +0200]: S: 0 record(s) [12-Apr-2012 10:34:53 +0200]: C: Search [(&(objectClass=inetOrgPerson)(|(cn=John Does)))][dn: cn=user@example.com,ou=private,ou=rcabook,dc=domain,dc=tld] [12-Apr-2012 10:34:53 +0200]: S: 0 record(s) [12-Apr-2012 10:34:53 +0200]: C: Search [(&(objectClass=inetOrgPerson)(|(mail=paul.smith@example.com)))][dn: cn=user@example.com,ou=private,ou=rcabook,dc=domain,dc=tld] [12-Apr-2012 10:34:53 +0200]: S: 0 record(s) [12-Apr-2012 10:34:53 +0200]: C: Search [(&(objectClass=inetOrgPerson)(|(cn=Paul Smith)))][dn: cn=user@example.com,ou=private,ou=rcabook,dc=domain,dc=tld] [12-Apr-2012 10:34:53 +0200]: S: 0 record(s) [..]
for all n contacts in the VCF file.
During a successul import, after "Search" and "0 records", I can see an "Add" command:
[12-Apr-2012 10:39:19 +0200]: C: Search [(&(objectClass=inetOrgPerson)(|(mail=joe.sixpack@example.com)))][dn: cn=user@example.com,ou=private,ou=rcabook,dc=domain,dc=tld] [12-Apr-2012 10:39:19 +0200]: S: 0 record(s) [12-Apr-2012 10:39:19 +0200]: C: Search [(&(objectClass=inetOrgPerson)(|(cn=Joe Sixpack)))][dn: cn=user@example.com,ou=private,ou=rcabook,dc=domain,dc=tld] [12-Apr-2012 10:39:19 +0200]: S: 0 record(s) [12-Apr-2012 10:39:19 +0200]: C: Add [dn: cn=Joe Sixpack,cn=user@example.com,ou=private,ou=rcabook,dc=domain,dc=tld]: Array( [cn] => Joe Sixpack [sn] => Sixpack [givenname] => Joe [mail] => Array ( [0] => joe.sixpack@example.com ) [objectClass] => Array ( [0] => top [1] => inetOrgPerson ) )
I can send you the short multi-card VCF off list if that helps.
Best regards, Achim
Hello again:
The exports from my RC SQL address book look like this:
BEGIN:VCARD VERSION:3.0 FN:John Doe EMAIL;type=INTERNET:john.doe@example.com END:VCARD
Importing that kind of card fails silently.
Manually adding the N line and editing the email line to read like this (adding ";type=WORK")
N:Doe;John;;; EMAIL;type=INTERNET;type=WORK:john.doe@example.com
so that the final VCF looks like this:
BEGIN:VCARD VERSION:3.0 N:Doe;John;;; FN:John Doe EMAIL;type=INTERNET;type=WORK:john.doe@example.com END:VCARD
makes the import from a default RC sql export work. Two observations:
exist (using FN)? 2) My fieldmap has 'email:work' => 'mail', which explains the requirement for type=WORK. Is it possible to map several field (HOME, ...) to the same destination LDAP field?
Thanks, Achim
On 12.04.2012 12:02, Achim wrote:
Hello again:
The exports from my RC SQL address book look like this:
BEGIN:VCARD VERSION:3.0 FN:John Doe EMAIL;type=INTERNET:john.doe@example.com END:VCARD
Importing that kind of card fails silently.
Manually adding the N line and editing the email line to read like this (adding ";type=WORK")
N:Doe;John;;; EMAIL;type=INTERNET;type=WORK:john.doe@example.com
so that the final VCF looks like this:
BEGIN:VCARD VERSION:3.0 N:Doe;John;;; FN:John Doe EMAIL;type=INTERNET;type=WORK:john.doe@example.com END:VCARD
makes the import from a default RC sql export work. Two observations:
The problem is with 'required_fields' in addressbook config. Also when handling VCARD data FN is mapped to displayname.
Maybe we can modify rcube_ldap::validate() to fill 'name' with displayname, if it's empty but 'displayname' exists. I'm not sure if it's all, this can be config dependent.
You should create a ticket for this.
On 12.04.2012 12:13, A.L.E.C wrote:
The problem is with 'required_fields' in addressbook config. Also when handling VCARD data FN is mapped to displayname.
Maybe we can modify rcube_ldap::validate() to fill 'name' with displayname, if it's empty but 'displayname' exists. I'm not sure if it's all, this can be config dependent.
My mistake, FN => 'displayname' => 'name'. So the problem is different. On import rcube_ldap::validate() with autofix argument enabled is not called. Maybe we just should use it before import().
On 12.04.2012 12:33, A.L.E.C wrote:
The problem is with 'required_fields' in addressbook config. Also when handling VCARD data FN is mapped to displayname.
Maybe we can modify rcube_ldap::validate() to fill 'name' with displayname, if it's empty but 'displayname' exists. I'm not sure if it's all, this can be config dependent.
My mistake, FN => 'displayname' => 'name'. So the problem is different. On import rcube_ldap::validate() with autofix argument enabled is not called. Maybe we just should use it before import().
r6067 should fix this issue. The problem was also with EMAIL;type=INTERNET which is recognized as email:other.
Hello Alec:
On 12.04.2012 13:53, A.L.E.C wrote:
On 12.04.2012 12:33, A.L.E.C wrote:
The problem is with 'required_fields' in addressbook config. Also when handling VCARD data FN is mapped to displayname.
Maybe we can modify rcube_ldap::validate() to fill 'name' with displayname, if it's empty but 'displayname' exists. I'm not sure if it's all, this can be config dependent.
My mistake, FN => 'displayname' => 'name'. So the problem is different. On import rcube_ldap::validate() with autofix argument enabled is not called. Maybe we just should use it before import().
r6067 should fix this issue. The problem was also with EMAIL;type=INTERNET which is recognized as email:other.
So I assume that this also solves the following situation (which I am also going to test with f3isar's configuration):
I exported a single contact from SQL like this:
BEGIN:VCARD VERSION:3.0 N:Last;First;;; FN:First Last EMAIL;type=INTERNET;type=WORK:first.last.work@example.com EMAIL;type=INTERNET:first.last@example.com TEL;type=MOBILE:1234 URL:http://acme.com ADR:;;Park Av 100;NY;;12345; ORG:Org END:VCARD
Importing back into an *SQL* addressbooks works as expected, all fields are read and populated.
Importing the same record back into an LDAP directory, some values are imported:
but several others are not (see below for logs):
email field in my fieldmap)
Extract of my fieldmap: 'email:work' => 'mail', 'website' => 'labeledURI', 'street' => 'street', 'zipcode' => 'postalCode', 'locality' => 'l', 'region' => 'st',
I am aware that this is probably due to a mapping issue (or rather restricton), but could somebody point out what to change?
This functionality is fundamental for migrating private SQL address books to LDAP, any help would be appreciated.
Thanks, Achim
Extract from logs/ldap
[12-Apr-2012 14:03:15 +0200]: C: Add [dn: cn=First Last,cn=user@domain.tld,ou=private,ou=rcabook,dc=domain,dc=tld]: Array ( [cn] => First Last [sn] => Last [givenname] => First [mail] => Array ( [0] => first.last@example.com )
[mobile] => Array
(
[0] => 1234
)
[o] => Org
[objectClass] => Array
(
[0] => top
[1] => inetOrgPerson
)
)
[12-Apr-2012 14:03:15 +0200]: S: OK [12-Apr-2012 14:03:15 +0200]: C: Close
On 18.04.2012 00:37, Achim wrote:
- Email other (might be a inetOrgPerson restriction, there is only 1
email field in my fieldmap)
There's no attribute for other email address in inetOrgPerson schema.
- Address other
This is something we could fix. Open a ticket, please.
- URL other
There's no attribute for URL in inetOrgPerson schema.
Am 2012-04-18 08:49, schrieb A.L.E.C:
On 18.04.2012 00:37, Achim wrote:
- Email other (might be a inetOrgPerson restriction, there is only 1
email field in my fieldmap)
There's no attribute for other email address in inetOrgPerson schema.
No, this is wrong! -> the mail attribute in inetOrgPerson is of type MULTIVALUE (like the most other attributes as well, see the schema file), this means, you can have more than one mail address just by adding another one. -> RC today is allready showing all email addresses in this case, but is not yet able to write them back (or even to create them in the case of import
I allready discussed this topic with Thomas, his proposition was to expand the fieldmap with an array size in the case of multivalue type... but this is not yet done :-)
- Address other
This is something we could fix. Open a ticket, please.
- URL other
There's no attribute for URL in inetOrgPerson schema.
inetOrgPerson is missing some attributes that are used in RC (or even in VCARD) like "c" (country), birthday, ... One (my) solution is to create a new "myPerson" class based on inetOrgPerson, but this is not possible if you have no control over the server... then you have to ask the servers admin for a solution.
Andreas
On 18.04.2012 14:10, Andreas Dick wrote:
Am 2012-04-18 08:49, schrieb A.L.E.C:
On 18.04.2012 00:37, Achim wrote:
- Email other (might be a inetOrgPerson restriction, there is only 1
email field in my fieldmap)
There's no attribute for other email address in inetOrgPerson schema.
No, this is wrong! -> the mail attribute in inetOrgPerson is of type MULTIVALUE (like the most other attributes as well, see the schema file), this means, you can have more than one mail address just by adding another one. -> RC today is allready showing all email addresses in this case, but is not yet able to write them back (or even to create them in the case of import
I allready discussed this topic with Thomas, his proposition was to expand the fieldmap with an array size in the case of multivalue type... but this is not yet done :-)
But you still need to assign address type, which would be hard. You can't say let first address be 'home', second 'work', third 'other', etc.
We can simply fix import by extending validate() method and writing all emails into 'mail'.
You are right! Is it possible in VCARD? Then only a extended class (in a nonstandart schema) will help!
"A.L.E.C" alec@alec.pl schrieb:
On 18.04.2012 14:10, Andreas Dick wrote:
Am 2012-04-18 08:49, schrieb A.L.E.C:
On 18.04.2012 00:37, Achim wrote:
- Email other (might be a inetOrgPerson restriction, there is only
1
email field in my fieldmap)
There's no attribute for other email address in inetOrgPerson
schema.
No, this is wrong! -> the mail attribute in inetOrgPerson is of type MULTIVALUE (like
the
most other attributes as well, see the schema file), this means, you
can
have more than one mail address just by adding another one. -> RC today is allready showing all email addresses in this case, but
is
not yet able to write them back (or even to create them in the case
of
import
I allready discussed this topic with Thomas, his proposition was to expand the fieldmap with an array size in the case of multivalue
type...
but this is not yet done :-)
But you still need to assign address type, which would be hard. You can't say let first address be 'home', second 'work', third 'other', etc.
We can simply fix import by extending validate() method and writing all emails into 'mail'.
-- Aleksander 'A.L.E.C' Machniak LAN Management System Developer [http://lms.org.pl] Roundcube Webmail Developer [http://roundcube.net]
PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl _______________________________________________ Roundcube Users mailing list users@lists.roundcube.net http://lists.roundcube.net/mailman/listinfo/users
Good afternoon:
More progress: going from 0.7.2 to svn, the number of successfully imported contacts increased automagically from 42 to 57 (out of 63) using the same underlying VCF file, thanks to Alec's changes.
Now only 6 cards (all with the following structure) are not imported because they miss the N: field:
BEGIN:VCARD VERSION:3.0 FN:first last EMAIL;type=INTERNET;type=OTHER:first.last@example.com END:VCARD
Once I add an N: field and a corresponding value manually to those 6 contacts, the import is successful also for those remaining cards.
I also tried exporting the same SQL-based contacts with the latest SVN version to see if that helps, but the number of imported contacts stays at 57.
Secondly, as mentioned in the other messages, the following card is imported, but only partially:
BEGIN:VCARD VERSION:3.0 N:Last;First;;; FN:First Last EMAIL;type=INTERNET;type=WORK:first.last.work@example.com EMAIL;type=INTERNET:first.last@example.com TEL;type=MOBILE:1234 URL:http://acme.com ADR:;;Park Av 100;NY;;12345; ORG:Org END:VCARD
On the first issue, would it be possible to generate the N: field automatically upon export if it does not exist in SQL already to improve compatibility with LDAP imports?
On the second issue (partial import), I understand that Andreas is addressing part of this with Thomas (for multiple email addresses), but for home and work addresses to work at the same time the solution is to use a schema different from or based on inetOrgPerson, such as evolutionperson.schema, in line with f3isar's suggestion?
Is there a schema that can make LDAP store everything that roundcube offers, and at the same time do so in a fashion compatible with iOS, Android and Outlook?
Thank you in advance, Achim
On 04/18/2012 05:45 PM, Achim wrote:
Now only 6 cards (all with the following structure) are not imported because they miss the N: field:
BEGIN:VCARD VERSION:3.0 FN:first last EMAIL;type=INTERNET;type=OTHER:first.last@example.com END:VCARD
Once I add an N: field and a corresponding value manually to those 6 contacts, the import is successful also for those remaining cards.
It should work. Show your fieldmap and required_fields definition again.
Hello:
On 18.04.2012 20:06, A.L.E.C wrote:
On 04/18/2012 05:45 PM, Achim wrote:
Once I add an N: field and a corresponding value manually to those 6 contacts, the import is successful also for those remaining cards.
It should work. Show your fieldmap and required_fields definition again.
'LDAP_Object_Classes' => array("top", "person", "inetOrgPerson", "organizationalPerson", "evolutionPerson"),
'required_fields' => array('cn', 'sn', 'mail', 'givenName'),
'fieldmap' => array( // Roundcube => LDAP 'prefix' => 'title', 'name' => 'cn', 'surname' => 'sn', 'firstname' => 'gn', 'email' => 'mail', // for importing existing contacts that only have field EMAIL;type=INTERNET (without type=WORK) // 'email' => 'mail', // 'email:home' => 'mail', 'phone:home' => 'homePhone', 'phone:work' => 'telephoneNumber', 'phone:mobile' => 'mobile', 'phone:workfax' => 'facsimileTelephoneNumber', 'photo' => 'jpegPhoto', 'website' => 'labeledURI', 'address:work' => 'street', 'zipcode:work' => 'postalCode', 'locality:work' => 'l', 'region:work' => 'st', 'country:work' => 'c', 'zipcode:home' => 'postalAddress', 'locality:home' => 'otherPostalAddress', 'region:home' => 'st', 'country:home' => 'co', 'organization' => 'o', 'notes' => 'note', 'jobtitle' => 'businessRole', 'manager' => 'managerName', 'assistant' => 'secretary', 'department' => 'departmentNumber', 'anniversary' => 'anniversary', 'birthday' => 'birthDate', 'spouse' => 'spouseName', ),
Thanks for looking into this, Achim
On 04/19/2012 05:13 PM, Achim wrote:
'required_fields' => array('cn', 'sn', 'mail', 'givenName'),
change to
'required_fields' => array('cn', 'sn', 'mail', 'gn'),
Hello Alec:
On 19.04.2012 17:53, A.L.E.C wrote:
On 04/19/2012 05:13 PM, Achim wrote:
'required_fields' => array('cn', 'sn', 'mail', 'givenName'),
change to
'required_fields' => array('cn', 'sn', 'mail', 'gn'),
Correct, "givenName" was a relict from the time before I used the evolutionPerson schema.
Unfortunately, still only 57 out of 63 contacts (the ones *with* a N: field value) get imported, independent of the above setting.
Any other ideas? What about generating a "N:" entry dynamically (similar to the "Noname" value) upon exporting from SQL?
Best regards, Achim
On 19.04.2012 19:30, Achim wrote:
Hello Alec:
On 19.04.2012 17:53, A.L.E.C wrote:
On 04/19/2012 05:13 PM, Achim wrote:
'required_fields' => array('cn', 'sn', 'mail', 'givenName'),
change to
'required_fields' => array('cn', 'sn', 'mail', 'gn'),
Correct, "givenName" was a relict from the time before I used the evolutionPerson schema.
Unfortunately, still only 57 out of 63 contacts (the ones *with* a N: field value) get imported, independent of the above setting.
Works for me.
BEGIN:VCARD VERSION:3.0 FN:first last EMAIL;type=INTERNET;type=OTHER:first.last@example.com END:VCARD
C: Add [dn: mail=first.last@example.com,ou=roundcube,o=roundcube,dc=dupie,dc=be]: Array ( [cn] => first last [sn] => lastnet [givenname] => first [mail] => first.last@example.com [objectClass] => Array ( [0] => top [1] => inetOrgPerson ) )
S: OK
'LDAP_rdn' => 'mail', 'required_fields' => array('cn', 'sn', 'mail', 'givenName'), 'fieldmap' => array( 'name' => 'cn', 'surname' => 'sn', 'firstname' => 'givenName', 'email' => 'mail', ... )
Hello Alec:
On 20.04.2012 14:13, A.L.E.C wrote:
On 19.04.2012 19:30, Achim wrote:
Unfortunately, still only 57 out of 63 contacts (the ones *with* a N: field value) get imported, independent of the above setting.
Works for me.
... because you slightly altered the test case ;-)
BEGIN:VCARD VERSION:3.0 FN:test test2 EMAIL;type=INTERNET;type=OTHER:test@example.com END:VCARD
works fine, just like your example does. This card does not work:
BEGIN:VCARD VERSION:3.0 FN:test EMAIL;type=INTERNET;type=OTHER:test@example.com END:VCARD
Please note that FN consists of only 1 "word" or element. This is also true for the 6 cards that I cannot import: so the problem is not the missing N: value [1], but the fact that FN: only has 1 element.
This could be a hidden assumption or an array handling issue somewhere, or is "FN:" required to have 2+ values?
Thank you in advance, Achim
[1] Which in itself is perhaps a side effect from RC trying to generate N: by assuming a "2-value FN:"?
On 20.04.2012 15:32, Achim wrote:
Works for me.
... because you slightly altered the test case ;-)
No, I just copied your test case.
BEGIN:VCARD VERSION:3.0 FN:test test2 EMAIL;type=INTERNET;type=OTHER:test@example.com END:VCARD
works fine, just like your example does. This card does not work:
BEGIN:VCARD VERSION:3.0 FN:test EMAIL;type=INTERNET;type=OTHER:test@example.com END:VCARD
Please note that FN consists of only 1 "word" or element. This is also true for the 6 cards that I cannot import: so the problem is not the missing N: value [1], but the fact that FN: only has 1 element.
This could be a hidden assumption or an array handling issue somewhere, or is "FN:" required to have 2+ values?
Thank you in advance, Achim
[1] Which in itself is perhaps a side effect from RC trying to generate N: by assuming a "2-value FN:"?
Yes. All of this because you have both surname and firstname in required_fields definition. Here one doesn't exist, that's why the contact isn't imported. The logic is in rcube_ldap::validate() method.
I'm not sure we should do exception for import.
Hello:
On 20.04.2012 15:36, A.L.E.C wrote:
No, I just copied your test case.
Then I redacted it too much, clearly: sorry for the wasted energy.
Yes. All of this because you have both surname and firstname in required_fields definition. Here one doesn't exist, that's why the contact isn't imported. The logic is in rcube_ldap::validate() method.
I'm not sure we should do exception for import.
I tried the following two values for required_fields and am quite surprised at the outcome (using the exact same VCF file with the original 63 contacts):
Successfully imported 42 contacts:
Successfully imported 57 contacts:
Shouldn't a more expansive (i.e. more restrictive) required_fields lead to the same or less contacts being imported, rather than more?
Perhaps there is some hidden logic that I don't understand?
Thanks, Achim
Achim achim+roundcube@qustodium.net schrieb:
Hello:
On 20.04.2012 15:36, A.L.E.C wrote:
No, I just copied your test case.
Then I redacted it too much, clearly: sorry for the wasted energy.
Yes. All of this because you have both surname and firstname in required_fields definition. Here one doesn't exist, that's why the contact isn't imported. The logic is in rcube_ldap::validate() method.
I'm not sure we should do exception for import.
I tried the following two values for required_fields and am quite surprised at the outcome (using the exact same VCF file with the original 63 contacts):
- 'required_fields' => array('mail'),
Successfully imported 42 contacts:
- 'required_fields' => array('cn', 'sn', 'mail', 'givenName'),
Successfully imported 57 contacts:
Shouldn't a more expansive (i.e. more restrictive) required_fields lead
No. InetOrgPerson requires sn and cn... if they are reqired fields, RC is trying to suggest them somehow and it works... but this is not perfect yet :-)
Andreas
to the same or less contacts being imported, rather than more?
Perhaps there is some hidden logic that I don't understand?
Thanks, Achim _______________________________________________ Roundcube Users mailing list users@lists.roundcube.net http://lists.roundcube.net/mailman/listinfo/users
Hello Alec, and sorry for the longer delay:
On 20.04.2012 15:36, A.L.E.C wrote:
On 20.04.2012 15:32, Achim wrote:
[1] Which in itself is perhaps a side effect from RC trying to generate N: by assuming a "2-value FN:"?
Yes. All of this because you have both surname and firstname in required_fields definition. Here one doesn't exist, that's why the contact isn't imported. The logic is in rcube_ldap::validate() method.
I'm not sure we should do exception for import.
As Andreas pointed out in his follow-up message:
On 20/04/2012 17:11, Andreas Dick wrote:
InetOrgPerson requires sn and cn... if they are reqired fields, RC is trying to suggest them somehow and it works... but this is not perfect yet :-)
Is there any chance to cover at least the most probable fringe cases, such as the one mentioned above (only first or only last name, not both)?
I would believe that it is more convenient to be able to import 100 contacts out of which some have autogenerated (i.e. perhaps "wrong") information added, which can be easily detected and corrected by even an untrained user. The alternative (a user having to find out and manually update those contact information that have *not* been imported from a single, not human-friendly VCF file) seems a lot less desirable.
Would adding some "autogenerator code" be possible for rcube_ldap::validate()?
Thanks for your consideration, Achim
I would believe that it is more convenient to be able to import 100 contacts out of which some have autogenerated (i.e. perhaps "wrong") information added, which can be easily detected and corrected by even an untrained user. The alternative (a user having to find out and manually update those contact information that have *not* been imported from a single, not human-friendly VCF file) seems a lot less desirable.
Would adding some "autogenerator code" be possible for rcube_ldap::validate()?
this could also help in the case of "adding" contacts from email addresses...
Andreas