IBM Connections – SDI (TDI) Sync problems

IBM Connections – SDI (TDI) Sync problems

During our work with IBM Connections, we often get customer requests where users in IBM Connections were deactivated or cannot be on boarded successfully.

I want to share my knowledge in this area with a kind of knowledgebase and errors that often occur

1) Users do not get on boarded onto IBM Connections

This is a common problem that can have several reasons (e.g.)

  • another user with the same UID, GUID or MAIL already exists in the database
  • A validation error occurred because one of the attributes contains a bad (or too long) entry

as an example, I created another entry with the same UID and MAIL in my LDAP

1. entry:

TDI3

 

 

 

 

 

 

 

 

2. entry:

TDI2

 

 

 

 

 

 

 

 

When you now start sync_all_dns – you will see the error in the log:

TDI10

 

Hmm this is not really saying why the sync did not work! Let`s activate the DEBUG level for log4j:

open the file …/ibm/TDI/V7.1.1/etc/log4j.properties and change:

log4j.rootCategory=INFO, Default
to
log4j.rootCategory=DEBUG, Default

Then start the sync_all_dns assemblyline again. The log will be a lot bigger but you can now see detailed errors:

TDI5

Here you can see that the check of duplicate email is true! This means the email address already exists in the database.  Yep, this is true.

Another common problem is if the validation rules fail. In this example, my phone number is more the 32 digits of length in the LDAP:

TDI6

The file “validate_dbrepos_from_source.properties” contains the maximum length of the database fields.

TDI8

 

 

 

 

 

 

In many cases, the attributes from LDAP are simply too long. And the account will not be on boarded to IBM Connections! I`ve also seen support cases, where customers tried to on board users with email addresses longer than 256 chars – hard to believe 😉

2) Users get inactivated / deleted

This can also happen quite often. A cause for this is that the source (LDAP) data has changed so that the hash value (the value that SDI uses to identify an entry in LDAP and Database) changed.

A common problem is the marriage of a person, where the UID changes (okay, marriage is for sure not a problem – but SDI often has problems with that).

Let`s assume:

Julius Schwarzweller with UID: jschwarzweller

marriages and gets another surname (well not really me, as I`m already married 😉 )

Julius Mueller with UID: jmueller

Then SDI will inactivate the user, if the hash value in profile-tdi.properties file is set to UID (which is the standard):

TDI7

 

 

 

That`s the reason, why we set the property to “GUID”, which is the unique internal identifier of the LDAP server:

domino --> dominoUNID

SDS --> ibm-entryUUID

AD --> objectGUID

SUN --> nsuniqueid

This guarantees, that if someone marries, the user will not be inactivated! BUT… a lot of your customers still work out ways either to change the unique IDs (e.g. dominoUNID by copy + pasting the document) or by deleting the user in AD and recreating it with the new username. For sure SDI is helpless in such a case. Then you need to become active and make manual changes.

So let`s assume the GUID does not match anymore – the user is not synced anymore and got inactivated.

There are some steps we need to perform in order to reactivate the user correctly…

1) find out the new GUID

You could either enable trace in SDI or I always use a small Javascript function that I`ve included (you can use this for):

AD –> then you modify the function: function_map_from_objectGUID

Novell eDirectory –> then you modify the function: function_map_from_GUID

Domino –> then you modify the function: function_map_from_dominoUNID

SDS –> For SDS you do not need to modify the function, as a 1:1 mapping is used of “ibm-entryUuid”

TDI8

This prints the GUID for all users into the ibmdi.log:

TDI9

 

 

2) We need to find out the “old” GUID from the database:

  • connect to the db server with the people database
    • db2 connect to peopledb
  • issue the command
    • db2 “select PROF_GUID from EMPINST.EMPLOYEE where PROF_UID_LOWER=’jschwarzweller'”
    • or
    • db2 “select PROF_GUID from EMPINST.EMPLOYEE where PROF_SURNAME is like ‘%Schwarzweller%’

    (you should not use PROF_MAIL here, as inactive users do not have an email address anymore)

TDI9

 

 

3) So that we have the new GUID, we can activate the user again:

  • Initialize wsadmin
    • e.g. ./wsadmin -lang jython
  • execfile(“profilesAdmin.py”)
  • ProfilesService.activateUserByUserId(“OLD_GUID”, guid=”NEW_GUID”) or ProfilesService.updateUserByUserId(“8f6a7a40-b7bf-1033-8531-9929ccee4321”, guid=”6AB21F17-BFFB-0186-C125-790C003D9C95″)
  • … the user is now activated again!
  • We now need to run sync_all_dns.sh to check, if the user stays “active”

4) This change generates an entry in the event table of the Profiles database. There is a profile.worker thread that reads this change from the table and send JMS messages to two topic spaces. All applications subscribe to these topics and get the changes. BUT… It might be necessary to synchronize the new GUID to all other applications because the this procedure sometimes does not work!:

  • Initialize wsadmin
    • e.g. ./wsadmin -lang jython
  • execfile(“communtiesAdmin.py”)
  • CommunitiesMemberService.syncAllMembersByExtId({“updateOnEmailLoginMatch: “true” })
  • The extID now gets synced into the Communities database

–> Repeat this for the features “Activities, Blogs, Dogear, Communities, Files, Forums, News, Wikis, Metrics” – description

5) Sometimes, the automatic sync does not work… Then I always swap the extId manually using the commands:

  • Initialize wsadmin
    • e.g. ./wsadmin -lang jython
  • execfile(“communitiesAdmin.py”)
  • CommunitiesMemberService.syncMemberByExtId(“OLD_EXTID”, {“newExtId”: “NEW_EXTID”, “allowExtIdSwap”: “true” })
  • or with our example from above:
    • CommunitiesMemberService.syncMemberByExtId(“8f6a7a40-b7bf-1033-8531-9929ccee4321”, {“newExtId”: “6AB21F17-BFFB-0186-C125-790C003D9C95“, “allowExtIdSwap”: “true” })
  • The extID now gets synced into the Communities database

–> Repeat this for the features “Activities, Blogs, Dogear, Communities, Files, Forums, News, Wikis, Metrics” – description

 

GENERAL INFORMATION: When you copy something from the mentioned commands, please replace the “-signs, as otherwise errors will occur!

6 thoughts on “IBM Connections – SDI (TDI) Sync problems

  1. Thanks for sharing Julius, i have one speedup tip:

    You mustn’t search the guid, when the guid had changed. You can use populate_from_dn instead, this script always uses uid as hash parameter, so when you add the name of the user to collect_dns.in and run populate_from_dn, then the guid is updated for this user, but the UIDs must match in this case.

    Regards
    Christoph

    • Hi Christoph,

      thanks for this hint… i was not aware, that the populate_ … always performs a sync by UID…
      Anyway I`m aware that the populate script might “heal” defect users like this. But sometimes it happens, that the GUID change is not synced to the other features – I had that several times – even though the worker thread did work. Even the syncAllMembersByExtId did not synchronized the changes to the affected users entry in database. That`s why I normally use the SWAP-GUID method, which is for sure far more complicated … But there I`m sure that it works afterwards 😉

  2. Thanks for putting this page up… I recently work on a project that needed to load bunch of user profiles from a SQL source to Connection Profile DB by using TDI AssemblyLine.

    Obviously there are duplicate e-mail addresses in the source causing the rejection of insert. Java exception raised and pointed out the issue from duplicated e-mail found.

    Is there any workaround on this? Creating a new field to hold the e-mail address is out of the call.

    Regards!
    Galant

    • Hi Galant,

      this is basically a problem, if your source does not contain unique data that you want to use as hash value.
      IBM Connections is only able to handle a user with a unique eMail-adress. So it does not make sense to import the user “peter.mueller@gmail.com” three times…
      I would try the following:
      Identify the duplicate records in the source database. Then let the customer choose, which entry should become a Connections profile. Then you can mark those duplicate entries with a flag in the database. Write a javascript function to skips those entries.

      Normally I ask my customer to remove the duplicate records from the database… But you could try something like this as a workaround.

  3. I’m having this problem during the population on IBM Connections 5.5 on RedHat 6.6.
    Any help?

    CLFRN0209E: Validation failed for field guid. Value is .
    2016-05-23 11:02:14,311 ERROR [com.ibm.di.log.FileRollerAppender.325075e9-7888-4a0a-8276-d33f1b957f7f] – CLFRN1183E: Validation failed for entry CN=Betty Heinz,O=demo.
    2016-05-23 11:02:14,312 ERROR [com.ibm.di.log.FileRollerAppender.325075e9-7888-4a0a-8276-d33f1b957f7f] – !com.ibm.di.exceptions.SkipEntryException: CTGDIS393I Throwing this exception to tell the AssemblyLine to skip the current Entry. If used in an EventHandler, this exception tells the EventHandler to skip the remaining actions.!

    • Hi,
      which LDAP are you using? Seems that the canonical String (guid) that is converted from:
      – DOMINO: dominoUNID
      – AD: objectSID / objectGUID
      is incorrect…

      What did you put as function in your map_dbrepos_from_file.properties file:
      e.g.
      guid={function_map_from_dominoUNID}

Leave a Reply

Your email address will not be published. Required fields are marked *