High DB2 activity of IBM Connections Metrics database – reloaded …
once a problem is solved, the next comes up … That`s a rule…
I had another big problem in one of our IBM Connections 4.5 environment with our databases.
After a maintenance window, where we updated the OS System we started all the involved user software and checked the functionality of the system. Everything seemed to work fine but after a while our Nagios monitoring informed us that the DB2 Server writes a lot of transaction logs ( well at least more than usual )
I checked this and found out that the metrics application of IBM Connections writes log files faster than they can be archived. See this screenshot from the archived log path:
Luckily I turned on archive log compression, so I had some time (or more disc space) to implement a workaround, which is to clean up all archival logs in a specified interval. If I would not perform this action, we would end up in a non available metrics application. Like this, the customer can continue working.
Immediately I wrote my IBM focal point and told him that we`ve having the same issue again that we used to have before (already solved) with the homepage database –> High Activity in homepage database
But now it is the metrics database which seems to have a problem with the application.
In the db2 log file, I could find out that the system tries to reorg some tables, even though auto_reorg option is turned off.
With the db2top tool I checked what there is going on inside the database
My IBM contact person opened up another Level 1 PMR himself, and gave all my logs and information to the IBM development team, so that they can find a solution for this.
Seems there is also another customer with exactly the same problem…
During the lifecycle of the PMR, our customer reported, that the metrics application stopped refreshing the metric values…
But finally IBM gave us the clue how to fix this problem with the logs.
The steps for this solution were:
– Ensure the indexrec parameter is set to restart
– Depending on your topology – shutdown the server or cluster that the database metrics is running on.
– Increase the size of the Logprimary and Logsecondary using the full 255 transaction logs ( we did not fully use all those parameters that IBM recommended – we decided to increase the values carefully to not risk running out of free disk space )
– On the metrics database run reorg.sql and updateStats.sql offline (when application server instance is stopped)
– Restart your servers and applications again
– The index should be rebuilt on restart
– Collect the db2diag.log and make sure you catch whether the index becomes invalid or not. I have to mention here, that I had one putty session open for looking at the db2diag file to see what is happening inside the DB2 process of the instance.
After starting the metrics application and cognos server the logging process looked normal again and the test of the metrics applications was successfully. PMR closed 😉