Hi Hardeep,
> We were able to bring billing process to normal by removing IDocs from long running BG jobs.
Ok great - so that performance tuning "task" was not suffered by compulsive tuning disorder. You got a performance problem - defined a valid goal and stopped after reaching it. That's how performance tuning should be.
> Now, we are trying to bring regular performance tuning into place which was never done in the past 10 years on regular basis. Which is why we started having performance issues since oct 2012.
Unfortunately this step is now suffered by compulsive tuning disorder. The end users are not complaining about old statistics or anything else - they maybe complain about a bad performance experience of their transactions / business processes. So do it the same way like with your billing process.
1. Select the user actions for which the business needs improved performance (and define tuning goals).
2. Collect properly scoped diagnostic data that will allow you to identify the causes of response time consumption for each selected user action while it is performing sub-optimally.
3. Execute the candidate optimization activity that will have the greatest net payoff to the business. If even the best net-payoff activity produces insufficient net pay- off, then suspend your performance improvement activities until something changes.
4. Go to step 1.
That' is. If your defined performance goals for the selected business processes are reached - stop tuning / looking at general stuff. What you have mentioned by "regular performance tuning" should be called "regular performance tracking". Track your defined transaction / business process performance (e.g. STAD, ST03n, etc.) and do some statistical forecasts, etc. based on your defined targets.
> I am sure that you will agree with me that having table stats which are older than year, knowing that CBO uses table stats to find efficient explain plan, would have lead us to performance issues?
No, i do not agree. Once again - table / index and column statistics don't need to be actual - the need to be representative (e.g. a 5% change of 500.000.000 rows does usually not effect the statistical/mathematical calculations so much). The cost based optimizer features are very limited in a SAP environment and so you usually do not benefit from collected (column) statistics and calculations based on low_value, high_value, out-of-value ranges, histograms, etc.. The optimizer needs to do guesses in many cases - check out my blog about some of these effects: [Oracle] DB Optimizer Part VI - Effects of disabled bind variable peeking, adaptive cursor sharing and cardinality feedback on the CBO in SAP environments
Regards
Stefan