I honestly believe that the addition of resources from the various stakeholders greatly contribute to the aspect Dr. Camillus spoke of, ‘The Problem Is Shaped Like A Blob’. The more hands in the clay mound pushing, pulling, patting, and shaping the more likely someone trying to ‘find’ a root cause(s) becomes impossible. Problems of this magnitude MUST be a carefully managed effort, otherwise, any beneficial effort will quickly be lost due to a rapidly changing problem domain.

Addressing symptom problems will temporarily provide some manner of resolution, but unless the root problem is addressed, symptoms will only appear in other areas of the system. It is not difficult to understand that when it comes to systems that have been architected with dependencies that once failure in one component begins, others are sure to follow. That is the nature of relational integrity constraints.

I did not read the article posted by Dr. Camillus, but I wonder if the problems in which he studied were ones where the managerial infrastructure completely understood the underlying system, its dependencies, and had very open and effective communication channels? In my experiences, most of the hard IT problems that became wicked ones were usually the result of efforts from decentralized business units attempting to resolve ‘their’ symptom not fully realizing what the impacts would be in other areas. Insufficient communication is only a precursor as it generally does not include the proper amount of detail needed for analysts/problem-solvers to determine whether or not the efforts of the segregated stakeholders will impact other areas of the system until it is too late.

Whether or not a problem like this has a solution is up for debate. There has to be some initial cause which in turn triggers a responding effect. In one problem I remember, the firm hired a developer to modify a piece of software to add an additional parameter to a function permitting it to perform ‘additional’ operations. The code modification was a success and the dependent business units altered processes and tasks to leverage the new feature.

Unfortunately, the modification was done to a COTS system. Aside from the fact that it was probably illegal, invalidated service contracts and warranties, and possibly infringed upon patents or copywrites it was done. What wasn’t taken into account was the fact that the COTS had a built-in auto-update mechanism. When the software producer published a new version, the software updated itself.

Naturally, the earlier modification was not included in the new version and entire business units experienced work stopages the next day. They were literally loosing thousands of dollars an hour and rapidly approaching service contract defaults. Needless to say, panic was the dominant sentiment and multiple business units scrambled to find resolutions.

Each unit, fueled by their own self-interest attacked the problem from different angles all thinking the problem was in ‘their’ specific area of concern. Not one thought to check the update logs prior to an all-out assault on the system. Several knee-jerk changes were made, modules were disabled, reconfigured, and network traffic was re-routed.

All of these things were done prior to the establishment of ‘any’ form of damage control, or emergency response team. Naturally, these activities further degraded the ability of the system to support business processes and the problem grew well beyond its original manifestation. The blob was born. Other problem-solvers from different business units were basically rendered useless because there were so many things happening within the system that it was hard to tell whether they had fixed anything or not.

This was the state of the system when I arrived. My first response was to establish the response team consisting of member from each of the affected business units so that ‘all’ stakeholders were working in a concerted effort. My second measure, given the complete breakdown of the system was to restore the most recent full backup and run the latest transaction log file against the database to restore the records.

This quickly brought the system back online and gave us full functional capabilities as the backup was prior to the version update. I asked which business unit first noticed the system was not functioning correctly and began to chase the problem back from there. I had the network administrator pull log files for that night to see who was accessing what and we found the entry for the version update.

At the time, I wasn’t aware of the ‘modification’ but when I had the system administrator install the system on a temp server and uploaded the code to their versioning database, I was able to run a ‘difference’ function and locate the modification. When I discovered it, I notified the appropriate members of the team, CIO, and CEO. I showed them where the problem had happened and why but refused to modify the code in the 3rd party software. I like my ethics right where they are.

My job was to isolate and resolve the ‘wicked’ problem and I did. Mission accomplished and I moved on. I can’t be entirely certain they restored the ‘modification’ in the newer version and I honestly feel it is none of my business. I do however feel that they learned a valuable lesson in properly introducing modifications, monitoring software auto-updates, and utilizing a true test platform.

Just my thoughts…

Jay