Posts Tagged ‘risk’

PayPal Outage Points To CIO Failure

Wednesday, September 2nd, 2009
Paypal's CIO Hasn't Been Doing His Job Correctly

Paypal's CIO Hasn't Been Doing His Job Correctly

The basic job of a CIO is to ensure that a company’s IT infrastructure operates smoothly and allows the company to conduct business. On Monday, August 3, 2009, PayPal’s CIO failed at this most basic of jobs.

A quick check of PayPal’s senior management structure reveals that they don’t have a CIO position (which in of itself is rather amazing), but Ryan D. Downs is their Senior Vice President, Worldwide Operations and so he’s their de facto CIO. What went wrong Ryan?

The Facts Behind The Failure

On Monday, August 3rd, Paypal experienced a world-wide outage that affected all of their customer facing systems. The effect of this outage is that millions of Paypal’s customers who rely on them to approve and complete financial transactions were unable to do so. This was a long outage – it started at 1:30 pm EST and lasted to until at least 6:30 pm EST.

Paypal is attributing this outage to “internal” issues.

Paypal is a huge business. In the most recent quarter, Paypal handled $16.7B in customer online commerce transactions. In the past the company has stated that they normally handle $2,000 in online transactions every second. Just in case you are doing the math, this means that this outage prevented at least $36M worth of business from happening.

What The CIO Did Wrong

I have no magic insights into what went wrong at Paypal, but it’s pretty easy to make a guess. Back in 2005, customers got shut out of Paypal for about 5 days when a software update went very, very wrong. I’m willing to bet that some sort of update process got away from them once again. This is just sloppy IT work.

This is exactly the type of basic “blocking & tackling” that CIOs have to get taken care of as part of building a solid IT foundation. Clearly this has not been done at Paypal.

The reason that this is such a scandal is that its happened at Paypal before. Once a problem is known, the CIO needs to step in and make sure that it will never happen again. We’re not just talking about establishing a fail-safe update process, but also making whatever changes are needed to the Paypal infrastructure in order to make sure that problems like this can’t ripple throughout the system.

Additionally, creating a process for rolling back changes is critical. If a bad change slips though the system and starts to go into production, you need to have the ability to get the system back to the way that it used to be.

Final Thoughts

Major outages like this reflect badly on all CIOs. There should be no reason that a outage like this should be allowed to happen especially since Paypal has had problems like this in the past. Paypal can’t claim that they didn’t have enough funding to prevent this problem – they are the fastest growing part of the eBay corporation.

In the end it all comes down to planning. Finding the time to gather the right people to run through “what if” scenarios and then following through with the recommendations that come out of these meetings is what every CIO needs to do. If Ryan takes the time to do this, then he will have found a way to apply IT to enable the rest of the company to grow quicker, move faster, and do more.

Click here to get automatic updates when The Accidental Successful CIO Blog is updated.

What We’ll Be Talking About Next Time

Hewlett-Packard is a huge IT products and services company that lives and dies by the actions of its sales teams. Making sure that the sales teams get paid should be a simple task right? Think again…

Halt – Who Goes There? CIOs Need Good Identity Management

Monday, August 24th, 2009
CIOs Need To Solve Their ID Management Crisis<p>(c) 2007</p>

CIOs Need To Solve Their ID Management Crisis(c) 2007

As though keeping all of those servers up, applications running, and end users happy seems like enough to make being CIO a full-time job, now CIOs also have to take on the role of data cop? The answer to this question is “yes”, in all honesty, they really should already be doing it. Most company’s most valuable asset, after their employees, is their corporate data. CIOs need to find a way to make sure that they know who is accessing it and why.

Just What Is Identity Management?

Identity management is how an organization controls access to its information based on an individual’s rights and responsibilities. It turns out that most IT shops have been doing a pretty poor job of this.

All too often most of us rely on our old friends Mr. Username and Mr. Password. How many dictionary based cracking events do we need to see in the movies in order to convince us that this is a very poor way to secure our data?

The right way to start to authenticate identities better is to use a second-factor authentication system such as biometrics, tokens, etc. Additionally, using single sign-on technologies can help you bring disparate systems together and save the end users from having to carry around lists of usernames/passwords.

What’s The Best Way To Do Identity Management?

The first step to creating a workable identity management solution is to establish some policies. These policies need to lay out just who is allowed to access what information. Clearly, if you’re not allowed to use some piece of information as a part of your job, then you shouldn’t have access to it.

One of the biggest pitfalls that is found in IT departments today is the existence of multiple different “silos” of data that end up creating a confusing and mixed up environment for access control. Once again, implementing a single-signon solution can solve this problem.

Final Thoughts

Taking the time to design and implement a good identity management solution is very much like buying insurance for your IT department. You hope that you don’t really need it, but you know that you probably do and it’s the grown-up thing to do.

Taking the time to solve your identity management issues once and for all will allow a CIOs to have found a way to apply IT to enable the rest of the company to grow quicker, move faster, and do more.

Click here to get automatic updates when The Accidental Successful CIO Blog is updated.

What We’ll Be Talking About Next Time

More firms are committing to implementing those really BIG process digitization projects. More often than not the CIO will find himself / herself in charge of not only the implementation of the new software application, but also the overall success of the project. How do you go about doing that?

Faux Market Secrets: How CIOs Capture Innovation

Monday, July 13th, 2009
CIOs Can Use Faux Markets To Identify Innovative Ideas

CIOs Can Use Faux Markets To Identify Innovative Ideas

So picture this: you’re a CIO and you desperately want to be seen by the rest of the C-level executives as something more than a simple cost center. What to do? If only there was some way that you could tap into all of that incredible creative energy that we all know lives in the IT department.

If you could harness that energy and apply it to innovative projects, you’d be a company hero. Guess what? The power of Faux Markets is exactly what you need to do this…

What Is A Faux Market?

You know that things are getting fancy when we start using French words! A Faux Market is simply a term that refers to using simulated market forces to make a decision. Perhaps an example would show what I mean. A good case study would be GE Research.

Back in 2005 GE Research had a problem. They had too many product ideas that had been submitted and only $50,000 to spend on investigating them. Clearly they need to make some hard decisions as to which ones they would persure.

The way they picked which projects to work on was by using a faux market. They had their 85 employees spend three weeks buying and selling any one of 62 proposed projects. At the end of the three weeks, GE ended up with a  prioritized list of the top projects that its employees thought had the most value. The project that won was an machine intelligence algorithm that a researcher had proposed but which had not yet traveled through the normal management bureaucracy.

Why Use Faux Markets?

All too often IT departments have a bewildering array of possible projects, technologies, or directions that the department can choose. Sometimes senior management will huddle and make a decision, sometimes no decision gets made. Faux markets offer an alternative.

A faux market tool allows a firm to quickly sort though large numbers of projects or proposals in order to attempt find those that will provide the most bang for the buck. Firms believe that this approach offers them the best chance of finding the next blockbuster product or solution.

Not A Silver Bullet

Faux markets can be a big help; however, as with everything else they do have their drawbacks. One such drawback is the that the voting process does not provide much insight - there may be no penalty for backing a bad idea. Just because a proposal is popular does not necessarily guarantee commercial success.

Final Thoughts

Using faux market tools to quickly sort though a large stack of ideas can provide IT departments with a way to identify innovative ideas no matter where they come from. However, a group vote alone isn’t enough in most cases.

A two step process where voting is initially used to narrow a large list down into a more manageable list of less than 100 candidates is a good first step. The next step can be to use a prediction market allow employees to buy and sell the candidates in order to see which ones go up in value. This will reveal the true winning ideas and you will have found a way to apply IT to enable the rest of the company to grow quicker, move faster, and do more.

Questions For You

How do you process new idea suggestions today? Do you have employees vote on things in order to sort them out? Are these just popularity contests or do they take market factors into consideration? Do you think that faux markets could help you capture more innovative ideas? Leave me a comment and let me know what you are thinking.

Click here to get automatic updates when The Accidental Successful CIO Blog is updated.

What We’ll Be Talking About Next Time

As a CIO, you’ve got some challenges facing you. You’re managing a diverse and potentially distributed work force of highly skilled and talented IT professionals. You need to find a way to keep them challenged, and yet at the same time enable them to find ways to work together. Have you considered Alternate Reality Games?

Risk Management In IT: How Do You Do It Correctly?

Monday, November 10th, 2008
IT Departments Need To Do A Better Job Of Risk Managment

IT Departments Need To Do A Better Job Of Risk Management

The financial melt-down of 2008 had at its core one simple mistake that a whole bunch of companies made at the same time: they did a lousy job of risk management. They made investments in things that were very risky without realizing just how risky they really were. IT departments face the same challenges: at the start of each year we have a number of different projects that we could possibly work on; however, we rarely if ever do a good job of evaluating the risk associated with each of these projects. Instead we focus on things like ROI, business alignment, and which Sr. VP is sponsoring the project to make our decisions. If we don’t want to get caught in our own special version of an IT meltdown, then we had better see if we can figure out a way to measure the risk of an IT project…

So what is risk when you are talking about an IT project? In the simplest terms risk is the chance that an IT project will fail to produce the results that you are expecting because of a given event or set of events. The purpose of risk management is to make sure that you fully understand the risks associated with a project before you start it as well as managing those risks while you are working on the project.

In the world of IT projects, risk is more often then not associated with the company data that we are in charge of collecting, maintaining, and processing. IT teams need to retrain themselves to focus on the value of the data that an IT project is going to be processing and then determine the likelihood that the project won’t be able to do the processing, or in the worst case will corrupt or lose some / all of that data.

What’s really interesting is that outside of IT, the rest of the business has always used risk analysis to determine when they should roll out new products, determine how to spend marketing budgets, and pick which capital investments they want to make. Implementing a good risk management practice within the IT department is yet another way that CIOs can better align their departments with the rest of the business.

Risk management needs to be baked into all of the steps in your IT department’s projects. This runs from project planning all the way to post-production. Everyone knows that fixing a risk earlier in the process is much cheaper than trying to fix it later on down the line.

How much is all of this going to cost? Actually, a fair amount if you end up doing it correctly. You’re going to have to spend money to determine the value of proposed projects, product lines, and any proposed services. Next you’ll have to assign risks to each of these. This can be quite time consuming; however, the process will pay off over time. The key is to have a strategy for how you want to go about doing this. Focusing on where you want the IT department to be in 5 years is a key part of the process because you want whatever project you select to help you to get there.

How can you tell if all of this effort is worth it? There are actually three ways to go about doing this. Most firms use internal audits in order to determine if their IT risk management activities are are paying off. Depending on the industry that you work in, another way is to use regulatory compliance as your measure. Finally, external audits are an expensive but more complete way to measure your effectiveness.

In most IT departments that have an effective risk management function, the funding for the activity comes out of the IT budget. In most companies the belief is that a well executed risk management program will end up saving them money.

In the end, a risk management program will help your IT department to choose the right projects to work on. Once those projects are selected, then it will help you to develop risk mitigation policies, and fix risk vulnerabilities that may end up yielding process efficiencies. It goes without saying that all of this can end up helping a company meet its regulatory compliance needs.

Does your IT department have a way of evaluating the risk of proposed projects? Does your risk management process exist throughout your project process from start to finish? Have you been able to see any savings since you implemented your risk management program? Leave a comment and let me know what you are thinking.