from www.erp4it.com

Joe, Chris, and Sue were talking over the coffee break at a local IT service management professional meeting...

Joe: "Our senior leadership is all charged up about ITIL, but configuration management is killing us. We're pretty well managed in terms of desktops and their configurations, and now they want us to move into the data center. We're really struggling with how detailed to go with servers and applications, and even more troubling, how to collect the initial inventory of whatever we do decide to track."

Chris: "Isn't that what discovery tools are for? Can't they just go out and collect everything and make sense of it?"

Sue: "I wouldn't recommend that. We've had a few discovery tools in and our experience hasn't been that good. They will give you a lot of technical information that doesn't make sense to the senior leadership. Knowing that there's fifty executables and three hundred libraries on a server isn't very helpful; even knowing that they are dependent on a database or executable on another server is only partially helpful."

Chris: "What's missing?"

Sue: "Our leadership thinks in terms of applications. Those fifty executables make up something called  PLV, or Plan/Lead/Value (whatever that means). The discovery tool doesn't know a thing about PLV per  se, and so anything it spits out needs to be mapped to this higher-level, logical concept of  application. This is a tedious manual process and the vendors seem a little clueless here. They are  pretty good about detecting low-level changes on a server, but we're increasingly realizing we're not  at a level of maturity to deal at that level."

Joe: "One of our biggest drivers right now is patch management and other infrastructure-driven changes. The server engineers are always patching things and rebooting, and they don't have a good understanding of what applications are affected. They've been trying to maintain a little Access database of just what applications are on a server."

Sue: "How's that working?"

Joe: "Not very well. The engineers have been pretty good (but not perfect) about entering in the application to server dependency. They also track the project manager they're working with. But then the project ends and they don't know who to contact in a year when that server needs a patch. Sometimes they'll call the enterprise architects, but the architects have a different list of applications and they don't match up - for example, the engineers called one application FirstTime (that was the vendor name) while the architects called it Quadrex (the actual package). A couple of junior people were trying to figure it out later and couldn't connect the dots; the FirstTime project manager was long gone and the server needed to be rebooted. Quadrex went down hard for a couple days with no notification and the application team (and their customers) were very unhappy. Got escalated to the CIO level."

Chris: "Ouch. Wasn't the server patch part of the change process? Why didn't that work?"

Joe: "The reboot was part of a larger, semi-urgent security patch - 50 servers were involved and while the list was reviewed in the change board, the person who could have raised a flag wasn't there that day..."

Sue: "We had a similar issue with not understanding full dependencies on an application server - it had  been purchased as part of a project implementing a new application and we had that tracked, but then  another team started using it, leveraging the first team's release process and relationship with their  production engineer. This second team wound up building something that supported very different business users and processes. The server was patched and while the first team was notified and took appropriate action, they didn't communicate with the second team, and that second application was completely disabled for a week. Not a good thing."

Chris: "I still don't get why discovery tools don't help."

Sue: "They don't tell you who to call. You have to figure out a process for maintaining that. We knew there was a bunch of code on there, and we thought we knew who owned it, but it wasn't enough."

Chris: "We're having issues with Web services and LDAP. If I publish a Web service that allows anonymous access, and people start using it, how do I know what's impacted when the Web service has problems? Even if it requires authentication, my infrastructure team is clueless that there's a new dependency - that's handled between the Web Services team, Security, and the application team. Seems like the industry is headed for a train wreck here that no-one is talking about."

Sue: "I think we've figured out how to manage this. Our new CIO is a fairly technical person and he took a direct interest in these issues. He said, 'Your basic issue is that the engineers will never be able to keep the dependencies up to date. No centralized team can, and that's where I differ with some of the ITIL interpretations out there. I have seen configuration management teams, enterprise architecture teams, and metadata teams all fail at the same thing: a comprehensive mapping of all the dependencies in a data center. Can't be done by one group. Don't try. You have to distribute it to the application teams.'"

Chris: "WHAT? That's ludicrous. The app teams will never keep it up to date.”

Sue: "That's what we thought. But the CIO was quite firm on this and gave explicit, public backing. Now, I've seen various processes sponsored by the senior leadership fail because they ran counter to human motivation and psychology - the world is not solely a command and control place, even in military organizations, and those who think it can all be done by marching orders are usually ineffective leaders. Our CIO knew this and he put in some smart carrots and sticks

- If an application team keeps its dependencies up to date, it can expect notification of infrastructure impact based on those registered dependencies. If the infrastructure team pounds a server and does not notify the registered application owners, that's a reprimand to them. (This fit well into our change process.)

- If a team does NOT keep its dependencies up to date, it has no expectation of ANY service or notification from the engineering organization. In fact, if no dependencies are registered on a server, the infrastructure team has the right to repurpose or decommission it at any time. THAT woke people up!

We did make a couple examples with some dev and test servers, and word got around FAST that this was serious. The thing is, it only takes a junior person on an app team a day or so to document their dependencies, so it's not that burdensome when you push it out to the right level. But centralizing it will never work."

Joe: "What dependencies are you tracking?"

Sue: "We had to be very clear about what the app teams were and were not responsible for. We track:

- Servers (that is, running operating system instances) - the engineering team maintains the master data set here.

- Applications (logical, large-grained groupings of software) - the architects maintain the master data set of these.

- Databases (catalogs of data managed by DBMS instances on big servers) - the DBAs maintain this in their metadata repository.

"These are our current Configuration Items - not very granular, and we're quite happy with them. Discovery tools and their thousands of *.exes and *.dlls are just way overkill for us right now.

"In terms of dependencies, we have the application teams track:

- App to Application Server

- App to Database Server (in this case, the team must identify which database they are dependent on)

- App to App (e.g. if you are using a Web service or exchanging data)"

The CIO calls it 'collaborative modeling' and we've got a publicity campaign going that says 'Everyone's an architect!' (The enterprise architects grumbled a little, but went along with the joke - this work really helps them do more value-add strategic planning)."

Chris: "Is entering dependencies part of the change process?"

Sue: "We want it to be, but there are a couple of issues:

- Not all changes result in new dependencies - how do we identify those that do and ensure that they are registered?

- What if there is an existing, undocumented dependency? Do we require a change ticket just to update the documentation? We pretty firmly said no, if you (as an app owner) have an existing, real dependency, just get it entered without a lot of formality - you're the one who benefits or pays the price based on this data; your fate is in your own hands. That's the mark of a successful process - when people perceive it's in their own direct interest to follow it."