Since my last post touched on dance, at least tangentially, I'll continue along those same lines again. Ballroom and other forms of partner dancing involve leading and following. A great deal about leading in the office can be gleaned from leading on the dance floor.
First a little background on lead and follow in dance. To dance with lead and follow, we must have connection. Our connection comes in various forms. There are visual connections - you see me do something. But more often it comes from a physical connection: hands, arms, shoulders, back, hips and thighs. Connections can’t be one-way. They are equal and opposite. If I pull, you have to pull back against my pull equally. If I push, you have to push back - again equally. If our connection is mushy, the language of our dance will be equally mushy. With a solid connection to transmit my intentions and suggestions to you, we can do anything.
So what happens when I lead? For example, if I want to lead a simple under-arm turn - how does that evolve from a thought in my mind to your action? It’s all in the connection. First, I push you away slightly - and you see “hey - something is happening here” - you push back and there is a nice firm pressure between our hands. Then I move back towards you, shifting the connection to a slight pulling and as you respond and start to pull back, I raise my hand up and to the side. You don’t even need to know what I’m doing. You will turn. Your balance and motion will dictate that. The only thing that is required: you maintain the connection.
So how does this related to management and leadership at work?
Let's start with the connection. If there's no connection, there's no leading and there's no following. It's pretty likely I'll fail miserably trying to get a room full of strangers to do something for me. So a connection must be cultivated over time and exercised. It's on ongoing process to be built.
Next, there's the message to communicate. You'd be amazed what can happen over a good connection. When dancing, if I have an instant of indecision, my partner will feel it. "Go left..NO! GO RIGHT!" She'll start going right and then get totally thrown off. Or you can give one signal which is open to multiple meanings. If I raise my hand and sort of move it to the side but sort of don't, my partner will be confused. "Does he mean for me to turn? Does he mean for me to move towards him?" I recall an instance when I failed to do this at work. I was proposing that we take on a certain task. I presented the pros and cons and in my effort to be balanced and explain things well, I left my audience unsure what I was advocating! You must me unmistakable. If you leave two options but only want one, you can just bet someone will take the wrong one.
But by the same token, in dancing, nobody wants to dance with someone muscles them around the floor. When leading on the dance floor, it's not my job to do my follower's job. I lead; I recommend; I suggest..and it's up to her to follow. If I strong-arm her into what I want, I will throw her off-balance, possibly injuring her or myself in the process. And if it's clear that's how I lead, I'd expect she'd find reasons not to dance with me in the future. Likewise at work, you can't just bark orders and expect people to comply. Perhaps Dwight Eisenhower said it best, "Leadership is the art of getting someone else to do something you want done because he wants to do it."
If you've selected good partners to begin with, good leadership looks effortless and efficient; a team (or a couple) moves as one.
Tuesday, December 29, 2009
Friday, December 4, 2009
Embracing Boredom in Operations
So, when I'm not working, there's a pretty decent chance that I'm dancing. You will note that IT folks/geeks and dancers are somewhat different demographics.
The other night I had a dance lessons with one of my professional instructors - Olga. Now, Olga is an amazing dancer, a fantastic teacher, a fabulous dress designer and a world-class hair doer-uper. But when it comes to technology? Ooof. Not so much her thing. In fact, it was when she was having trouble getting my next lesson set up in the on-line appointment calendar that this conversation began...
"Olga, I'm not sure if you're allergic to technology or it's allergic to you - but this always seems to be harder than it needs to be. I guess watching you deal with technology is a lot like you watching my dancing."
[I will not attempt to phonetically convey Olga's adorable Russian accent]
She noted that she was lucky that she had a job which didn't really require a lot of high-tech skills and then she asked what I actually do, work-wise.
"I oversee operations for the financial and acquisition systems for a huge civilian agency. When FDA or CDC or other parts of HHS buy things or spend money - many many billions of dollars - our systems track that."
"Oh! That sounds exciting," she said with eyes wide.
"It is sometimes. But it shouldn't be. The goal is to make it boring."
Now, sometimes we have a little language barrier between us, so she wasn't sure if she understood me, "You want boring? Why would you want your job to be boring??"
So I did what I do: I gave her an analogy.
"When you fly, do you want the pilot to walk off the plane thinking, 'Wow - that was boring?' or 'Wow - that was the most exciting flight ever!'
At this point Olga informed me that she hates flying no matter what. So I tried again, more on her terms.
"Imagine you've been asked to design a dress that a woman is going to wear in a competition in several months. So you find out what she wants - is it for ballroom or latin? Does she have a color in mind? All that stuff. Then you take measurements and come up with a design. You find the exact fabric you want, you sew it, put stones on, do fittings - all that stuff in whatever order it's meant to happen. And maybe it fits just right - maybe there are problems. Finally, the competition comes - she wears the dress and it looks good. Let's call that boring. Exciting is when you find out you don't have quite enough fabric or she changes her mind about the color just after you've ordered the fabric or you do a fitting and you realize some measurement is now WAY off and you'll need to do a lot of tailoring or she puts the dress on the day of the competition and the seam rips or maybe the worst case, she's dancing in the competition and the dress pretty much falls off. Now THAT is excitement. But not in a good way."
At this point, the the light bulb was lit brightly over her head. But then she paused, "Isn't that almost all just good planning?"
Why yes, Olga, it is. It really really is.
The other night I had a dance lessons with one of my professional instructors - Olga. Now, Olga is an amazing dancer, a fantastic teacher, a fabulous dress designer and a world-class hair doer-uper. But when it comes to technology? Ooof. Not so much her thing. In fact, it was when she was having trouble getting my next lesson set up in the on-line appointment calendar that this conversation began...
"Olga, I'm not sure if you're allergic to technology or it's allergic to you - but this always seems to be harder than it needs to be. I guess watching you deal with technology is a lot like you watching my dancing."
[I will not attempt to phonetically convey Olga's adorable Russian accent]
She noted that she was lucky that she had a job which didn't really require a lot of high-tech skills and then she asked what I actually do, work-wise.
"I oversee operations for the financial and acquisition systems for a huge civilian agency. When FDA or CDC or other parts of HHS buy things or spend money - many many billions of dollars - our systems track that."
"Oh! That sounds exciting," she said with eyes wide.
"It is sometimes. But it shouldn't be. The goal is to make it boring."
Now, sometimes we have a little language barrier between us, so she wasn't sure if she understood me, "You want boring? Why would you want your job to be boring??"
So I did what I do: I gave her an analogy.
"When you fly, do you want the pilot to walk off the plane thinking, 'Wow - that was boring?' or 'Wow - that was the most exciting flight ever!'
At this point Olga informed me that she hates flying no matter what. So I tried again, more on her terms.
"Imagine you've been asked to design a dress that a woman is going to wear in a competition in several months. So you find out what she wants - is it for ballroom or latin? Does she have a color in mind? All that stuff. Then you take measurements and come up with a design. You find the exact fabric you want, you sew it, put stones on, do fittings - all that stuff in whatever order it's meant to happen. And maybe it fits just right - maybe there are problems. Finally, the competition comes - she wears the dress and it looks good. Let's call that boring. Exciting is when you find out you don't have quite enough fabric or she changes her mind about the color just after you've ordered the fabric or you do a fitting and you realize some measurement is now WAY off and you'll need to do a lot of tailoring or she puts the dress on the day of the competition and the seam rips or maybe the worst case, she's dancing in the competition and the dress pretty much falls off. Now THAT is excitement. But not in a good way."
At this point, the the light bulb was lit brightly over her head. But then she paused, "Isn't that almost all just good planning?"
Why yes, Olga, it is. It really really is.
Friday, November 27, 2009
Practice makes perfect (but perfect practice makes perfect perfect!)
I'm back with another in the series of IT missteps. Your feedback and stories are welcome.
If you asked me where my most painful operational missteps came from, I'd have to tell you they were the times when we did something in production without properly rehearsing it elsewhere. So today's rule is this:
Don't do it for real until you've done it somewhere safe first.
Yes, yes, I know there are exceptions. I'll discuss that, too. First let me enter the IT Confessional and I can describe a few of these painful experiences.
Examples and lessons learned
Now, I've known this as a generality for a long time, but there are always nuances. A few years ago we were doing an upgrade to an application, and at the same time we were upgrading the underlying database. On the positive side, we'd done several dry run upgrades - we knew what we needed to do, and we'd tested the functionality as completely as we could. So we'd practiced - and we'd perfected the thing we practiced. Yay team! Here's the problem: we weren't practicing it the same way we were going to do it "for real."
Basically, we were installing a new copy of the database and then moving the data over in test, and we were upgrading a database in production. In the end, our functionality was failing in production for a small (but critical) corner of the application because of a parameter that determined how database audit log files should be written. Now, I'd like to argue that there no way that parameter should matter - it was a fundamental bug in our database. And that's wonderful, except that after we backed out all the changes and tried to figure out what happened, that's exactly that the evidence showed - clearly and repeatably.
Another painful example we ran into was during another deployment. Part of the work involved was running a program which performed some update for each user's profile. Again, we'd tested this several times. This time the problem was simple and foreseeable. In our tests, we didn't need to lock out users during the execution of this program - they weren't going to log in to our test environment. But in production we did lock them out because we wanted to be sure they weren't in during the time the changes were being made. The catch? Well, the program which updated the profiles ignored all the locked-out users. Ooops. Again, we'd done a great job of perfecting what we practiced, but not a good job of practicing perfection.
Exceptions: how do they change the game?
Let's talk about the exceptions. Sometimes you simply can't have your rehearsal be exactly the same. The laboratory of real life is difficult to duplicate. But there are different facets to rehearsals and we can look at those.
Processes sometimes have to differ between a rehearsal and real execution. The key is to understand where they're different and understand the risks. For example, we do periodic tests to ensure we can bring up our disaster recovery alternate site. We don't actually fail production over, so we have to do a few steps differently. We understand the risks - perhaps that we're not testing a required DNS change - but we've weighed that again the risks of test-induced failure and decided it was the right approach.
Technology is usually going to differ based on some sort of economic/funding issue. Some systems are large and expensive enough that you can't fully duplicate them in a practical way. Perhaps your production and test share the same SAN behind the scenes. Maybe you've got faster, more expensive hardware for production, but older/slower/cheaper hardware in test. So when you make the decision to have a 2-node cluster in test and an 8-node cluster in production, just remember that some of that savings is offset with the cost of analysis to understand the differences, and the occasional "ooops" that happens when they behave differently.
Schedule is tricky. For example, some tasks may require going 24x7 to complete, but while rehearsing them that isn't practical. Also, there are some things which just can't be duplicated. My favorite example was during a large deployment we did earlier this year. We purposefully chose a weekend in January because the following Monday was a holiday and the Tuesday was Inauguration Day in the US, and since we're based near DC, about half our customers had the day off. That seemed like excellent timing. We could have the time to do the work properly while minimizing the impact to our customers. Which was all dandy until we went to have some of our remote customers perform validations. "The system is too slow to use - we can't validate." Just as panic was setting in we realized, it was a few minutes before noon and their office network was flooded with people streaming video the ceremonies. Schedule is probably the most vulnerable facet in terms of unexpected external forces.
Recommendations and summary
Decide what "right" is and then rehearse it as you're planning to actually do it. If you practice something other than what you plan to do, you'll get really good at doing something other than you want.
Set the rules, and then understand the exceptions. Of course there will be exceptions. Have a process for describing, analyzing and accepting (or rejecting them). We don't want to live in the wild west, but by the same token, a strict, purist approach is usually not practical. Do what you do with forethought and intent.
In fact, this leads me to the set of rules I have posted in my office...which I'll cover in an upcoming post.
I'd love to open the door to the IT Confessional to hear what stories you have. Please feel free to add comments with your best (and worst) related stories.
If you asked me where my most painful operational missteps came from, I'd have to tell you they were the times when we did something in production without properly rehearsing it elsewhere. So today's rule is this:
Don't do it for real until you've done it somewhere safe first.
Yes, yes, I know there are exceptions. I'll discuss that, too. First let me enter the IT Confessional and I can describe a few of these painful experiences.
Examples and lessons learned
Now, I've known this as a generality for a long time, but there are always nuances. A few years ago we were doing an upgrade to an application, and at the same time we were upgrading the underlying database. On the positive side, we'd done several dry run upgrades - we knew what we needed to do, and we'd tested the functionality as completely as we could. So we'd practiced - and we'd perfected the thing we practiced. Yay team! Here's the problem: we weren't practicing it the same way we were going to do it "for real."
Basically, we were installing a new copy of the database and then moving the data over in test, and we were upgrading a database in production. In the end, our functionality was failing in production for a small (but critical) corner of the application because of a parameter that determined how database audit log files should be written. Now, I'd like to argue that there no way that parameter should matter - it was a fundamental bug in our database. And that's wonderful, except that after we backed out all the changes and tried to figure out what happened, that's exactly that the evidence showed - clearly and repeatably.
Another painful example we ran into was during another deployment. Part of the work involved was running a program which performed some update for each user's profile. Again, we'd tested this several times. This time the problem was simple and foreseeable. In our tests, we didn't need to lock out users during the execution of this program - they weren't going to log in to our test environment. But in production we did lock them out because we wanted to be sure they weren't in during the time the changes were being made. The catch? Well, the program which updated the profiles ignored all the locked-out users. Ooops. Again, we'd done a great job of perfecting what we practiced, but not a good job of practicing perfection.
Exceptions: how do they change the game?
Let's talk about the exceptions. Sometimes you simply can't have your rehearsal be exactly the same. The laboratory of real life is difficult to duplicate. But there are different facets to rehearsals and we can look at those.
- People - who is doing the work
- Processes - how is the work being done
- Technology - what hardware and software is being used
- Schedule - when does the work happen
Processes sometimes have to differ between a rehearsal and real execution. The key is to understand where they're different and understand the risks. For example, we do periodic tests to ensure we can bring up our disaster recovery alternate site. We don't actually fail production over, so we have to do a few steps differently. We understand the risks - perhaps that we're not testing a required DNS change - but we've weighed that again the risks of test-induced failure and decided it was the right approach.
Technology is usually going to differ based on some sort of economic/funding issue. Some systems are large and expensive enough that you can't fully duplicate them in a practical way. Perhaps your production and test share the same SAN behind the scenes. Maybe you've got faster, more expensive hardware for production, but older/slower/cheaper hardware in test. So when you make the decision to have a 2-node cluster in test and an 8-node cluster in production, just remember that some of that savings is offset with the cost of analysis to understand the differences, and the occasional "ooops" that happens when they behave differently.
Schedule is tricky. For example, some tasks may require going 24x7 to complete, but while rehearsing them that isn't practical. Also, there are some things which just can't be duplicated. My favorite example was during a large deployment we did earlier this year. We purposefully chose a weekend in January because the following Monday was a holiday and the Tuesday was Inauguration Day in the US, and since we're based near DC, about half our customers had the day off. That seemed like excellent timing. We could have the time to do the work properly while minimizing the impact to our customers. Which was all dandy until we went to have some of our remote customers perform validations. "The system is too slow to use - we can't validate." Just as panic was setting in we realized, it was a few minutes before noon and their office network was flooded with people streaming video the ceremonies. Schedule is probably the most vulnerable facet in terms of unexpected external forces.
Recommendations and summary
Decide what "right" is and then rehearse it as you're planning to actually do it. If you practice something other than what you plan to do, you'll get really good at doing something other than you want.
Set the rules, and then understand the exceptions. Of course there will be exceptions. Have a process for describing, analyzing and accepting (or rejecting them). We don't want to live in the wild west, but by the same token, a strict, purist approach is usually not practical. Do what you do with forethought and intent.
In fact, this leads me to the set of rules I have posted in my office...which I'll cover in an upcoming post.
I'd love to open the door to the IT Confessional to hear what stories you have. Please feel free to add comments with your best (and worst) related stories.
Saturday, November 21, 2009
There Are No Temporary Solutions
In my first series of posts, I'll look at common missteps in the IT world. These are things which I've seen over and over again. For each, I'll give some examples and share some ways to see them coming - oh, and I'll throw out some self-defense techniques for avoiding them, too.
So - the first item on the list of IT Missteps is the temporary solution.
Here's the rule - say it with me: There are no temporary solutions.
What is a temporary solution?
Call it a one-off, a work-around, or a temporary solution - these are the things which were meant to be in place as a stop-gap, but end up becoming calcified into the environment. Sure, sometimes they really are temporary, but you should look at them with skepticism and with an eye towards the long-run.
Did you ever get into a situation like this:
You are trying to get data from one system to another. You've heard that someone is working to build out a new architecture for data transfer but that's just a start-up project. But hey, you need to get this data moving now. So what happens? You end up with some horked up batch file transfer to feed the data across. Later, you find out that the data transfer you need is outside the scope of that architecture project. Welcome to the not-so-temporary Temporary Solution.
Or maybe you run into a bug in some internally-supported software. You can't run the report...but it works fine if you exit and restart the application. So everyone who needs to run the report does just that.
Why does it happen and why is it a problem?
These sorts of solutions are often a matter of risk transfer or effort transfer. It's easier for me to have you do extra work. Other times it's a matter of not having a choice at all - you have to do something now, but in doing that, the future isn't considered.
Temporary solutions become permanent for a few reasons. Sometimes they're just good enough to live with. It's like having a minor chronic injury - you can live with it, you can function - just not as well as you should, and it's hard to know when you should stop living with it and seek treatment. And sometimes there is hand-waving about future fixes, but that cavalry doesn't make it over the hill.
What makes these one-offs a problem? Well, if it really were temporary it wouldn't be so bad. But in the long run, one-off solutions are expensive and difficult to support. There's typically a lack of thought about the workaround since it's only temporary anyway.
These one-offs are like stray animals looking for a (support) home. An ex-boss of mine once put it this way: "shoot it, or adopt it." Okay - maybe that's a harsh way to say it - but the point remains the same - you either need to make it part of your business or you need to end it. The in-between state where you feed the stray and it stays constantly on the edge of starving doesn't help anyone.
What's a good self-defense against the temporary solution?
First, realize that having a workaround shouldn't let anyone off the hook for a root cause. In ITIL, this can be a matter of the graduation from incident management to problem management.
Second, being conscious of when something is about to become the way you do business. Do the processes and support mechanisms exist for it? Do the resources exist? What costs does this impose?
More generally, there needs to be a connection between the temporary solution and the long-term solution. We can set up the batch file method along with adding this to the scope of the architecture project. I'll restart the applications in parallel to you looking for the root cause of the report issue.
This is a problem which demands a holistic approach to get the optimal results. You need to have a good problem management process in place. You need to have an analysis which informs a gating decision to adding workarounds and pieces of solution to your business. And finally, you need the governance to ensure you don't get trapped in the next misstep: this is too important to wait. We'll take a look at that in an upcoming post.
So - the first item on the list of IT Missteps is the temporary solution.
Here's the rule - say it with me: There are no temporary solutions.
What is a temporary solution?
Call it a one-off, a work-around, or a temporary solution - these are the things which were meant to be in place as a stop-gap, but end up becoming calcified into the environment. Sure, sometimes they really are temporary, but you should look at them with skepticism and with an eye towards the long-run.
Did you ever get into a situation like this:
You are trying to get data from one system to another. You've heard that someone is working to build out a new architecture for data transfer but that's just a start-up project. But hey, you need to get this data moving now. So what happens? You end up with some horked up batch file transfer to feed the data across. Later, you find out that the data transfer you need is outside the scope of that architecture project. Welcome to the not-so-temporary Temporary Solution.
Or maybe you run into a bug in some internally-supported software. You can't run the report...but it works fine if you exit and restart the application. So everyone who needs to run the report does just that.
Why does it happen and why is it a problem?
These sorts of solutions are often a matter of risk transfer or effort transfer. It's easier for me to have you do extra work. Other times it's a matter of not having a choice at all - you have to do something now, but in doing that, the future isn't considered.
Temporary solutions become permanent for a few reasons. Sometimes they're just good enough to live with. It's like having a minor chronic injury - you can live with it, you can function - just not as well as you should, and it's hard to know when you should stop living with it and seek treatment. And sometimes there is hand-waving about future fixes, but that cavalry doesn't make it over the hill.
What makes these one-offs a problem? Well, if it really were temporary it wouldn't be so bad. But in the long run, one-off solutions are expensive and difficult to support. There's typically a lack of thought about the workaround since it's only temporary anyway.
These one-offs are like stray animals looking for a (support) home. An ex-boss of mine once put it this way: "shoot it, or adopt it." Okay - maybe that's a harsh way to say it - but the point remains the same - you either need to make it part of your business or you need to end it. The in-between state where you feed the stray and it stays constantly on the edge of starving doesn't help anyone.
What's a good self-defense against the temporary solution?
First, realize that having a workaround shouldn't let anyone off the hook for a root cause. In ITIL, this can be a matter of the graduation from incident management to problem management.
Second, being conscious of when something is about to become the way you do business. Do the processes and support mechanisms exist for it? Do the resources exist? What costs does this impose?
More generally, there needs to be a connection between the temporary solution and the long-term solution. We can set up the batch file method along with adding this to the scope of the architecture project. I'll restart the applications in parallel to you looking for the root cause of the report issue.
This is a problem which demands a holistic approach to get the optimal results. You need to have a good problem management process in place. You need to have an analysis which informs a gating decision to adding workarounds and pieces of solution to your business. And finally, you need the governance to ensure you don't get trapped in the next misstep: this is too important to wait. We'll take a look at that in an upcoming post.
Subscribe to:
Comments (Atom)
