As promised before, here are some more notes from the classes that I’ve taken as part of my soon-to-be-completed Master of Software Engineering degree. This time: (software) project management. Most of this information comes from Effective Project Management by Robert K. Wysocki.

FYI, this was one of the few classes where most of my in-class notes weren’t about the course material at all but were reflections about how we do project management where I work. Once I discovered that I was a project manager, I realized that I had best become better at doing it. Funny how obvious that seems in retrospect.

Basics

Project = “A sequence of unique, complex, and connected activities having one goal or purpose and that must be completed by a specific time, within budget, and according to specification.” (Wysocki, 4)

Program = A collection of projects with multiple goals.

Most “interesting” software projects involve some degree of unclear requirements or unknown solution. These projects should ALWAYS use an adaptive/agile or iterative approach.

• Examples: Evolutionary waterfall (for low risk/easy projects), SCRUM, Rational-Unified Process (RUP), Dynamic Systems Development Method.
• These methods separate high-level and detailed planning. Each must be done, but the detailed planning is not done all “up-front.”
• (These are not iterative approaches: Pure Waterfall, Rapid/Parallel Development, Staged Delivery.)

Continuous quality managment and process quality improvements appear as keys to successful projects.

What every project should have . . . to some extent or another

Linear/waterfall, iterative/agile/adaptive, and extreme project management techniques all have the same phases, they just appear in different ways. They are:

• Define the project: Take the problem, proposed solution, and objectives and make a project charter and scope document
• Develop detailed plan(s) — preferably iteratively and just-in-time
• Launch the plan(s)
• Monitor and control project progress: Reporting, change control, problem escalation, revising plans
• Close out the project — Acceptance, installation, party! Seriously, you must party.

Risk Management

The major responsibility of the project manager is to manage risk in the project.

• Identify risks:
• Quality and performance with respect to technology
• Resource allocation
• Planning process
• Organizational support
• Changing legal and regulatory requirements/availability
• Suppliers and contractors
• Assess risks:
• Separate risk, magnitude, and probability
• Exposure = Probability of loss times cost of loss
• Consider using a risk matrix (high-medium-low cost v. high-medium-low probability) to track exposure
• Consider whether solution costs more than the loss
• Assess risks at each project phase/iteration
• Respond to risks:
• Accept — Do nothing
• Avoid — Don’t do that part of the project
• Contingency planning — Reframe the plan to deal with risk
• Mitigate — Reduce the probability and/or the magnitude of loss
• Transfer — Outsource the risky part to someone more capable of handling it
• Monitor and control risks:
• Make a risk log.
• Review risks at status meetings.
• Add triggers to risks so that countermeasures are taken at the appropriate time.

Project estimation

The average worker efficiency in IT is 50-65%. That’s the amount of time actually devoted to project work. That doesn’t include ad hoc interruptions, which takes another 33% of so of the workday. And there’s a lot of variation in duration for the same task, since everyone works at different capacities. So . . . It’s best to think in terms of task size and not the time that it takes to complete a particular time.

Methods for estimating task size:

• Similarity to other activities already done — Usually a very good predictor
• Historical data — Usually very objective and concrete
• Expert advice — Be weary of using just this
• Delphi technique — Iterative planning poker. Result is the average of the third round (or consensus)
• Three point technique — E = (Optimistic + 4*Most Likely + Pessimistic) / 6 for however you define those three terms
• Wide-band Delphi — Delphi technique with three point computation instead of a simple average

You can (and should) determine duration from the effort values and from that cost.

Project task management

Having a work breakdown structure (WBS) does not mean that the project must be managed like a waterfall, with all of the tasks defined to a fine precision before implementation can start (though some tools make this more likely).

Parts of the WBS can (and should) be high-level at the start. The plan gets more detailed with each iteration. Instead, treat a WBS as a represention of the functional/modular breakdown of the system. It’s useful for visually thinking about the project, designing the architecture, planning and estimating, and reporting status.

The network diagram is more useful in actually planning the project than a Gantt Chart, which is good because Gantt Charts suck. Then network diagram contains sequencing information, and you can use it to find the critical path of tasks that define the full project duration.

Random thoughts

This stuff — plus copious amounts of Hindi and Arabic scribbling — filled the spaces between my notes from reading Wysocki.

• If I had a time machine and could redo parts of a project, when would I go to add more value or lower costs?
• Lucky + smug = ?
• Consider keeping a historical journal for estimation: size of project, time, resources, technology.
• Engineers are creative, problem-solving people. Rule-following is not a creative act and implies a solved problem. If software engineers are going to do project management, the project management techniques must not get in the way of actually solving the project’s problems or it just won’t happen.
• I am the very model of a modern major general.
• How well does agile planning and development scale? Can you do critical path analysis with it? Is it even worth trying to do that?
• Measure quality, productivity, maintenance work v. feature work, time to market. Measure when starting, when passing milestones, when encountering defects.
• I like postage stamps.
• Iterative development should have deliverables that can actually be met at the end of each iteration. The iterations should be tied to deliverables. Milestones shouldn’t just be mile markers; that’s what a calendar is for.
• Do risk management at every stage of the project.
• Use a pull system for features with insertion for bugs and technical support assistance. Translate “do interruptions now” to “do next.” Finish up what’s in progress if it’s worth doing.
• Product != Project != Program != Product

read more from this topic.....

Here are a few notes from the yellow notepad I’ve been using for school over the last six months. This first installment touches on C. Later I’ll add some UNIX programming and C++ notes. Basically these are the things that struck me as I was quickly reading the “refresher” material for my spring class: “Advanced C Programming for UNIX.” After more than a decade with C, I feel I know the core language pretty well; and here are the things that struck me as worth noting.

Is there a portable way to refer precisely to datatypes of a particular size? Yes, in C99 #include <stdint.h>.

Use signed integer types larger than char with getc() or getchar() to accomodate EOF (-1).

A good running average is avg += (x - avg) / n; It reduces the chance of overflow, is fast, and doesn’t require you to keep track of all values.

Each comment collapses to a single blank character during parsing.

Many system names start with an underscore ( “_” ). Avoid creating new identifiers that start that way.

All functions have extern storage, just like variables defined outside of a function.

Use volatile when it’s possible that the variable’s value will change from outside the normal flow of control. Technically this prevents compiler and processor instruction reordering optimizations around the use of the variable.

The register storage class is for small, fast-changing variables like loop indices (not for bigger things like arrays). And you can’t do &variable for register-declared storage.

Array names are const pointers, so an array name is not a valid l-value.

It’s not possible to take an address of a literal value, like 3.14 or “I am the very model of a modern major general.”. So . . .

Because C-strings are either arrays (char s[]) or pointers to memory areas (char *s), it’s illegal to reassign a value to a string variable directly (e.g., s = “new string”; /* won’t compile */).

When passing multidimensional arrays to a function, the function’s argument list must give the size of all but the slowest changing dimension.

Function pointers: int (*f)(int, double) is the same as int f(int, double). The parentheses are necessary to ensure the * binds to f and not int. To use the function pointer f either of these is acceptable: (*f)(37, 3.14) or f(37, 3.14). (This is “declaration follows usage.”)

const int *p is a pointer to a constant int.
int const *p is a constant pointer to an int.

“LP64″ scheme means long ints and pointers are 64-bit values.

Some compilers use precompiled headers to speed up compilation. This can make the inclusion order of header files significant. Also, some compilers (like Xcode) use “prefix headers,” which are always included in a project.

Defining NDEBUG turns the assert() macro into a no-op.

read more from this topic.....

It’s Friday here at The Metrowest Homeopathic Imaging and File Rehabilitation Center, the day when I usually spend part of my time on 20% projects. [1] But I’ve basically been working on the results of my last such project full time, so perhaps it’s time to take a moment and look at the world rushing by and catch up on a few things before getting too far behind.

I moved offices for about the tenth time since I started working last week. That’s not a bad thing; we’ve been growing consistently over the last 9 1/2 years, and the company likes to keep work groups physically co-located without stuffing people into cubicles. While packing my office, I decided to purge my files, getting rid of a lot of old, old papers. (The process was actually a lot like Merlin Mann’s War on Clutter.)

It was during this time that I realized how much information I have tied up in paper notes. The same goes for e-mails. (Over those 9 1/2 years I’ve managed to keep more than 13,000 7,000 messages.) That information isn’t doing anyone other than me any good at all . . . coworkers, amateur color enthusiasts, no one! So expect to see some random color-related stuff coming your way.

[1] - A 20% project is a self-directed project that’s “off the books.” There are two main reasons for doing projects like this. It gives people freedom to explore and dive into topics that are risky or somewhat out-of-model. These projects also add slack to a system, preventing workers from being overutilized. Studies show that beyond a certain point the more you use any resource — whether it’s an engineer, a highway, a drillbit, whatever — you get lower productivity.

read more from this topic.....

There are two things you should know about me: I am not a good person, and I hate being bored or having my time wasted. Those two things combined pretty much say everything you might need to know about me, not unlike Dostoevsky’s underground notetaker:

I am a sick man. . . . I am a spiteful man. An unattractive man. I think that my liver hurts. But actually I don’t know a damn thing about my illness. I am not even sure what it is that hurts. . . .

I am quick to judge and wrong as often as not. Sometimes it takes me months or years to realize the folly of my first impressions. An artifact of my Iowan-ness, a genetic marker from my place of origin, it’s my curse for being extroverted and open-minded. Occasionally I publicly call out those I mistake for what they’re not, but more often I nurture these mistakes quietly on the sweet milk of my incorrectness. Knowing that I’m prone to being wrong doesn’t make me less likely to be wrong, but it does keep me quieter . . . in general — my recent excoriation of Martin Amis notwithstanding.

Unfortunately, naming my demons doesn’t give me complete control over them — I’m not one who thinks that analyzing the progenitors of my peevishness gives me total will-to-power over my future — but I have been trying to say “hmm . . . how interesting” more. Fortunately, I’m rather good-natured despite my wickedness and, by and large, get along with almost anyone.

Which brings me ’round to that other thing and the inspiration of this dispatch: Lectures are hard, nigh on unbearable sometimes.

You see, I do the required reading before class and (usually) the optional readings, too. I am a slow reader, mostly because I hear the words in my head and because my inner voices engage the author’s disembodied voice in spirited discussion. I deliberate and I question. And I either suspend disbelief when I don’t know much on the subject or fill in gaps or deconstruct assumptions (mine or the author’s) when I do. I read and I edit and I rewrite books and articles in my head, which is probably why I remember most of what I read and less of what I see or hear.

I suspect few other people read quite so pathologically, but I got spoiled at Grinnell by everyone actually having read the assigned material in advance. I chafe when I sit down in class and see the same material from the readings presented on overheads as Powerpoint bullets: tiny, clipped bons mots struggling to break free in search of true pith and vim. My same inner voices that discussed the work with the authors wait anxiously for a nibble of something new, but all I hear from them is . . . well, let’s take a look at my notes from tonight’s class.

• αβγδεζηθικλμνξοπρστυφχψω
• ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ
• ῷ μοι κακαδαιμων. εγενετω τθφλος. χαλεπως εστι ό βιος.
• वीर-ज़ारा
• ॐ
• Lots of people talk. [We have a class of 25 people graded on participation.]
• Pair programming is a bit Stepford Wives-ish.
• People think XP is just about pair programming.
• [Instructor F]’s 3 + 1 rules — Lame.*
• [Instructor F] wanted a semester-long class on requirements!
• How to get the most out of these classes? Zen-like and let it wash over me? Pick out the good stuff as it floats bye like Wracker Quoyle? “Hmm . . . how intersting” for the rest?
• Other people are real people. I like books more than lectures, but we do learn from others experiences.
• How do people [like me] with partial (but real & sometimes deep) experience [from the workplace] learn [how to do what we already do better]?
• What should be our goals? How do [high-level] Aquinas-like books [based on distinguishing between this and that] like our text by Sommerville fit in?
• Why don’t I trust instructors? Do I unjustly expect infallibility? Why? Is it my Old Testament upbringing? Perhaps it fits with my Iowa-ness. Perhaps I’m just mean. Probably . . .
• I don’t think [Instructor F] really knows the difference between functional and nonfunctional requirements, but I don’t think that’s important. (See above) But we are wasting a lot of time trying to distinguish them . . . poorly.

Welcome to the purgatorial life of a night school philosopher-engineer. At least I’m getting therapy as well as an education.

* — “#1 - You are responsible. #2 - ‘Stuff’ happens. #3 - If #2 happens see #1.” #4 was slightly more helpful, but I won’t bore you with it. [Instructor F] presented these individually via Powerpoint with a long dramatic pause after the first. So, seeing “You are responsible.” on the screen with a pregnant pause led me to ask “Responsible for what?” thinking it might be some sort of ploy to get us to a sort of uber-insight into Software Development Methodologies. “For yourselves!” was the reply. Giggles all around.

What are we, seven years old? Funny story there. The first week of class we had to sing “Happy Birthday” to [Instructor F]’s first-grader at the course break. “Today is my son’s birthday. Since I can’t be there with him, it would mean a lot . . .”

read more from this topic.....

Going through some old e-mail at work, I found this review of Steve McConnell’s book Professional Software Development: Shorter Schedules, Higher Quality Products, More Successful Projects, Enhanced Careers that I wrote for my group a couple years ago. It changed my perspective on my career a bit and helps explain why I’m in a software engineering master’s program.

McConnell’s central point is that software development is not living up to its full potential. Anyone who has ever read any part of Code Complete shouldn’t be surprised to find that McConnell blames improper software construction and project management practices for some of this. Professional Software Development goes further in arguing that the software profession lacks the level of rigor that other professional careers have and that it needs to get it. In the book’s four sections, McConnell first surveys the problem and then focuses on individual, organizational, and industry-wide ways to build a better software engineering profession.

In the first part, “Software Tar Pits,” McConnell examines (with data) why software projects fail. The most frequent cause of software project failure is requirements problems. Beware of “last-minute syndrome”; continual progress or tool-building is necessary at all times, and project leaders should look daily at how much progress was made. Beware of not preparing and doing “code and fix” development. Changing requirements is the most common source of cost/schedule problems because “software isn’t soft.”

For McConnell the reason why software developers get into these “tar pits” is that most code creators (often despite their other skills) lack formal software engineering skills. The use of the word “engineering” is intentional and is intended to reflect both a training methodology and a level of professionalism. Computer science is research-oriented, with a goal of advancing knowledge; software engineering, on the other hand, is application-oriented, with a goal of applying knowledge and problem solving. Software engineering principles deal with the “essential difficulties” of software development; software is complex, changeable, must conform to messy reality, and is difficult to visualize. Fortunately, the “stable core” of software engineering’s professional knowledge is pretty well advanced in dealing with these complexities. Information doesn’t get shared around enough, and we just haven’t been well trained in this core knowledge:

• SW requirements
• SW design
• SW construction
• SW maintenance
• SW configuration management
• SW quality
• SW engineering management
• SW engineering tools and methods
• SW engineering process

According to McConnell, these are the core competencies that software engineers must have (to varying degrees depending on their job).

In part two, “Individual Professionalism,” McConnell discusses what individual software engineers should do to improve their proficiencies. Much of this is unremarkable: start with professional education and develop skills in a practical setting. But to be a full professional requires more controversial steps: certification and licensing. (It’s worth noting earlier than McConnell does that professionalism and high-quality aren’t necessarily synonymous and that most software developers won’t require the stringent requirements of certification or licensing. As the risks to the public and the novelty of the applications increase, McConnell feels the need for professional accountability–along with legal accountability–also increases.)

The characteristics of great designers include

• A large set of standard patterns to apply to problems
• Mastery of tools
• Being drawn to complexity
• Curiosity
• Seeking out criticism
• Experiencing and learning from failed projects
• Not being afraid to use brute force during discovery, and
• A desire to create and apply knowledge.

In the third part “Organizational Professionalism” McConnell uses data from the literature to suggest that team cohesiveness and an organization’s use of best practices are bigger contributors to team success than individual qualities. Having good developers is important, but the biggest schedule/cost improvements come from merging principles with creativity. The best organizations know the rules aren’t one-size-fits-all and have the confidence to break them and trade certain principles in favor of others.

Projects need specialization in the core knowledge areas listed above as much as companies do. The larger the project, the more specialization is needed in the areas of construction, design, planning and tracking, business management, QA, and requirements. Even ad hoc, this leads to better follow through and ownership.

Some interesting statistics from this section of the book include the gem that the return on investment of improving practices has been measured at 500-900%. That doesn’t include harder to quantify benefits such as faster time to market and improved customer confidence. Another one is that the typical effort in months for a project is 2.94 * KSLOC ^ 1.1, where KSLOC is number of lines of code (in thousands) that a project will require. NASA’s Software Engineering Lab, which is one of the best, does much better 1.27 * KSLOC ^ 0.986. They’re actual more efficient with larger projects.

It’s possible to measure the level of maturity of a software organization using the SW-CMM model. Companies fall into one of these levels:

1. Initial: Chaotic, underperforming, unshared knowledge, “heroic” or code-and-fix development. This is the default level of most organizations.
2. Repeatable: Basic program management practices vary project-by-project, shared responsibility, falter with novel problems.
3. Defined: Standardized PM practices across the organization, process group, training, projects are on-time/budget, more advanced than code-and-fix.
4. Managed: Highly predictable results, stable processes, database of project data for effectiveness analysis, organization-wide processes for all projects.
5. Optimizing: Continuous/proactive process improvements, results are measurable and measured, QA focus on defect prevention, root-cause analyses.

Research shows that the SW-CMM method is cost-effective, allows risks, doesn’t lead to extra bureaucracy, but does take a while to implement (on average 3 years per level) and is costly.

In chapter 16 McConnell lays out an interesting, laddered professional development plan for software engineers that organizations can use. The system is the one his company uses, and it measures engineers’ proficiencies in the knowledge areas listed above (putting them into five broad competencies); has plans for training, mentoring, and public recognition of improvement; is tied to salary; and presents a roadmap for becoming a “professional.”

McConnell revisits that idea of a software engineering profession in the final part of the book, “Industry Professionalism.” It seems to be the most controversial section, though it really shouldn’t be. According to McConnell’s thinking, software development is already on the way to becoming an engineering field, despite having a ways to go. When it finally arrives at the professional stage–which comes after commerce is wed with science to produce a high quality, dependable end-product–certification and licensing will be the recognition of the status quo. Customers will expect important software to behave well and will be able to hold one or more licensed individuals who worked on the product civilly or criminally liable. That might sound extreme but the potential of some software to do harm (or good for that matter) is just as great as the “products” of other professionals (doctors, lawyers, civil engineers, etc.).

Certification is voluntary and shows proficiency in a field. Licensing is not mandatory or universal but might be required by some jurisdictions when accountability is required. (Consider that your friend can do your own taxes and you can blame no one if things go wrong, or you can use a CPA and have legal recourse in the case of negligence or ethical violations). McConnell estimates that only 1-5% of all software professionals will need licensing.

Beyond that, the industry as a whole has an enormous responsibility to the public trust to act in a professional, ethical manner. Moreover, the industry should do a better job applying computer science and software engineering knowledge that was developed many years ago and is currently unused.

On the whole, Professional Software Development is a compelling, well-argued read. However you feel about its most controversial parts, if you construct, test, or manage software, it is well worth being familiar with the issues that will likely affect your career in the next decade or so. Beyond that, I found it contained a very compelling argument for seeking out and fixing the inefficiencies in individual projects and throughout the whole organization.

read more from this topic.....