Florent Crivello asked question that really triggered the quality assurance officer in me:
It's easy to just hand-wave and say "Have really high standards!" But how does one transmit their high standards across an organization?
Naval nuclear engineering is an organization with a maniacal approach to quality. Extremely tight feedback loops are the main drivers towards quality. Tight feedback loops prevent low standards from spreading and crush errors in infancy.
We created tight feedback loops through the following practices:
- Audit and inspect
- Hold managers accountable
To be clear, this is not how I would run a product org! Submarines are run in a relentless, sometimes ruthless, manner. There's a reason only 8% of my cohort are still in doing submarine things. But here's how I imagine Naval Reactors would run a product org.
To create tight feedback loops, communicate a lot and communicate precisely.
At sea, the Captain (CO) received operational updates every 6 hours. After a shift of overseeing nuclear plant operations or navigating the boat, myself and another manager would eat our meals and then report to the CO. Our brief started as a mundane report reaffirming the very basics of the ship's conditions. "Yes, the reactor is running. No, we haven't shot any weapons." The point here was to make 100% sure the CO, the person with ultimate responsibility, was aware of the condition of the boat.
There was a long list of items that required immediate phone calls up the chain of command. Not a text, email, or a heads up the next morning. Call. Now.
Someone goofed up and didn't perform a proper tagout? A valve was found in the wrong position? People need to work past 8pm? Somewhere between 5-8 people, including the CO, are getting called for any of the items on this list. Leadership demanded to be aware of changes in material condition, personnel problems, or management failures.
Overcommunicating is not without its drawbacks. One submariner I mentioned this to reflected that "overcommunicating...pulls the experts away from executing their tasks."
Attention to quality requires precision in communication. We did not permit guesses to knowable questions.
A "knowable" question was one that had a discrete answer. For instance, "What time is sunset?" "What's the status of torpedo tube #1?" "What time are we due at periscope depth?" All of these questions are answerable, and "Uh, approximately..." or "I think..." were not permissible responses. If I didn't know the precise answer when questioned, I was to say, "I'll find out."
Real harm can come from guessing/swagging/approximating the answers to knowable questions. Hearsay and "tribal knowledge" becomes accepted over documentation and Truth.
When communicating, units were required when discussing numbers. If I failed to include units, I'd have to repeat myself.
"Report the level in the tank."
"500 gallons." ✅
It's easy to find this exasperating. Obviously, the tank levels are in gallons, air pressure is measured in psi, distance is in yards, etc. But nuclear engineering demands precision, and precision was demanded in all things.
Audit and Inspect
Managers on submarines spend a significant portion of their time auditing and inspecting work. In fact, their process of auditing and inspecting is itself subject to audits and inspection! 🥴
It's a little boring for me to just describe the process. Just try imagining a world where this is standard to your product organization:
You are a senior product manager and today you're reviewing the work on ticket MKTECH-123. As you get ready, your VP of Product messages that he'll be joining you. The VP joins your meeting room and you pull up the ticket. While he silently watches, you verify the following:
- Is there a clear user story?
- Is the acceptance criteria clearly defined?
- Is the status of the work correct?
You then message the dev assigned to the ticket that you're ready for them.
They join the room and you ask some basic questions:
- What was the acceptance criteria for this ticket?
- How did you ensure you met this acceptance criteria?
- Who performed the code review? How was the person chosen?
- Can you show me the pull request associated with this ticket?
You thank the dev for their feedback and they leave the room. You review your note with your VP:
Let's see...the ticket was marked "in review" but the pull request was fulfilled. Developer #1 said the code review was performed by Developer #2 so I think they just forgot to update the status of the ticket. I will make sure we cover 'updating ticket status' at our next retro.
The VP agrees with your comment and proposed plan to correct the deficiency. He then notes:
Are you comfortable with Developer #2 performing that code review? Why? What familiarity do they have with that part of the codebase?
You, the PM, don't know as that's something the engineering manager handles. You tell the VP you'll find out and get back to him on that question. He tells you that you have until the end of the day to get back to him.
On submarines the process of regular audits and inspections ensure the engineering organization has an immediate feedback loop on the quality of their processes and work.
Hold Managers Accountable
The accountability of the manager was so beaten into us that one guy even (seriously) told his boss, "My inadequacies are a reflection of your leadership." That may have not been the...smartest...thing to say, but it reflected a deeply ingrained value: it's always the manager's fault.
I think this principle of accountability is intuitive, but what does it look and feel like?
- Getting woken up because a radioman didn't understand the urgency of a message and the Captain is angry that it hasn't been sent yet. Why didn't you properly convey the urgency to him?
- Giving an unsatisfactory brief and watching your boss get yelled at instead of you. Why didn't he prepare you adequately?
- Getting temporarily relieved of duties because someone under your cognizance hit the wrong button. Why didn't you stop him?
Managers felt the heat to deliver quality work.
It's important to note that high accountability does not mean "fire fast." The Navy doesn't have the ability to make outside hires: there are a finite number of individuals of a given rank that they can draw from. As a result, senior managers are expected to provide immediate, raw feedback and coaching to their reports.
Quality in the Submarine Force is driven by extremely tight feedback loops. To create the tight feedback loops we overcommunicate our precise intentions and actions. We routinely audit and inspect processes. Accountability of managers ensures they recognize and enforce standards of work.
If that sounds onerous and burdensome, it may well be! However, the Submarine Force points to its record of safely operating over 50+ nuclear plants every year for the past five decades to justify its quality-focused culture. Repeatable quality isn't easy.
I have not explored what my colleague Russ Rodewald (sorry no Twitter presence) left me with:
How does this dovetail to how you prepare a software team for example? Where do you tolerate faults in order to build a stronger team? How do you pair people up to allow both delivery and learning?
Perhaps more to come...when I have those answers.