Friday, December 6, 2013


The predictably dismal performance of the rollout reminded me today of some other computing revolutions that I was once a part of. Way back in 1984, my first ship was the Middle East Force flagship, USS LASALLE. When I reported aboard in March it had a total of one computer. (We don't know what was in SSES, do we? No. No we don't.) It left the ship and Bahrain on the C5 I arrived on after a little mishap with some coke and an Operations Specialist in the Combat Information Center. Six months later it was returned and locked up. It was pre-JOTS-In-A-Box but I have no idea what it was used for because it was secret. Right about then, the Chief Engineer returned from leave with a brand new Commodore 64 computer. The only real computer on the ship. It didn't contribute much to the war effort.

There were other significant computing milestones in my navy. Remind me to tell you about them sometime. My next major entanglement with navy computing happened a year after I reported aboard Space and Naval Warfare Systems Command when the Navy and EDS rolled out the Navy Marine Corps Internet. It was an unqualified disaster that took YEARS to resolve. From: DTIC.
Abstract : In October of 2000, the Navy's leadership entered a multi-billion dollar IT service contract with a private company to build and maintain the Navy-Marine Corps Intranet (NMCI). The hope was to have the new intranet fully operational in just two years, but the program encountered so many difficulties that, almost six years later, the initial implementation process is still underway. Aside from the unexpectedly high number of applications that needed to be migrated to the new network and the repeated attacks by members of Congress and other government agencies, by far the largest obstacle to NMCI s success has been the end users resistance to change. The Navy s leaders underestimated the significant cultural change brought on by the implementation of NMCI, and as a result, they were not adequately prepared to deal with the negative user response. After providing a historical account on how NMCI was conceived, planned and delivered, this thesis goes deeper into NMCI s implementation process by recounting the experiences of those who used NMCI at the site level. Once the history and site case study are presented, this thesis ties in the theme of change to show how proper communication can facilitate the success of future transformation initiatives.
 The gory bits are in the thesis which is here.

What happened is spelled out very well in Mr. Taylor's Thesis. A "good idea" was had at the very top by profoundly important people who directed their minions to find a loophole in the Acquisition Rules that would allow the Navy to do an end-run around ALL of Congress and implement the little SIX BILLION dollar contract without troubling the Congress. That was profoundly stupid.

Then they grossly underestimated just how much the navy relied on computers and applications unheard of within the Beltway.

Then they really screwed up and chose what they called a "disadvantaged" navy facility and the very heartbeat of the Navy's Command, Control, Communications, Computers, Intelligence Surveillance and Reconnaissance (C4ISR) as the first two sites to implement the initial roll out of NMCI. Guess how the people that work at the core of C4ISR felt at being totally excluded and left out of the loop on designing, building and installing the biggest thing in navy computers since POLARIS? And the profoundly important people thought that using them as fodder and unheralded minor guinea pigs in the network catastrophe was a "good idea."

The NMCI took up to an hour to log onto and up to an hour to log off. A lot of the time it didn't log me off and I just turned off the machine. We all did. We started to complain and found ourselves promptly labeled as resistant to change and unwilling to change our ways. It was no use pointing out that our work required access to working computers and the network and if change was necessary couldn't we wait until after EMP took the whole thing down worldwide before we re-adapted to working in the 19th century?

The profoundly important people decided that the time and dollars spent qualifying and approving all the myriad applications that people used to do the work was too burdensome and so they just took an axe to whole swathes of programs. They axed a lot of purely Navy programs. Obviously, they quickly paid through the nose to buy newer better and NMCI compliant software from contractors who got serious friction burns on their palms from rubbing their hands together in glee.

The process was so fiendishly complex (and made even more so in every way by the gracious researchers, scientists, technicians and programmers) at Space and Naval Warfare Systems Center San Diego, that the profoundly important ones decided, after a year or two of open warfare, to permit the SSC people to have an exemption from NMCI and retain their own networks, computers, servers, etc.

When I look back and compare the implementation of NMCI with the implementation of Obamacare I am surprised by how they both managed every pie.
"And then Epaminondas was careful how he stepped on those pies!
He stepped - right - in - the - middle - of - every - one."


Buck said...

I took the early-retirement buyout EDS offered me in late 1999, so I can assure you I was in NO way part or party to that lil adventure known as NMCI. ;-)

That said, EDS also ran me pretty hard to join their nascent Military Systems Division in the early 1990s. I politely declined those invitations... mainly coz I KNEW what a CF the military procurement process could be. I wanted NO part o' that krep.

HMS Defiant said...

I think you chose wisely. EDS probably executed the flawed NMCI about as well or better than anybody could have given the environment and the contract. I've been on both ends of the wide-open subject to interpretation military contracting tussle and a company has a right to expect to be paid for services rendered. We were always careful (at both ends of the arrangement) to make the milestone payments something that was immune to interpretation simply based on what happened with NMCI. It was pretty straightforward: design review-pay them, first article delivery-pay, first article testing and certification-pay. Full rate production-pay incrementally for each delivery following system operation and testing. With the last one my govie boss was trying to hold the contractor hostage for the government's failure to deliver the equipment to a unit for verification and testing. The problem being that the Navy had no available units to take delivery since they were all deployed by the war. It took a sledge hammer to make her see that this was no fault of the contractor who had delivered. IT TOOK MONTHS to get that unscrewed.

virgil xenophon said...

Ye Gods! I've been out of the loop for TOO long. I intuitively SUSPECTED it was bad, just based on general experience with all things military, but MY GOD!

HMS Defiant said...

It was foredoomed by the need to work from the outset. It is one thing to put a new system in place at a lonely little base in the middle of nowhere but to come into the Navy Research and Development Command, Navy In Service Engineering Agent and the most advanced labs in the navy and trying to tell those guys that there email and network availability was going to average 3 to 12 hours of downtime every day while 'teething' problems were worked out was a huge mistake.