Fort Meade, MD --
In October 2016, Dr. Eric Schmidt, then the Executive Director of Alphabet, Inc., Google’s parent company, and chairman of the Defense Innovation Board (DIB), independent technology and capability advisors to the Secretary of Defense, were touring the Combat Air Operations Center (CAOC) at Al Udeid Air Base in Qatar.
Within the CAOC, U.S. Air Forces Central Command (AFCENT) oversees air operations over many countries across the Middle and Near East. Those missions and their planning and assessment generates massive amounts of data – including that used to coordinate aerial assets fighting ISIS in Iraq.
Then, among the sea of computers, displays and Airmen handling that data, the members of the DIB spied something curious – a group of Airmen crowded around a dry-erase board moving around different colored magnets.
The technology experts were amazed to learn that the team was planning air refueling missions by entering mission data on an Excel spreadsheet and visualizing that data by drawing lines with dry erase markers and arranging the magnets to organize each day’s missions. If the data, available assets or ground missions changed the team had to erase everything and start over.
The software they had to plan missions was useless, dating back to Desert Storm, and the contracting process to update it had been hopelessly bogged down in the acquisition process for years.
Traveling with the board was Raj Shah, then Managing Director of Defense Innovation Unit Experimental (DIUx). He had a team back in Silicon Valley, led by Col. Enrique Oti, developing a CAOC planning tool in partnership with Pivotal Inc., which was already hosting Air Force software engineers at their San Francisco office to learn commercial software development practices.
He made a phone call and told the AFCENT command that they could deliver a solution in months by having the Defense Innovation Unit Experimental (DIUx) team send their coders, designers, product managers and Air Force software engineers to work side-by-side with CAOC personnel, thus enabling them to provide feedback on-the-fly.
The Air Force team, along with their Pivotal coworkers, flew to Al-Udeid and sat side-by-side with Airmen in the CAOC to design a warfighter-friendly tool.
The resulting tanker planning software, called Jigsaw, turned an eight-hour task for six people using magnets and markers on a whiteboard into one that a single person could accomplish in three hours on a touch-screen interface.
The collaboration not only provided a solution for the Air Force, but, perhaps more importantly, that DIUx collaboration with Pivotal was transitioned to Air Force Life Cycle Management Center (AFLCMC) and is now an ongoing Air Force agile software development program called Kessel Run. The program combines Airmen coders with Pivotal software experts to effectively and efficiently provide the Air Force software, hardware and technical solutions.
According to Capt. Bryon Kroger, Air Operations Branch Chief of Kessel Run, the tanker planning tool was created and operational in only 120 days and cost the Air Force about $1.5 million.
After six months of input from CAOC users, software updates and analysis, “the Air Force determined it had recouped its cost for the software in a single week,” Kroger said. “The efficiencies it had created was saving about 400,000 to 500,000 pounds of fuel each week and they were accomplishing their missions with one less refueling aircraft. This saved the Air Force $750,000 to $1 million every week.”
Over The Waterfall
Secretary of the Air Force Heather Wilson said during her keynote address this year at the Air Force Information Technology and Cyberpower Conference that the predominant process used by the Air Force to acquire software is archaic and needs to modernize now if the service is to win in a peer-to-peer conflict.
In response, she has “turned loose” the Kessel Run developers to work directly with warfighters to quickly develop software solutions. The program has just opened a new lab in Boston.
This shift to a new software development paradigm was born from a spectacular failure of the old acquisition system.
In July of 2017, after the Senate Armed Services Committee balked at shifting resources from other areas to continue supporting the project, the U.S. Air Force terminated its AOC 10.2 modernization contract with Northrop Grumman to upgrade Air Operation Center networks used to plan and conduct air operations, like mid-air refueling.
After nearly a decade of development, costs had ballooned from $374 million to $745 million and the program was more than three years behind schedule.
The failure was not the fault of people, but of the acquisition system itself, according to Lt. Col. Jeremiah Sanders, Deputy Director of Kessel Run, who is a career acquisitions officer and the material leader and program manager for the Air Operations Weapons System at Hanscom Air Force Base, Massachusetts.
“The acquisition model in the DoD is really modeled after that industrial base of ‘I’m going to build some big thing’, be it a plane or a ship, a tank, whatever,” said Sanders. “We take a lot of time upfront to try to build a plan and understanding exactly what that thing is. That requirements process can take anywhere from two to five years… This is all in an effort to avoid program risk. You have one chance to lay the keel of a ship, so you need to be very deliberate about it.”
However, this industrial-age approach to risk mitigation, often referred to as the Waterfall Method, is from a time when technological change was comparatively slow and is not in tune with the rapidly changing technological environment of today’s digital age.
“In software related fields, we’ve learned from commercial industry that an upfront planning process and that deliberate layer upon layer of detailed requirements definition is actually increasing risk of delivering the wrong thing… from a functional perspective it won’t meet the need of the user,” said Sanders.
“Designing, developing and fielding a software product builds in complexity over time. If it’s not tested in an operational environment until final delivery, often years later, you just might be building something that ultimately won’t work.”
DevOps — short for development operations — is based on delivering a minimum viable product as quickly as possible to the warfighter, in months or weeks instead of years, getting real-time feedback from the end-user and continually upgrading the system’s capabilities to address user needs, known as iteration.
Warfighters In The Loop
One of the other benefits of the DevOps approach is that warfighters finally feel heard as their feedback is making it back to the people who can provide them with capability, according to Kessel Run Air Operations Lab Director, Adam Furtado, who spent eight years as an Air Force intelligence analyst working at AOCs and within the targeting community.
“We talk to them on basically a daily basis. We do a lot of user testing where we fly out to the sites and we shadow them. We also do user interviews over video conferencing software,” said Furtado.
“Nothing we’re doing is truly innovative when you compare it to what’s been happening in the commercial world for decades. But it is innovative within the DoD. We are taking those commercial principles and applying them so the warfighter is part of the development process and gets the right solution quickly.”
Kroger, who is stationed at Hanscom Air Force Base, is one of the project’s founders and now runs the day-to-day operations. After seven years of targeting assignments and targeting deployments, as a 14N core AFSC intelligence officer, he applied to cross-train as an acquisitions officer.
“Part of my motivation was the fact that I was an end user and I was unhappy with my software,” said Kroger. “My software always worked against me. It never worked for me. And that’s not the way software should be. I thought there had to be a better way.”
His initial outreach to DIUx for support in transferring a portfolio of targeting software to the cloud snowballed into what is now Kessel Run.
Kroger says that while coders need to empathize with the users, they must base their solutions not on user wants, but on the problem to be solved.
“Henry Ford said, ‘If I had asked people what they wanted, they would’ve said a faster horse.’ If you go to the user and ask them what they want, a lot of times they’re going to get that faster horse response,” said Kroger.
“You have to break it down into what the users’ real problem is and design and deliver a solution to the problem. Getting from point A to point B might be a horse, might also be a model T.”
The key, says Kroger, is to quickly deliver that minimum viable product to the field of operations, so the warfighter can start using it immediately without disrupting their existing workflow. Then begin making incremental changes, additions and upgrades, based on user feedback.
“So we don’t build them a whole car right away—they would be waiting years to get the first value from our work. We say, ‘instead of a car, maybe we give you this skateboard first,’ said Kroger. “Then they say, ‘that works, but it’d be great if you gave me something to help balance.’ Then we add a handlebar and give them a scooter. Eventually, they get a car, but along the way they got value and we got valuable learning.”
Kessel Run’s methodology has led to other software solutions for the Air Force. Airmen were using 22 different applications to manage the dynamic targeting process. Because there was no unifying software, they had to enter the same information up to a dozen times in a dozen different forms among all those applications.
When Kessel Run knocked the number of applications down to 17 with its first iteration of “Chainsaw”, Airmen only had to enter coordinates once and they automatically populated the appropriate fields on all the forms.
Sharing The Story
In order to provide the benefit of problem-solving products for the Air Force, Kessel Run must have coders who embrace agile software development concepts. To answer its manpower demands, Kessel Run recruits coders from across the Air Force to come to Boston and learn industry-driven software practices.
It was a tough sell to commanders at first, but they have largely come to see it as a win-win proposal.
“It’s tough for squadron commanders to let some of their best talent come to us for six months to be able to learn these new skill sets and how to build capabilities in this new modern way,” said Furtado.
“But what they’re getting back is an Airman who not only has really good software development talent but is [also] able to solve complex problems in a different way, using lean startup principles and user-centered design. So, while we’re getting the benefit of manpower from around the Air Force, I think that we are also being a force multiplier for the Air Force.”
Those that are accepted walk into a workspace that is light and airy with no walls or cubicles to hinder communication, teamwork or creativity. There is a kitchen and common area stocked with snacks, sodas and coffee. Frequent breaks are encouraged to facilitate creativity – a la Silicon Valley.
The work environment at Kessel Run also embraces a flat command structure on the development floor – there is no rank and there are no uniforms.
“When I interact with my Airmen while we’re building software – they’re the coders, not me, so it should be their ideas that matter most, not mine. That’s the culture that we want to engender here. It’s a very flat organization and the best ideas win,” said Sanders.
Kroger adds that design teams are encouraged to embrace the commercial industry concepts of “pair programming” and “failing fast”.
Two Kessel Run programmers sit at the same computer working on the same story – a portion of software code – according to 1st Lt. Justin Hohman, practice lead engineer for Kessel Run.
This enables a second set of eyes to catch potential conflicts and brings another perspective on avenues to explore for possible solutions. Those pairings rotate each day within each of the 13 development teams, which helps to provide context to all team members on the project.
The code is tested as it is written for functionality and cybersecurity. Then logged into a library each day to test compatibility with what other teams produced, known as continuous integration, according to 1st Lt. John (last name deleted for OPSEC), a lead engineer for the 7th Intel Squadron capabilities testing team, at Fort Meade, Maryland, who worked at Kessel Run for six months.
“If a piece of code fails, the team knows it quickly and can change it or pursue other ideas… That way, you don’t develop this giant merge conflict that’s impossible to work through. It makes it easier to integrate a new feature you’re working on with the existing code base,” said 1st Lt. John.
After integrating some of the DevOps concepts into his team at Fort Meade, 1st Lt. John believes that the Air Force will gain exponential benefit as Airmen like himself, spread the Kessel Run culture and methods. He even believes that Kessel Run could end up being a model for the Air Force programming tech school of the future.
“So much that you learn in tech school now, you don’t actually use in your job. Things you’re learning could be outdated by the time you get to your unit and it takes so long to update some of those training pipelines,” said 1st Lt. John. “An agile and immersive experience like Kessel Run would make a great way to train.”
As news of Kessel Run’s success spreads throughout the Air Force, new customers are seeking them out to provide solutions.
Cameron Conger, technical director for the 363rd Intelligence, Surveillance and Reconnaissance Wing at Langley Air Force Base, Virginia, journeyed to Kessel Run shortly after the new lab opened in Boston, to kick off a project to completely rethink the process of target systems analysis, the way by which the Air Force understands potential adversaries in greater detail.
The wing then embedded three airmen for six months to support Kessel Run’s effort to create a software solution for the 363rd ISRW as well as to build their own software skills.
“The most important thing we can achieve through some of the modernization with Kessel Run is taking the rote and repetitive tasks that our Airmen have to do in the ISR world, simple but time-consuming tasks, and get those off their plate so they can really focus their time, energy and their mental bandwidth on doing real high-end analysis… to spend less time on sifting through information and more time on making sense of it, “said Conger.
“This agile development capability that Kessel Run brings to the Air Force will help us get there. They delivered the first minimum viable product on a highly-classified network for the 363rd in less than half the time the traditional process used to require.”
Risking Change, Changing Risk
Sanders believes Kessel Run not only answers Air Force needs but its methodology provides more accountability for program managers, leadership and, ultimately, Congress. Still, he sees a long road ahead as ultimately a change in culture is needed to get people to learn a new way of doing business.
The Federal Acquisition Regulation, coupled with the DoD 5000.x (acquisition regulations specifically for the Department of Defense), was written at a time when the Air Force was basically acquiring planes and rifles. For the most part, it’s not quite geared for how fast technology is moving today, according to Tori Galvin, Agile Acquisitions Branch Chief for Kessel Run, who runs the acquisitions arm of the project.
But there is some latitude if you know where to look.
“The FAR says ‘timely delivery and software that is a valuable [asset] to our war fighter’. We’re just taking it one step further – instead of just having a timely delivery, we want to have a continuous delivery. And that’s really where the breakthrough is,” said Galvin.
“You have the ability to create your own model if one of the models that they’re recommending for acquisitions doesn’t work for you. They’ve put in flexibility there. It’s just about getting leadership to buy in on utilizing something that’s they’re not traditionally used to seeing.”
She says the biggest hurdle is often how people define risk.
“We’re talking to Congress right now and it is a difficult conversation when you say, I can’t tell you what I’m building next year, but I can tell you that working in this way it will be valuable. For them, that’s a leap of faith,” said Galvin.
“They say, ‘At least tell us what products.’ And we have firmly pushed back on ever committing to products a year or two years out. We’re firm on that belief because we want to be able to allow our teams the flexibility and not get locked into a plan that we know we can’t march to.”
In that endeavor, Sander’s, Kroger, Furtado and Galvin and the rest of the Kessel Run program express gratitude for senior leader champions like Secretary of the Air Force Heather Wilson providing top cover for their island of innovation.
“These Airmen are helping our Air Force be more ready and more lethal,” Wilson said of Kessel Run.
“They are solving problems that Airmen have given them, whether it’s from the flight line or an air operations center. We have given them some tough problems, and we’ll see how this goes. It doesn’t matter to me if they get it right the first time. It matters to all of us that we keep innovating constantly, rather than sitting back and analyzing people for failure.”
