June 13, 2024

Enhanced DoD Wargaming According to GAO and Relevance to Hobby Wargaming

RockyMountainNavy, 11 May 2023

In late-April 2023, the United States Government Accountability Office (GAO) released a report to the House Armed Services Committee titled “Defense Analysis: Additional Actions Could Enhance DOD’s Wargaming Efforts” (GAO-23-105351). The timing of the report release is a bit interesting, coming in the same week the House Select Committee on Strategic Competition Between the United States and the Chinese Communist Party participated in a highly publicized wargame on the Taiwan Straits run by the Center for New American Security (CNAS). [The CNAS read-out is available here.] While the GAO report is certainly aimed at wargame practitioners, a study of the common quality principles identified can also be useful for hobby wargame designers, developers, publishers, and even players to consider.

FILE PHOTO: House Select Committee on the Strategic Competition Between the United States and the Chinese Communist Party holds a meeting on the wargames simulation “Taiwan Tabletop Exercise (TTX),” on Capitol Hill in Washington © Thomson Reuters 

click images to enlarge


The GAO website for the report provides some “Fast Facts” on what the report is about:

“‘Wargames’ are simulations of conflict that DOD uses to prepare for actual warfare. Organizations within DOD and external providers—such as federally-funded research centers and contractors—provide wargames. But DOD hasn’t assessed the risks of relying on external providers.

“We identified steps DOD takes to ensure quality wargames. However, DOD stores wargame information in systems that don’t share data with others, which hinders collaboration. Also, none of the five military services have established standard education or qualifications for their wargamers.”

Our 10 recommendations would improve wargaming across the department.”

The purpose of the study is described by the GAO in this manner:

“Wargames are an analytic method that can provide valuable insights to complex problems and inform decisions about warfighting concepts, capabilities, and plans. DOD credits wargames with making key contributions to military planning for pivotal operations in the Pacific during World War II. In 2015, DOD began an initiative to reinvigorate wargaming in line with the strategic shift to prepare for near-peer power competition. GAO was asked to review DOD’s use of analytic wargames. This report examines: (1) the scope of DOD’s wargaming activities; (2) DOD’s use of internal and external wargame providers; and (3) the extent to which DOD ensures wargame quality.

GAO analyzed wargame data for fiscal years 2017 through 2021; relevant DOD guidance and documentation; and leading practices for wargame quality. GAO also observed wargames and interviewed wargaming officials from DOD and external wargame providers.”

The GAO wargaming report section on “the extent to which DOD ensures wargame quality” should be of interest to those in the hobby wargame space, be they designers, developers, publishers, or players. It is in all those parties best interest to see quality wargames published. It is possible that a study of the GAO wargaming report could help.


What’s a wargame?

The GAO report pulls few punches and starts off by making it clear what it defines as a wargame:

“Wargames—representations of conflict in which the game’s players make decisions and respond to the consequences of those decisions—are used widely across the Department of Defense (DOD) to provide insights on challenges and to inform decisions. DOD and other wargame providers plan, conduct, and analyze wargames (see figure), and DOD uses the results in conjunction with other analytic methods like military exercises and modeling. For example, in May 2022, GAO observed a Navy contested logistics wargame that included over 200 participants from over 40 organizations across DOD as well as international partners. The Navy planned to use the results to inform logistics planning in support of a distributed naval force.”

The CNAS wargame for the House Select Committee on China referenced above is a very good example of the type of wargame DoD often engages in. Although the CNAS read-out describes the event as a “tabletop exercise” it fits the GAO definition of a wargame focused on decision-making:

“On April 19, 2023, the Center for a New American Security’s (CNAS) Defense Program and the Gaming Lab ran a tabletop exercise that examined how the United States could use diplomatic, military, and economic tools to defeat a Chinese invasion of Taiwan with members of the House Select Committee on Strategic Competition Between the United States and the Chinese Communist Party (CCP). This game focused on American and Chinese strategic decision making and sought to provide insights into what the United States could do to win should deterrence fail. The exercise also sought to provide committee members with an understanding of the range of potential Chinese options in line with their strategic guidance and military doctrine. The ultimate goal of this TTX was to identify how to strengthen deterrence and uphold peace in support of a free and open Indo-Pacific.” (“Bad Blood: The TTX for the House Select Committee on Strategic Competition Between the United States and the Chinese Communist Party (CCP),” CNAS, April 27, 2023)

Screenshot 2023 04 28 at 2.55.01 PM
Courtesy CNAS “Bad Blood”

Without starting yet another “What is a wargame?” rant  argument  discussion I will say that the GAO definition is actually very progressive [using that word in a non-political, positive way] as it gets away from defining a wargame by how it is presented and instead focusing on the objective of a wargame—which is to explore decisions.

The GAO wargaming report focuses on providers of wargames which it identifies as a mix of organizations internal to the DoD, federally-funded research & development centers (FFRDC), and contractors. As the GAO report highlights, “The mix of wargame providers used across DOD varies and comes with advantages and disadvantages including varying capacity, timeliness, information access, expertise, and independence.” The GAO comments that the DoD has never assessed the use of this variety of wargame providers and therefore resources for wargaming may be misaligned.

The wargame hobby also has a variety of game “providers.” Comparing big and small publishers, independent designers, amateur and professional, similar advantages and disadvantages become apparent. As much as wargame practitioners and hobby wargamers try to see themselves as unique from each other, the GAO report serves as a reminder that both are in many ways wargame compatriots.


The DoD wargaming process

The GAO wargaming report has a simple graphic on the DoD Wargaming Process.

Screenshot 2023 04 28 at 12.47.57 PM

Even with my limited experience, I see that this Process does take time into account. In the earliest step of the DoD process a sponsor recognizes they have a problem that might benefit from wargaming and then pursues that path. All too often, however, wargaming is thought of late in the defense or acquisition planning process or the sponsor puts forward an insufficiently scoped question that demands a wargame impossible to deliver within project timelines. Does that not sound like the same issue hobby wargame designers, developers, and publishers always face when trying to scope a game in design, development, or production to meet customer demands?

It is important to note that playing the wargame is not the end of the DoD process but the midpoint. This is very different from hobby wargaming where the process that started with design and then production and delivery “ends” with the wargame in the hands of players. Granted, there is a bit of a “feedback” loop after play in the form of reviews and questions (and the dreaded “errata”) but generally speaking the hobby wargaming process ends with the players playing the game. Contrast that with the DoD Wargaming Process that continues past play with game analysis and the potential for applying that analysis to future decisions or to drive research.

Simply put, DoD wargaming is often for analysis whereas hobby wargaming is focused on providing entertainment (yes, I consider any “learning” from hobby wargames a subset of entertainment).


Quality principles in quality frameworks

The GAO wargaming report notes that, “While DOD organizations conducting wargames use no single quality framework, the frameworks used by DOD organizations share common quality principles.” The 13 common principles identified by the GAO are divided into three broad areas: Wargame design and development, Wargame conduct, and Wargame documentation and analysis. Some of these quality principles should be considered by hobby wargame designers and developers, though occasionally in an indirect or abstracted manner.


Wargame design and development

“Sponsors and designers should coordinate to set appropriate wargame objectives.” From a hobby wargame publisher’s perspective, I see this as akin to a publisher “knowing their customer.” The rise of pre-order systems like the GMT Games P500 program is but one expression of how publishers try to gauge a players interest in playing a title (the “objective” of the hobby wargaming process). For a hobby wargame designer the objective might also attempt to satisfy a deep personal interest…which doesn’t always find an audience (or publisher).

“Wargame play should focus on activities that are well-simulated by wargames.” Personally, the hobby wargamer in me disagrees with this GAO statement. I hope instead that every hobby wargamer agrees with me that wargames should not be afraid to explore on a wide variety of topics. On the surface, this principle appears to be a call to use wargames in, for lack of a better phrase, a classic or traditional manner. I fear in the DoD context this principle will be interpreted to limiting wargames to military confrontations and exclude wargaming from use in exploring other areas of competition. I point to the work of wargame practitioners—like hobby wargaming’s own Mark Herman who double-hats supporting policy wargames—who use wargames to frame policy issues ranging from space to cyber to nuclear escalation as worth consideration.

[For example, in the research journal article “Cyber Operations and Nuclear Use: A Wargaming Exploration,” the authors used an “an experimentally-designed war game to explore the role that vulnerabilities and exploits within a hypothetical NC3 [nuclear command, control, and communications] architecture play in decisions to use nuclear weapons. The game, which uses 4-6 players to simulate a national security cabinet, includes three treatment scenarios and one control scenario with no vulnerabilities or exploits. Players were randomized into the scenario groups and games were played over the course of a year in seven different locations with a sample of elite players from the U.S. and other nations. Together, a longitudinal analysis of these games examined the role that culture, cognitive biases, and expertise play in the likelihood of thermonuclear cyber war with significant implications for both cyber strategy and nuclear modernization.]

For the hobby wargame industry, I am very happy to see the wide variety of topics that are the subject of wargames today. In this respect the hobby wargame industry is certainly ahead of where the DoD is right now.

“Wargame design should include an analysis plan that involves appropriate data collection and analysis.” The most direct application of this principle for hobby wargame designers it to have a data collection plan during play playtesting. This principle is intended to apply to the later steps of the DoD Wargaming Process (analysis and further research) but it is equally relevant to hobby wargame designers at the start of their design process. How many times have we seen a hobby wargame that claims to be historical but isn’t? Or uses a single viewpoint or data set as the basis for the design? I believe that if a designer “does their homework” in advance many of these types of problems/issues can be avoided.

“Players should be selected based on their expertise related to necessary game roles.” Alas, hobby wargame designers and publishers don’t get to select their players. That said, the wargame hobby has often been characterized in an unflattering manner. Jim Dunnigan, in the earliest editions of his Wargaming Handbook, was very frank in his characterization of the wargaming hobby which he described as “the hobby of the overeducated” (James Dunnigan, Wargaming Handbook, Third Edition, Writers Club Press, 2000, p. 300). Recent clarion calls for “accessibility”and “inclusion” in the wargaming hobby are efforts to break what some feel is a culture of exclusivity and self-selection.

“Scenario design should have flexible methods available and should select a design to best address the wargame objectives.” This is an area I believe hobby wargame designers, developers, and publishers excel at. Hobby wargame designers certainly take advantage of many innovative design approaches when creating a wargame. I believe this is an under-appreciated aspect of the wargaming hobby. I recently saw some discussions that pointed out the revolutionary advances in hobby boardgame design has made since the mid-2000’s and laments that hobby wargame design was stagnant. I disagree. I believe that hobby wargaming is actually a vibrant and highly evolutionary hobby. For too long hobby wargaming has tacitly accepted the label of being nothing but hex & counter with a die roll on an odds-based combat results table (CRT). Accepting that label totally ignores decades of wargames that use hexes or areas or points on a map or tiles or cards. Or counters to blocks to minis. Or an odds to differential to diceless combat resolution. It also fails to acknowledge game mechanisms like card-driven games (CDG) or asymmetric conflict as in the COIN series. Or bots and card-driven AI (“solo assistants,” if you will). For example, in the 15 “wargames” I personally have acquired to date in 2023, not a single one is a wargame using hex & counter with a traditional odds-based CRT. Instead, only seven of the 15 wargames use hexes whereas three are miniatures rules and four use area or point-to-point maps (the last is a card game where the cards form the battlefield somewhat similar to tiles). If I truly wanted to be expansive in my definition of a wargame, I could add in four other games that I characterize as boardgames but some consider wargames. Is hobby wargaming truly stagnant with designs like the adventure wargame Firefly Adventures: Brigands and Brownouts (Gale Force Nine, 2018) or the historical strategy cubes-as-political influence CDG Votes for Women (Fort Circle Games, 2022) or the 2023 COIN-inspired Robotech: Reconstruction (Strange Machine Games, 2022) and the semi-cooperative CDG of Land and Freedom (Blue Panther, 2023)?

“Players should have access to information, databases, and models that support game decisions.” Tabletop hobby wargame players usually have access to the information used in play but not always to the database behind the research or design. A notable example of an accessible database are the data annexes for Admiralty Trilogy Games like Harpoon V. Access to the models is harder to quantify; if one looks closely at the documentation for Admiralty Trilogy Games you can find virtually all the “model” in there. What the GAO report is driving at with this principle that doesn’t always align with hobby wargaming is that the models—and especially adjudication—should be transparent to players. One pitfall of defense wargaming is the occasional reliance on “expert adjudication” which is subject to undue influence. The banner example of this pitfall are the decisions by the Imperial Japanese Navy when the Battle of Midway was being wargamed. Mitsuo Fushida describes the event in his book Midway: The Battle that Doomed Japan, the Japanese Navy’s Story (Naval Institute Press, 1955, 1992) thusly:

“Still more amazing, however, was the manner in which every operation from the invasion of Midway and the Aleutians down to the assault on Johnston and Hawaii was carried out in the games without the slightest difficulty. This was due in no small measure to the highhanded conduct of Rear Admiral Ugaki, the presiding officer, who frequently intervened to set aside rulings made by the umpires.”

“In the tabletop maneuvers, for example, a situation developed in which the Nagumo Force underwent a bombing attack by enemy land-based aircraft while its own bombing planes were off attacking Midway. In accordance with the rules, Lieutenant Commander Okumiya [incidentally, Fushida’s co-author], Carrier Division 4 staff officer who was acting as an umpire, cast dice to determine the bombing results and ruled that there had been nine enemy hits on the Japanese carriers. Both Akagi and Kaga were listed as sunk. Admiral Ugaki, however, arbitrarily reduced the number of enemy hits to only three, which resulted in Kaga’s still being ruled sunk but Akagi only slightly damaged. To Okumiya’s surprise, even this revised ruling was subsequently cancelled, and Kaga reappeared as a participant in the next part of the games covering the New Caledonia and Fiji Islands invasions. The verdicts of the umpires regarding the results of air fighting were similarly juggled, always in favor of the Japanese forces.” (Fushida, p. 124).

Hobby wargame designers and publishers may find it difficult to be totally transparent as wargames often demand abstraction and simplification that is not always apparent to players.

Screenshot 2023 05 03 at 7.41.14 PM Screenshot 2023 05 03 at 7.41.32 PM

“Wargames should have an assigned control group of adjudicators or umpires to manage game actions and determine outcomes of actions.” In defense wargaming, players almost always don’t have time to learn the game in advance of event execution, and therefore are best served by having a guide. This is very clearly the play model implemented by Sebastian Bae for his DoD version of Littoral Commander: Indo-Pacific (The Dietz Foundation, 2023) which is unashamedly called an “educational wargame.” This is also the secret behind the success of many teach-n-play events at hobby gaming conventions.

IMG 4229
Ignore the backwards “70” it’s still an educational wargame!


“Wargame designers should document their design through rules that can be understood by the players.” In the hobby wargame space, the design of many games is presented in the rules, designer notes, or even play books. This principle also intersects with those who call for wargames to be more “accessible” as mostly evidenced by their demand for wargames rules to be written in a more approachable manner. All that is evidence that wargame practitioners and hobby wargame designers face a similar problem in presenting rules that can be easily understood.

“Wargame design should be tested through a development process.” This is a lesson that the most successful wargame publishers embrace. That said, there still are some companies that somehow get by on barely-developed game designs and without much development support to designers. Generally speaking, I believe this is a dying trend as publishers have discovered the monetary value (i.e. stronger sales driven by reputation) of investing in hiring a developer and actually taking the time (and effort) to develop wargame designs.


Wargame conduct

“The adversary or “Red Cell” should be played by relevant experts in a faithful but unpredictable manner.” The GAO report points out that the Red Cell is usually staffed by either intelligence analysts responsible for analyzing the threat or operations staff who understand the warfare domain in question. Hobby wargame players are usually not “experts” in a field though they may be experts or more experienced in certain game systems. Want to play OCS? Call Ardwulf! Where this principle is most applicable to hobby wargaming is perhaps when solo game design is considered. Providing a “realistic” yet unpredictable AI is the goal of many solo game systems such as Bots or the solo assistant in games like Solo Generator for Conflict of Heroes: Awakening the Bear (Academy Games, 2018) or the Solo Assistant for Nations at War/World at War 85 (Lock ’n Load Publishing, 2019).

“Wargames should have concluding discussions with participants that identify reasons for decisions made during the game and lessons learned during the game.” This principle is required to continue the DoD Wargaming Process past play which I already pointed out is a distinguishing feature between the DoD and hobby wargaming process. The playtest and development process of hobby wargaming should still take note as many designers and developers have noted, even if only anecdotally, that keeping good notes and organizing data collection during a playtest series or development is very helpful in advancing the game design. As hobby wargame players, I think many enjoy reading or listening to designers as they discuss their design process, like Sebastian Bae recently did for his freshman publication Littoral Commander at the SDHistCon Conflicts of Interest site.


Wargame documentation and analysis

“Wargame design, conduct, findings, and conclusions should be documented in a post-game report.” In the hobby wargame world, this is known as a “review” or “after-action report.” Indeed, there is an entire industry of content creators that has grown up around this concept.
Oh yeah, let’s NOT talk about errata!

“Conclusions from wargames should be used for analysis but should not be the sole source for decisions; rather, they should be integrated with other analytic methods to help inform decisions.” Many hobby wargamers, myself included, are fond of saying we play wargames to learn about history. The ability for a game to teach history in an approachable manner that may be suitable for use in an educational environment is seen in wargames like Shores of Tripoli (Fort Circle Games, 2020). The use of games for education is one of the goals of The Dietz Foundation that has “the dream of making a difference in American society by helping teachers learn alternative means of education in the classroom.” As much as wargamers would love to think that games can help educate, the principle identified in the GAO wargaming report also should remind those same hobby wargame education advocates that wargames are but one tool in their kit that includes other media forms such as books or even movies.


Put it on the calendar

Consolidated Calendar Splash
Click image to see calendar

The GAO wargaming report has ten recommendations to improve the DoD Wargaming Process. A few could be applied to hobby wargaming too.

“The Secretary of Defense should identify a lead organization to create and maintain a common operational picture or master calendar for wargames. (Recommendation 4) / The Secretary of Defense should issue guidance requiring DOD organizations to share information about their planned wargames with the lead organization designated to maintain a common operational picture or master calendar for wargames. (Recommendation 5).” Regimental Commander Brant right here at the Armchair Dragoons hosts a wargame calendar that tries to list the many, many wargame events across the U.S. Though Brant may see himself as a “Secretary of Defense” for hobby wargaming the reality is the calendar is only as good as those that participate by sharing data since Brant has no power to compel compliance1.


ForgingWargamers 1
Courtesy usmcu.edu

“The Secretary of the Army/Navy/Air Force/Commandant of the Marine Corps/Chief of Space Operations should evaluate the costs and benefits of developing standard wargaming education and qualifications for wargaming personnel, including red cell players, and implement any findings from the evaluation. (Recommendations 6-10).” It appears that wargame practitioners have a similar problem to hobby wargame designers; there is no “career path” that imparts the education and experience necessary to “make a wargame.” For a robust discussion of this issue with regard to wargame practitioners, I invite everyone to read Forging Wargamers: A Framework for Professional Military Education edited by…Sebastian Bae (again!) and available for FREE (download or dead-tree versions) from Marine Corps University Press.



The GAO wargaming report focuses exclusively on the Department of Defense. Actually, as Brant pointed out to me, the GAO wargaming report actually focuses exclusively on analytic wargaming, which is but one of four types of wargaming. Colonel Jeff Appleget, U.S. Army (Retired) with co-authors Col. Robert Burks, USA (Ret.) and Fred Cameron in The Craft of Wargaming: A Detailed Planning Guide for Defense Planners and Analysts (Annapolis: Naval Institute Press, 2020) “identify three purposes of wargames: educational, experiential, and analytical” (Appleget et al., p. 5). They acknowledge a fourth purpose, entertainment, which they note “dwarf all other types, in terms of dollars spent on them” (Appleget et al., p. 5). Here I compared the GAO standards for analytic wargaming to hobby wargaming (i.e. “entertainment wargaming”). Like Brant, I also wonder what one might discover if the GAO wargaming report was evaluated against educational or experiential wargaming. Subject for a later post?

There are, of course, other wargaming efforts in parts of the U.S. government beyond the DoD. From what is available to the public those other agencies appear to be no better off than DoD. For instance, in the November 2022 issue of The Foreign Service Journal, former Department of State wargame practitioner Robert Domaingue argued for an “Office of Diplomatic Gaming” to be stood up:

“The Department of Defense (DoD) and the intelligence community (IC) routinely use games to examine assumptions, test concepts, and explore alternative courses of action. The Department of State, however, is lagging far behind in the use of policy gaming, and this hinders the department’s ability to proactively engage on issues rather than reacting to them as they occur.” (“Why the State Department Needs an Office of Diplomatic Gaming,” The Foreign Service Journal, November 2022).

The GAO wargaming report should be studied by other government agencies as well as international partners and allies interested in the art of wargame practitioners. Alas, it will also very likely be studied by adversaries who can use it to enhance their own wargaming efforts. Yet, as grand as all that sounds, the principles laid out in the report are relevant even to those in the non-governmental hobby wargame space. Taking the GAO concluding paragraph as a start and making a few tweeks to it the result is (unsurprisingly?) highly relevant to hobby wargame designers, publishers, and players.

DOD organizations The wargame hobby should use multiple frameworks of leading practices to guide the design, execution development, and analysis production of their wargames. However, DOD hobby wargaming could be enhanced through several actions. First, DOD hobby wargaming could better benefit from wargames and the insights they offer by developing an approach for effectively managing wargame data. Second, DOD hobby wargaming could take steps to improve collaboration by designating a lead organization to create a master wargaming calendar and guidance requiring DOD organizations encouraging wargame publishers to share information about their upcoming wargames. This would foster more enterprise-level hobby-wide knowledge, awareness, and expert player participation. Lastly, the services wargame hobby could evaluate the costs and benefits of developing standard education or qualifications for their wargaming personnel wargame designers, including those who serve in the important role of representing adversaries developing wargames. By taking these actions, DOD hobby wargaming could better ensure the quality of its wargaming efforts as it prepares for the challenges of countering near-peer adversaries boardgame disrupters who desire to alter the hobby wargame space to fit their own political views.”

Courtesy boardgamegeek.com


Appendix thoughts

Table 3 in “Appendix III: Wargame Quality Frameworks Identified by Department of Defense Officials” is a very good bibliography that serves not only wargame practitioners, but also offers useful knowledge for hobby wargame designers, developers, publishers, and even players. Almost all are available for free if one scours the internet enough.

Screenshot 2023 04 28 at 1.07.37 PM

(Feature image from the Cold War hacking “documentary” Wargames, 1983. A little known piece of trivia regarding the movie is that it inspired then-President Ronald Reagan to conduct a major review of digital vulnerabilities within the U.S. nuclear command, control and communications (NC3) infrastructure and ultimately led to the first major warning about the impact of emerging cyber technologies on nuclear stability. The warning that started with a high school hacker and a WOPR is National Security Decision Directive Number 145, “National Policy on Telecommunications and Automated Information Systems Security,” September 17, 1984.)


Thank you for visiting The Armchair Dragoons and saddling up with the Regiment of Strategy Gaming.
You can find our regiment’s social media on MastodonFacebook, Twitter, and YouTube.  (We have an Instagram page and we never use it.)
You can support The Armchair Dragoons through our Patreon, also, and find us at a variety of conventions and other events.
Feel free to talk back to us either in our discussion forum, or in the comments below.



  1. ed note: Gawd, I wish!

Tell us what you think!

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: