BWAPI Beta 3.4 has been released

BWAPI Beta 3.4 has been released. This release adds a new mode to the Local PC network module that uses pipes to enable communication between multiple instances of Starcraft: Broodwar, adds a number of functions to Game, Player, Unit, and UnitCommand, adds macro unit types, and also fixes a number of bugs. Here is the full list of changes.

BWAPI Beta 3.3 has been released

BWAPI Beta 3.3 has been released. This release adds a Local PC network module which enables games between multiple instances of Starcraft: Broodwar, native Windowed-mode, support for Heroes, Beacons, and Standard Powerups, new functions for upgraded unit type stats, and also fixes several bugs. Here is the full list of changes.

BWAPI Beta 3.2 has been released

BWAPI Beta 3.2 has been released. This release adds several menu automation options to bwapi.ini, adds the ability to disable the Starcraft GUI/graphics system for increased speed, and also fixes several bugs. Here is the full list of changes.

BWAPI Beta 3.1 has been released

BWAPI Beta 3.1 has been released. This release fixes a number of bugs, adds the ability to disable latency compensation, adds the ability to associate arbitrary information with each unit, and more. Here is the full list of changes.

Testing Framework is Complete

The testing framework now tests map information functions and as well as events/callbacks. These were the last two groups of test cases that I planned to add to the initial testing framework, bringing it to completion. That said, most functions in BWAPI use information obtained from Starcraft's memory, and like any RTS game, Starcraft has a very, very large state space, so of course the testing framework is no where near exhaustive. However, for now its safe to say that the framework tests unit commands, unit, player, and map information functions, and verifies that events/callbacks are generated when and only when they should be. However, if any bugs pop up in the future, I can add test cases to the framework to ensure that the bugs stay fixed.

I've also run the testing framework on the BWAPI client and it runs without error, so it looks like the client is nearing completion as well.

BWAPI Beta 3.0 has been released

BWAPI Beta 3.0 has been released. This release fixes a number of bugs, improves the performance and functionality of the client-server architecture, adds several new functions, merges several Orders together, and more. Here is the full list of changes.

Unit Command Test Cases Complete

After pulling an all-nighter, I'm happy to say that I've completed test cases for every unit command and they all pass on the server and client without a single failed assert. Perhaps the hardest test cases of the 43 unit commands were cancelTrain(slot), cancelMorph, and cancelConstruction, as different attributes of the unit had to be latency-compensated for different lengths of time, making them rather tricky puzzle pieces.

An off-by-one mistake in the latency compensation code would over-compensate or under-compensate the unit or player attributes (i.e. giving the player the appearance of having more or less resources than he actually has), leading to a failed assert that would only pop up for one or two frames at a time and then vanish in a game that's being run at 180 frames per second. Nonetheless, I have determined the shape of these puzzle pieces and now the latency compensation code is a perfect match for even these three unit commands, allowing for a seamless transition from latency-compensated values to actual values read from Starcraft memory.

As explained in my last post, my next main goal is to make test cases for events and callbacks, and more importantly start making the internal adjustments in GameImpl.cpp that will be necessary to make those new test cases run successfully on the client.

Test Framework Update

As planned, I've added test cases for the following unit commands:

• siege
• unsiege
• cloak
• decloak
• burrow
• unburrow
• lift
• land
• load
• unload
• unloadAll
• unloadAll(Position)

I've also added test cases for the following unit commands:

• setRallyPosition
• setRallyUnit
• rightClick(Position)
• patrol
• holdPosition
• stop

All test cases run without error as a DLL in Broodwar memory and as a separate client process. Now only the following unit commands are left to be tested:

• attackMove
• attackUnit
  
• rightClick(Unit)
• follow
• repair
• returnCargo
• cancelConstruction
• haltConstruction
• cancelTrain
• cancelTrain(Slot)
• cancelAddon

Once I add test cases for these last 11 functions, I plan to add a test map that specifically deals with unit destruction and AI module call backs. For example, when a unit dies it should generate an onUnitDestroy callback. I also plan to re-write the way BWAPI generates events and callbacks to facilitate the process of making them behave the same way on the client as on the server. To do this, I'll make BWAPI just generate Events in the appropriate places, rather than both Events and perform the callbacks immediately. Then, right before AIModule::onFrame() is called I'll translate the events into callbacks and perform them. This will make the AIModule callback behavior virtually identical to the Client Events since they'll both be called at the same point of execution in GameImpl::update() for the most part. I'll also need to make some modifications to the way units get destroyed or become invisible so that their last frame of valid unit information is still available at the time of the callback or Event. For example, right now when an enemy unit dies, the client doesn't have access to the last frame of valid unit information because it is erased in GameImpl::onUnitDestroy(). I plan to make this function just set a flag in the UnitImpl class to indicate that the unit is dying, and only add UnitDestroy events. The UnitData struct for the recently destroyed unit would then be cleared after the AIModule::onFrame() and Client update functions have been called.

Testing Framework Runs Successfully on the Client!

Since my last blog post almost a week ago I've re-written the latency code so that it is all nicely written in a single template class which is then used both in the BWAPI dll (server) and on the client-side of the client-server connection. I've also recently fixed a few other bugs in the client.

As a result, I have now successfully ran the testing framework through all the test cases on all three major test maps (TerranTest, ProtossTest, and ZergTest) without error. This is a major milestone toward completing the client-server architecture.

My long-term goals now are to make the testing framework comprehensive as well as make sure all test cases continue to succeed both server-side as a normal AI module DLL and client-side. My next immediate goal will be to add test cases for siege/unsiege, cloak/decloak, burrow/unburrow, lift/land, and load/unload/unloadAll/unloadAllPosition.

Client-Server Optimization Progress

The server code is now optimized for getting unit and map information into shared memory for the client. To get a ballpark estimate of how much of a speed improvement was attained, I ran both the most recent version revision of BWAPI and the most recent release of BWAPI - 2.8 - on the same replay and measured the frames per second each version could achieve when playing with /speed 0 and the replay mode on Fastest x16. Before optimization, the client and server could run at 53 FPS. After optimization, the speed increased to 87 FPS. While these numbers are specific to my computer, other computers will likely get a similar 50-70% speed increase.

Also, the test framework now tests every special ability, using related unit information commands like isStimmed, isStasised, and isLockedDown to verify that the abilities were executed successfully.

Client-Server Optimization

I've decided to take a break from the testing framework to work on the client server architecture.

The client-server architecture is a new alternative way of making AIs with BWAPI. Traditionally, people would write their AI in a DLL which would get loaded by BWAPI at the start of a match. In contrast, the client-server architecture lets users write their AI as a separate executable program which connects to their local BWAPI server (loaded in Broodwar memory), and communicates with it via shared memory.

This new architecture is almost fully functional, however up to this point it has been extremely slow due to the method that BWAPI uses to get information from Broodwar into the shared memory bank. The inefficient/simplistic implementation currently just calls relevant BWAPI::Unit member functions, and copies the results into shared memory. Since each unit's state is comprised of about 100 different pieces of information, this results in about 100 BWAPI function calls per unit. And each of these functions call other functions in order to generate the appropriate error codes when needed and determine if the user/AI is allowed to access the given unit (hidden units are not accessible unless Flag::CompleteMapInformation is enabled).

To optimize the process of getting information into shared memory, BWAPI will compute the UnitData struct directly from Broodwar memory, bypassing the BWAPI function calls. Thus the last few days I've been moving implementation details of each individual Unit member function to a single function, called Unit::update(), so that it computes the entire UnitData struct directly from Broodwar memory without using any unnecessary or redundant function calls. In addition, to prevent duplicate code I've been simplifying the Unit member functions so they just compute their answers from the UnitData struct, just like the client-side implementation of the Unit object does. This should also go a long way to ensuring that BWAPI behaves consistently whether you're writing you AI as a DLL or as a separate client process.

In short, I'm making a rather large internal change to how the BWAPI::Unit class works in order to optimize the client-server architecture, however the end-user functionality should go unchanged. Also, the test framework module is a DLL, so it can be run both as a DLL in Starcraft memory the traditional way or be loaded into a separate program such as BWAPI's AIModuleLoader and run in a separate client process, so the test framework will be able to test both implementations to make sure they behave correctly and indistinguishably once everything is complete.

Test Framework Update

As planned, I've created test cases that research each TechType and fully upgrade each UpgradeType. Like before, adding these test cases has revealed several bugs which I have since fixed:

• Fixed a bug where Unit::getRemainingResearchTime would sometimes return 0 due to latency.
• Fixed a bug where Unit::getRemainingUpgradeTime would sometimes return 0 due to latency.
• Fixed a bug where Game::canResearch would return true while another unit was researching the same TechType.
• Fixed a bug where Game::canUpgrade would return true while another unit was upgrading the same UpgradeType.
• Fixed a bug with UpgradeType::Apial_Sensors.whatResearches() so that it now correctly returns UnitTypes::Protoss_Fleet_Beacon.

I've also added the following two new error codes:

• Error::Currently_Researching - generated when you try to tell a unit to research a TechType that is already being researched (at that unit or another unit).
• Error::Currently_Upgrading - generated when you try to tell a unit to upgrade an UpgradeType that is already being upgraded (at that unit or another unit).

Since each special ability affects the game state in a unique way, creating UseTech test cases for each special ability may take a few days. For example, to check that the Stasis ability works, I'll need to check the target unit to verify that it is Stasised, while to check that Dark Swarm works correctly I'll need to verify that a Dark Swarm unit is produced at the target position.

Earlier today I fixed the following bugs related to Protoss Interceptors:

• Fixed Unit::getTransport so that it works for Protoss Interceptors
• Fixed Unit::isLoaded so that it returns true if and only if Unit::getTransport!=NULL.
• Fixed Unit::getLoadedUnits so that it also returns the Interceptors currently loaded inside a Protoss Carrier.
• Fixed a bug where Unit::getPosition and Unit::getTilePosition would return incorrect positions for loaded units.
  

I also added the following functions to the interface:

• Unit::getCarrier - return the Carrier that created this Protoss Interceptor
  
• Unit::getInterceptors - returns the Interceptors controlled by this Protoss Carrier.
  
• Unit::getNydusExit - should return the connect Nydus Canal Exit (untested)
  

The one remaining bug with Protoss Interceptors is that the interceptors cannot be detected until they leave the carrier they are loaded in. It should be possible to fix once I change unit detection so that it traverses the other linked lists of Units, not just the main one. Also once this is implemented it should be easy to fix bugs related to unit deaths/removals.

Test Framework Progress

I've been working on the test framework over the last few days and I've finished step two of the plan I outlined in my last post. The TestAIModule now produces every possible building and unit which doesn't require an upgrade or special ability for each of the 3 test maps (TerranTest.scm, ProtossTest.scm, and ZergTest.scm). While these units and buildings are created, lots of asserts are performed to make sure relevant functions return the correct information. By checking the results on every frame, I've been able to find and fix bugs that would only appear for a frame or two, and then disappear. While this test framework is still far from complete, creating these test cases has allowed me to discover and fix the following 13 bugs:

1. Fixed a bug where Unit::getType would not immediately switch to Egg/Lurker Egg/Cocoon after issuing a morph command to a unit
2. Fixed a bug where Unit::getBuildType would return UnitTypes::None for Terran buildings that were constructing add-ons
3. Fixed a bug where Unit::getBuildType would sometimes return UnitTypes::None for morphing Zerg units
4. Fixed a bug where Unit::getBuildUnit would return NULL for incomplete Terran add-ons that are being constructed.
5. Fixed a bug where Unit::getRemainingBuildTime would sometimes return incorrect values for morphing Zerg units
6. Fixed a bug where Unit::getRemainingTrainTime would incorrectly return 0 on some frames due to latency.
7. Fixed a bug where Unit::isBeingConstructed would return false for morphing Zerg units
8. Fixed a bug where Unit::isConstructing would return false for Terran buildings that are constructing add-ons
9. Fixed a bug where Unit::isConstructing would return false for incomplete Terran add-ons that are being constructed.
10. Fixed a bug where Unit::isConstructing would return false morphing Zerg units
11. Fixed a bug where Unit::isIdle would sometimes return true when the unit is constructing
12. Fixed a bug where Unit::isIdle would return true morphing Zerg units
13. Fixed a bug where Unit::isTraining would not return true for Reavers or Carriers.

While writing these test cases was a pain, I'm starting to see how useful and important a proper test framework is for finding and fixing bugs :). My next goal will be to create test cases that research each upgrade and special ability. After that I will create test cases for using each special ability.

BWAPI Test Framework Plan

This week I'm at a family reunion at the Mahoney State Park in Nebraska. I brought my laptop but the internet connection is really flaky, so I can't really work on BWAPI this week. However I'll be coming back in two days and I've made a plan for how to create the BWAPI test framework.

First I will create test maps for Zerg and Protoss and simplify the test map for Terran. Once a test map for each race is completed, I'll then write test cases that proceed in the following chronological order from match start:

1. Test initial game/force/player/unit information - make sure they all return the correct initial values for the given test map.
2. Test unit production, construction, and (for Zerg) morphing by creating every possible unit for the given race. This will also include asserts to make sure functions like Unit::getBuildType and Unit::getTrainingQueue return the correct information while the units are produced.
   
3. Re-test the game/player/unit information - make sure they all return the correct updated values for the given test map now that step 2 has completed. This will also verify that step 2 completed successfully.
   
4. Test the Unit::upgrade and Unit::research functions by researching each upgrade and tech, making sure the relevant functions in the Player class return the correct information at each step along the way.
5. Test the Unit::useTech functions for each TechType (special ability). This will verify that BWAPI issues each special ability correctly for each spell-casting unit, and also test to make sure the correct error codes are generated in cases where the special abilities cannot be executed.
   
6. Test the other unit commands, making sure the the unit information functions return the correct values before, during, and after each command. Each command will be tested in isolation and return the unit to a non-cloaked, non-burrowed, non-sieged, non-lifted idle state.
   

BWAPI Beta 2.8 has been released

BWAPI Beta 2.8 has been released. This release adds menu automation, support for bullets, and fixes several bugs. The full list of changes are in the change log.

Progress toward bullet support

Over the last couple days I've added the following functions to the Bullet interface, which is now documented on the wiki:

• getID
• getPlayer
• getAngle
• getVelocityX
• getVelocityY
• getTarget
• getTargetPosition

I've implemented all of these functions except for getID using information found in Starcraft's memory, specifically in Starcraft's bullet array, which we define in the BW::Bullet struct. This struct has been updated many times over the last several days and contains everything we currently know about the 112 bytes that each bullet occupies in Starcraft memory.

The process of adding new functions to the BulletImpl class, such as getTargetPosition(), usually consists of searching through the bytes in this struct and looking for bytes that appear to contain information that define the bullet's target position, or where on the map it is heading. Once we're sure the bytes represent what we think it does, we can rename the field for those bytes from some generic name, like unknown_0x18, to something more descriptive, like targetPosition, and use that field in the implementation of the getTargetPosition() function.

The getID() function returns a unique ID for each bullet and is updated whenever a bullet switches from non-existent to existent. Since bullets are created and destroyed much more frequently than units, unlike BWAPI Units, new BWAPI Bullets are not created when a Bullet is destroyed, which means bullets do not have a unique address throughout the course of a match. Only 100 BulletImpl objects are created (1 for each element in Starcraft's bullet array), and the IDs of these are updated when the slots are re-used for new bullets.

Bullet::isVisible() is now implemented correctly and does not depend on the visibility of the source unit.

I've also added a Game::getBullets() function which returns the set of accessible bullets (all existing bullets if Flag::CompleteMapInformation is enabled, otherwise just visible bullets).

Bullet support has also been added to the client and server code, however it has not been tested yet.

RCOS Presentation

Today I gave a presentation about BWAPI at the Rensselaer Center for Open Source Software (RCOS). In the presentation I gave an overview of BWAPI, mentioned two other projects I've been working on (BWTA and BWSAL), outlined my plan for what I hope to accomplish this summer with the BWAPI project, described the Starcraft AI Competition that the Expressive Intelligence Studio at UC Santa Cruz is hosting at AIIDE 2010, and concluded with a video of the EISBot, which really shows what people have been able to accomplish using BWAPI.

The slides are available here.

My last few commits to the BWAPI svn have been working toward the goal of adding Bullet support to BWAPI. So far the exact size and location of the Bullet table have been determined, and the following functions have been added to a new BWAPI::Bullet interface and have been implemented in the new BWAPI::BulletImpl class:

• getType()
• getOwner()
• getPosition()
• getRemoveTimer()
• exists()
• isVisible()

isVisible() is currently implemented in a simple/naive way that should work correctly in most cases, but will need to be rewritten at some point to work with cloaked/burrowed units. The other functions should be implemented correctly but have not been tested very much yet.

Next I'm going to try to figure out how to read the Bullet's target, direction/heading, and velocity, and also start a wiki page that documents the Bullet interface.

Menu automation completed!

Well, not entirely. But we've completed the original goal of automating menus for single player matches, replays, and UDP LAN games as of r2310. You can now specify the game type, and for single player games you can specify the number of computer opponents and their race in the bwapi.ini file. For more information, see the new wiki page on menu automation. Also some non-critical actions can still be automated if needed. For example, I didn't bother to automate game speed selection since you can control it directly with Game::setLocalSpeed. If you need some part of the menu system automated for your AI, create a new issue on the BWAPI Issues List for it and I'll get to it eventually.

Since completion, several users have expressed interest in the automation of Battlenet menus, which sounds useful since it would enable users to create a bot that listens in an IRC channel or Battlenet channel and automatically joins games when requested. I've created a new issue for this though I'll probably focus on some other aspect of BWAPI for at least the next few days.

More menu automation

2 updates:

1. Menu automation for replays is complete. Bots can now iterate through a list of replays with no manual user input. To do this, in the bwapi.ini file set the map parameter to the first replay in your list, set auto_menu = SINGLE_PLAYER, and add some code to your bot's onEnd() function which updates the map variable to the next replay in your list and also calls Broodwar->leaveGame() to avoid the Exit Replay dialog box. This will be especially useful for participants in the AIIDE 2010 Starcraft Competition who are data mining thousands of replays and applying machine learning algorithms to determine the best strategies.
2. Race selection for single and multiplayer games can now be automated - just set the race parameter in the bwapi.ini file to the race you want to play as.

Next steps for menu automation: Automate game type selection (Melee/UMS/Free For All/etc), game speed, number of computer players and their races for single player and UDP LAN games.

Progress toward menu automation

After a few hours of searching through memory and playing around with different strings that look related to the map name, I found two offsets in memory that control which map Starcraft loads and plays on (at least in single player games). As of the latest revision (r2262) you can now edit the bwapi.ini config file and set 'map' to the name of the map you want to play on and set menu = 1, then you just need to click 'Start' on the Chaoslauncher to start a single player game on that map. So far it looks like it works every time, though there is room for improvement.

In order to select a particular map, BWAPI right now has to enter the Create Game screen, then cancel out, and then re-enter the Create Game screen. While this repetition is automated, it does add a second or two to the setup time which could be avoided if I can find a way to make it work correctly the first time BWAPI enter's the Create Game screen.

Also it only works for single player games, not replays or multiplayer games, so I will keep looking to find a way to automate map/replay file selection for those game types.

BWAPI Beta 2.7.2

The latest version of BWAPI is BWAPI Beta 2.7.2. This release just fixes a crash that was accidentally added in BWAPI Beta 2.7.1. BWAPI Beta 2.7.1 was the first release in several months and has quite a few changes and bug fixes, which are detailed in the change log.

Yesterday I started adding some test cases to the TestAIModule (which is in the BWAPI svn). So far it mostly tests basic things, though already I discovered some unexpected output - the player's minerals and cumulativeMinerals are 0 during onStart, and Game::isPaused is true during onStart. I think the easiest way to fix these problems would just be to ignore the first frame and start the ai module on frame one (though adjust the numbering so the AI module still thinks it is starting on frame 0), since it appears the values of these functions are correct on the second frame.

BWAPI Blog

The Brood War Application Programming Interface (BWAPI) is a free and open source C++ framework for creating AI modules for Starcraft: Broodwar. Using BWAPI, programmers can retrieve information on players and individual units in Starcraft as well as issue a wide variety of commands to units, opening the doors for custom AIs with new micro and macro algorithms.

With BWAPI you can:

• Write competitive AIs for Starcraft: Broodwar by controlling individual units.
• Read all relevant aspects of the game state.
• Analyze replays frame-by-frame, and extract trends, build orders, and common strategies.
• Get comprehensive information on the unit types, upgrades, technologies, weapons, and more.
• Study and research real-time AI algorithms in a robust commercial RTS environment.

Links:

• The BWAPI web site
• BWAPI Documentation
• BWAPI Google Group
• BWAPI Forum
• IRC Channel: QuakeNet #BWAPI