Technical musings on AutoIt MMO botting (OCR, etc)

Post Reply New Topic

So I am no coding guru, but I have had some interesting thoughts on how to implement some features commonly needed by MMO bots written in AutoIt that I would like to share here.

To all who are eager to flame me for writing bots, consider that video games are made to be enjoyed, and I do this for fun. I make scripts that allow me to bypass aspects of a game that I do not enjoy, have already completed countless times before, or just plain want to (such as when I bot the epsxe emulator). I do not run these scripts to excess, nor do I share them with others.

Okay so down to the meat of the post:

Assume you want to know item is in an inventory slot. This is a simple and common undertaking. Further assume that memory reading is momentarily off the table. In a game like WoW that either has solid inventory backgrounds (zero alpha) or is natively modifiable (in WoW's case, both) it might be relatively simple to do a pixelchecksum of an area of the inventory sufficiently large to be universally salient, but let us assume that the game uses partially transparent inventory space, and is not modifiable for this purpose.

For example, I direct you to examine Guild Wars' interface (a dead game that I've played to death, and my primary focus while I came up with this post late last night):



Note that the above is a random screen shot I found on Google and is in no way related to me.

Be aware that while the inventory (left hand side) is obviously partially transparent, the item tool tip is actually semi transparent as well (just much darker). The window with the paper doll is irrelevant to this post.

This is what I suggest should be done for an autoit bot to 'visually' identify inventory items:

Create a function that is capable of aligning your in game windows to a specific place (Ive seen implementations that simply reset everything to default, then drag and drop into place). Use this for reproducible consistency in the GUI.

Create a function that is capable of generating an array that stores pixel locations / color values... like so: array[a][ b] , where [a] is an incremented key, [a][0] is the x location of the a pixel, [a][1] is the y location of the a pixel, and [a][2] is the color of the pixel at [a][0],[a][1]. Note that when I say location, I do not mean absolute location, but some relatively location offset within the game's window.

Create a function that compares two such arrays by returning a third array that is composed of either everything that is similar, or everything that is different. ie, if array1[12][ b] == array2[47][ b] for all b within its range, then array3 will have some [a][ b] that is equal to array1[12][ b] (for b=0,b=1, & b=2).

During the bots creation provide it with a repository of such pixel value arrays, stored in text files and generated by hand. Generating them by hand is readily made trivial, as all you have to do is create a pair of hot keys that call certain functions. The first hot key would creates a baseline of an empty inventory slot (against a stationary background!!) and the second would capture what has changed within its scope after an item has been added to the inventory slot, and ask for the name of this item's visual signature.

Now you have the bare minimum necessary to do OCR, but there is a hitch: speed. It is needlessly wasteful to brute force looking for these, so I have thought up a better method: reducing the workload while the bot is loading by pre-optimization of signatures.

Take each item's visual signature and compare it to every other signature. Mark every location within its visual signature that has at least one other signature with the same color at the same location as a candidate for deletion. If all locations within a specific signature are so marked, then ensure that signature1 <> signature#, and that signature1 can not be entirely defined as a subset of signature# (if so, then you will later have to provide that signature1 must NOT equal signature2 at at least one location, or sort the later mentioned VLA of signatures by descending order of the number of pixel locations. This unlikely problem is not addressed further.).

Then prune all but at least one (wiser to choose several) randomly selected locations from those who that were marked for deletion. Sort all locations that were flagged for deletion to the bottom of the signature, so that ones salient to the item are kept at top. Keep the original signature in an unused section of the signature's file in case new items are found with troublesomely similar signatures (it wont be addressed further, as it is merely kept available as a contingency). Each signature will now be salient from all other signatures at all locations.

In theory, you could iterate through a VLA of these visual signatures, checking as little as one location per signature for a specific color value to determine what the item is.

Provisions must be made for unrecognized item signatures - you must be able to know if a specific slot is empty or not. This is a simple enough undertaking. (also, in guild wars you would have to restrict the range of pixels from which the visual signature is generated from so that it does not include the numerical display of stack sizes)

There is however the slightly more complex matter that not all items which share the same visual signature are the same. In Guild Wars you can not only have different types of items that look the same, but the same item type may be of multiple rarity levels (denoted by tool tip text color), and may possess prefixes/suffixes.

It occurred to me that you can read text in a similar way, by checking specific relative offsets for pictures salient to each character. The problem with this is that for optimizations sake each character's signature would have to also possess information about how much the next character would be offset (lower case l would generally have a low offset, while upper case W would have a larger offset). It could well be necessary to ensure that certain locations do NOT match any expected font color (in the example Guild Wars tool tip this could be done by a test of darkness - while text is always some shade of white blue purple gold or green, the background is always clearly darker then the text), such as to tell the difference between lower case i and lower case l.

It would be ideal to teach the bot to learn to read only enough characters so as to discern what is necessary - if the first two characters read are "Ic", and the item's visual signature is that of a weapon, then you know the item is a "Icy" weapon and you can begin searching at some offset value specific to "Icy" to read "Dra", so you know that it is a "Dragon Sword", and so forth. Because we can control and/or predict where text will appear, what color it will be, what its font will be, etc we are able to envision methods of drastically simplifying the task of programmatically reading 'visual' input.

All the above could well be used to create systems that only sells junk to merchants while sending rare finds to alts, automatically cataloging your inventory over multiple characters / accounts, spamming to sell items in game, etc.

Another task that Ive been interested in automating is the vanquishing of areas in Guild Wars. The approach I have considered can be simplified as such:

Have your character decked out for support - only use abilities that target allies, and allow them to do the actual killing. Use TexMod to remove the moving, cloudy effect on the area map (bound to U key by default). Set up a system where the bot does a pixelchecksum of a small rectangle of the area map at a certain location (doesnt really matter where exactly as long as its consistent - not too close to the center, and not too close to the edge, though). Movement is achieved by flagging the heroes/henchmen one pixel north west east or south of where you currently are and then following them. Background pixelchecksum can tell you when you stop moving, reading health bars over time can tell you if you are in combat or not. When the background pixelchecksum holds steady after moving you have arrived, and you can check the area map pixelchecksum. If it is not identical, then run a pixelchecksum on all adjacent pixels from the area maps reference point, if not found, repeat the process outwards a few times and ultimately toss an error. If found, get a new area map pixelchecksum for a 0 x,y offset, mark the array visited[x][y]=1 (this array is initialized to 0 and x,y is the cumulative offset from all previous area map pixelchecksums) and continue by using a nearest-first, space-filling heuristic to visit every value of visited[x][y] that is 0 within certain bounds of x and y (this heuristic must of course prefer to visit least attempted x,y locations, to prevent it from getting stuck at impossible to reach locations). Refrain from attempting to visit areas within certain bounds, to prevent the bot from walking into a portal.

The above paragraph's explanation is a bit un-clear, but I'm pretty sure that I've thought of everything necessary for an inefficient programmatic area-vanquishing bot. I could provide a better presentation of it should I ever decide to program when its not so late at night that its starting to get light out.

So now that I've talked that out of my system, anyone have any input?

Well I'm sure it was fun to make and it might improve your game enjoyment. But... that being said if I knew you were using it in game on the same server as me I would report you to the GMs.

Regardless of what's enjoyable and what's not, MMOs are generally based on getting rewards as a result of putting hours into it. That's why bots are illegal... so that everyone is on a level playing field.

Seems to me though that what you're describing can be accomplished with legal addons in the games I play, namely WAR and WoW. I have seen multiple "sell useless junk automatically" type addons that won't get your account banned. In fact, I use an addon in WoW like you've described to catalog items over all characters. I believe it's called Bagnon Forever or something like that.

Perhaps your efforts are better put forth into making addons.

When I played WoW I used those same addons and wrote my own. I no longer play WoW or GW, and GW has no addon API and falls flat on its face in the customization department. Thankfully the third party software TexMod allows some minor alterations be made with the consent of ArenaNet. (it interestingly has large pvp applications - something that what I detailed does not)

Like I said, GW is a dead game, but I failed to mention how it has a ton of problems: for example, when I preordered the original game it was because it allowed you to instantly make max level chars, there was no functional difference between super-expensive and dirt-cheap weapons (just aesthetics), and that I was promised that it was a grind free game where everyone was on an even footing. Now Guild Wars has the worst reputation grind of any game I know of (two factions that both require >14 days of continuous botting at super-human efficiency to max out). Worse yet standing with those factions increases powerful skills that allow you to do things in game that are else wise impossible...

So yeah, GW sold out. But that is neither here nor now.

I have played GW to death: I fully unlocked everything without spending a single point of Balthazar faction, had a level 17 presearing ranger (achieved by hand) and was even widely known in game for guides that I posted on forums (which many people assumed were fakes because of how exceedingly difficult some of them were). I have a max level character of every class, formed and ran a 6 guild alliance (we had the world-first clear of Urgoz's Warren - and it was with a balanced build, too), acquired >1,000 plat through legitimate means, and have beaten every instance on normal and hard mode (including two man dual boxing Urgoz's Warren to completion).

There really is not anything left for me to do in game, so I play around with what I am not supposed to. People who take a holier then thou approach to all forms of botting or cheating are often unable to either appreciate the effort it takes to write a good bot (I always write my own, never use anyone else's, except their source for inspiration) or are unable to define how I am hurting them. After I had bought every skill in the game on my Monk (which requires tens of millions of EXP and costs over 1,000 plat) I used a bot to get other, freshly created characters to have enough EXP to buy a few dozen essential skills. How did I hurt anybody by doing that?

Did I hurt the economy? I'm going with no - there was a known exploit that went unpatched for months that allowed duping of gold, the pittance my bot made while I was at school was nothing comparatively. Did I create an unfair environment for other players? No - it was a roleplaying character in a game that allows you to create, at will, max level PvP characters. Everything in Guild Wars is instanced, I stole nobody's kills. The items I gained did not allow me to perform any better in combat - at worst, I got something that looked prettier then a collector's item that can be bought for <1 plat. If anything because the occasional useful item that I have no use for I sell for well below market price, I have allowed some people easy access to what they wanted (and possibly helped lower market price a few times).

My bots are always made to be undetectable by anything but the most exacting of scrutiny, not just from self preservation but because it is fun to do so. I reject the entire illusion that the achievements of my personal effort should be denied to me strictly because I did it in an unconventional manner. Yes, a hundred thousand bots running at once and selling the gold are going to cause some problems, but that is not the issue when I am being attacked for creatively circumventing the most boring and repetitive aspects of a game (aspects that would, if this were real-life, would have their functional equivalents performed by real-life robots) to go and do what ever it is I still find interesting or challenging enough to demand my attention.

Fun fact: did you know that GW will happily allow 3rd party programs to hook into its memory? Why would it do that if not to allow bots to interact on a visceral level? (which is something beyond the scope of what I proposed here - and a LOT more vulnerable to exploits, trust me)