Pos Exploit Devblog:
http://myeve.eve-online.com/devblog.asp?a=blog&bid=626 Link has the images my c&p doesnt.
"War makes thieves and peace hangs them." - George Herbert
EVE pilots go through a lot of hardship and mind-boggling planning to gain the upper hand in the great wars that are ongoing in EVE. Some go beyond the bounds of legitimate gameplay and, in the end, pay the ultimate EVE price - getting themselves banned from the game. Such was also the story with the latest POS bug, exploited by a relatively small number of players, but on an industrial scale.
The POS exploit has now been research thoroughly both from the technical perspective as well as with regard to the impact on the economy in EVE Online. We have left no stone unturned in order to be able to understand how this bug came into our code and why it was not spotted sooner, learning from this event in order to help prevent this from happening again in the future.
We also feel strongly that you, the starship pilots of EVE, should be able to understand this as well so we have decided to present, in detail, the information that we have on bugged reactors that has become more commonly known as "the POS exploit".
The preliminary findings of this report were introduced to the Council of Stellar Management around mid-January. The CSM members gave us very good feedback and helped us in fine-tuning our research and reporting on it.
As you'll see in the report that follows, work like this requires a collaborative effort from many different departments within CCP. From the software team, we have CCP Pleognost who tells us all about the initial code, how this bug was introduced and what we did to fix it. CCP Diagoras and CCP DrEyjoG researched economic data estimating the overall impact of the exploit on the EVE economy and CCP Grimmi explains the actions taken against those utilizing the exploit for their own benefit. Many more were involved at different stages of the research so we would also like to thank all of them for their involvement as well.
It is a lot of information to sift through but we hope that you will take the time to read it and share your thoughts with us in the corresponding discussion thread.
The POS: What it is supposed to do and what it did by CCP Pleognost
Understanding the POS exploit is not, from a technical standpoint, terribly difficult. The problem is that you need to know what the heck is going on underneath the hood.
In this case, the objects involved represent three things in the game: POS Control Towers, POS Silos and POS Reactor Arrays. For the purposes of this discussion, Coupling Arrays are really just tiny silos. Moon Mining Arrays are similarly extremely simple Reactors - they basically just spit out a steady stream of free stuff without consuming inputs.
POS Structures: What They Do
Behind the scenes, the arrangement of these objects is of the utmost importance.
POS Control Towers are the big bad boys of the POS system, composed of a half-kilometer of code that does everything from detecting when a POS should switch over to Reinforced Mode to sending annoying alliance mail to your enemies when it gets anchored in claimed space. The bit we're concerned about is the POS Production control system, itself a large chunk of the already-titanic Control Tower file.
In this role, it's useful to think of the Control Tower as the boss in a really tightly-controlled factory; it tells its workers when to work and is really particular about the order in which their jobs are done.
POS Silos are basically big cargo bays in space. They love holding stuff. It's really all they do. We have some restrictions slapped on top, though. A POS Silo is only allowed to hold one type of commodity, and even then it's only allowed to hold one stack of one commodity. It makes sure to cache-off the size of the stack it's holding, allowing us to manipulate that stack freely over the course of the production cycle without hitting the database.
This is good, because database lag sucks. Lag Sucks will be an important theme moving forward.
POS Reactors are complex beasts, but not quite so bad as POS Control Towers. They seem rather simple on the surface. In the naïve view, reactors accept loads of stuff from silos, convert it into more valuable types of stuff according to the reaction blueprint they contain, and then hope that stuff gets taken from them by someone else before they have to make more stuff.
In reality, we don't actually do all that item manipulation. Moving stuff around is expensive; it hits the inventory system - the system that tracks the locations of everything in EVE. Unduly using this system can sap precious CPU and database time from the rest of EVE. We need to avoid that, if at all possible.
How Stuff Works
Presume we have a simple POS with a simple reaction: Silo A contains Resource A; Silo B contains resource B. Reactor X eats up Resource A and produces Resource B.
Figure 1: A basic production POS and its flow-of-control, in abstract form.
Look at the line of execution at the top of this picture. You'll notice that the reactor spends most of its time empty; it's only got stuff inside it while it's waiting for Silo B to accept its output. We really don't need to maintain any resource stacks inside Reactor X; we just need to know if it's been supplied with enough of Resource A to run and produce a unit of Resource B. We can then just inform Silo B that it's free to add a unit of Resource B to itself. This, in fact, is what we do.
Two More Little Tricks
Before we get to the main issue, you need to know two more things. Both relate to POS Control Towers.
First, we run reactions in two big phases. The first phase is when we validate that the chain can operate in its current state, ensuring that reactors are properly supplied and can/should consume resources. During this phase, if everything goes well, reactors consume resources. They also set a little flag internally to remind themselves that they've actually produced during this production cycle.
The second phase is when we actually "produce" stuff, which is to say that silos go through and take outputs out of Moon Mining Arrays, other Silos and Reactors. At the end of this phase, we save off the state of the POS in case Bad Things, like downtime, happen before the next production cycle.
Now, since reactors are supposed to spend a cycle actually "producing" the stuff they make, they need to do one more thing. At the very start of the first phase, they save off the current value of their "production" flag, which then tells us if they were producing during the previous production cycle. If so, then silos can retrieve resources from them. If not, the silos need to wait until one cycle passes successfully, which takes one hour, provided the reactors are properly supplied.
Second, recall that POS Control Towers are basically the bosses of the whole operation. They tell the silos when to slurp up stuff from whatever is dumping stuff into them, and they tell reactors when to consume/produce stuff based on their reactions. If the Control Tower just told stuff to run in some random order, silos might suck up stuff out-of-order, or a reactor might try to start up before the moon miner it depends on. It would be chaos. Stuff would shut down randomly. We need to be really specific about the order we evaluate the POS in.
Warning: Math Ahead
If you're a real POS guru and you've sketched out POSes on paper, you might've realized that POSes can be represented by what mathematicians call a Directed Acyclic Graph. In our case, we have structures (Mathspeak: vertices) connected by links (Mathspeak: edges). The proper order in which to evaluate a POS is essentially breadth-first traversal, meaning we want to visit all structures that have nothing linking to them first, then move on to the things they link to and so on until we've followed all the links. This is analogous to breadth-first search, although not identical. We're not actually searching for anything and we could - in fact, we probably will - have multiple root vertices.
Figuring out where to start our traversal is at the crux of the POS issue. Actually finding all the places to evaluate first is relatively easy. We scan through the full list of a POS's production links and maintain a list, noting how many times each structure is linked to. Any structure that is linked to precisely zero times needs to be evaluated first, since these structures will be at the "head" of your POS's production chain.
Smarts R' Us
Now, there's a lot of starbases out there in EVE, and if we could shave a few CPU/database cycles off by simply not evaluating a few of their structures, so much the better. To do this, we looked at the type of objects we'd found at the head of a production chain.
For silos and Moon Mining Arrays, being at the head of a production chain is entirely logical. Both can supply stuff, either out of the ether (Moon Mining Arrays) or out of their stockpile (silos). We process them normally.
Reactors, though... Reactors always need inputs, right, guys? Right. Let's save cycles here and just not evaluate this reactor! I mean, it'll never get evaluated and thus never come online, right?
... Right?
Oops.
So, presume you've run a few cycles of your POS. Your reactor is humming along nicely. It has produced stuff this cycle. It has produced stuff last cycle. The Control Tower is running all of your stuff in the right order. Everything is fine. Until something unexpected happens.
The user cuts off all the links to the reactor.
The Control Tower, crazed by its optimization logic, careens through the production code. Wide-eyed, it reaches your reactor first. In its addled eyes, it sees only that the poor reactor has no links.
The Control Tower speaks.
"We can't stop here! This is bat country!"
Onward the Control Tower drives, speeding towards the silo at the far end of the reactor's link.
The reactor has not been evaluated. It does not know that another cycle has passed. It still remembers, fondly, grazing on inputs during its previous, un-bugged production cycle. Without this information, the silo goes ahead and adds another cycle's worth of goods to its stack.
Free stuff has entered the system.
The Aftermath
Once identified, the problem was relatively simple to fix: we merely evaluate reactors normally, even if they have no links.
All information on the production states and links of running POSes was available in the database. After developing a fix, we slapped together a query that ran through all the POS Reactors on TQ, checking which ones had links pointing to them, and checking their production states. Any online reactor that had no links pointing to it, and yet was still reporting itself as having produced output last turn, was compiled into a list.
Once that list had been reviewed, we examined all the structures that had been duping. Using some other data available in the database, we figured out what resources they had been producing, and when they had last entered their production state.
And that, as they say, was that. Now we needed to know how many ISKies this exploit had created, and how it had impacted the EVE market. On to the economics of the exploit!
Economic Impact by CCP Diagoras and CCPDr.EyjoG
From the technical side we know that it was possible to get free stuff. The economic impact of such an exploit is, as can be imagined, something that is of great concern to us. This section will focus on the economic impact of the exploit on the EVE economy.
The majority of our analysis has been focused towards the effect that this additional supply of high-end moon materials has had on the economy. Those using this exploit focused on the most profitable items though other materials (such as materials used in the production of boosters) were also produced as well. The impact on the overall economy from boosters and low-end moon material is of less concern to us as than the high-value materials used in Tech II production. Once the exploit was detected, the following reactors were found producing high-end moon materials:
Table 1: Number of reactors per type of reaction.
Ferrogel
98
Fermionic Condensates
52
Dysporite
35
Ferrofluid
13
Prometium
5
Of these materials, ferrogel and fermionic condensates are ‘advanced materials' used in the manufacturing of Tech II components. Dysporite, ferrofluid and prometium are ‘processed materials', which are used in reactors in combination with other materials to produce advanced materials, such as ferrogel. One thing of note is that all of these materials require either or both of the following base moon materials: dysprosium and promethium. These two moon types are, at the time of this writing, the rarest in the EVE universe and some of the most fought over. Therefore, having reactors that can avoid using these scarce resources and produce stuff out of thin air has a great potential to have a big impact on the EVE market.
In order to estimate the impact of this exploit on the EVE economy, we need to figure out how much was produced and what it was used for.
So what do we know?
First of all, we know how many reactors were in the bugged state and how much they were producing at the time the exploit was uncovered. We also know when these reactors were put online. Since the amount produced from each reaction is homogenous we can deduct from this information the maximum possible amount that was produced using this exploit. So during the first phase we can find out how much currently operating POSes have been producing.
The next step was to calculate the total value of the exploited material. Multiplying total volume produced through exploits and multiplying that by the average price of each material, and then subtracting the cost of operating the POS, resulted in an estimated total figure of 6.7 trillion ISK. That is a lot of ISK'ies for sure. But putting that into some kind of perspective is necessary. On a daily basis the total trade on the EVE market (not counting contracts or direct trade) in 2008 was between 2.5 and 3.0 trillion ISK. That gives us an annual trade worth more than 964 trillions in 2008 alone. So the maximum value of the exploit in 2008 is only about 0.7% of the total trade in one year.
But of course having capital helps in making even more capital, and that is indeed what they did. Using the profit from the exploit, the exploiters purchased high-end items and traded in high-end minerals in order to hide their tracks. Hence, though the impact was not big on the global market of EVE, it was surely felt locally and for some specific items. We can safely say that the size of the EVE economy saved the day here since even free stuff in big quantities can´t destroy the main markets of EVE.
Our logs also show that there were only three corporations that had 88% of the 6.7 trillion ISK profit from the exploit. These corporations were obviously setup specifically to harness this bug and when discovered were in the process of expanding their operations. More on their activities follows in the chapter below from the Law and Order team but we will focus on the impact on several different items.
The limited supply of dysprosium and promethium is one of the main reasons why the economic impact was so great, as this turned something that only had a finite supply into something where a small group was able to produce effectively infinite quantities. We will begin by looking at the end products, ferrogel and fermionic condensates.
Fermionic condensate is an advanced material made out of lower-end moon material used in Tech II production as well as the production of advanced capital components. Figure 1 shows the average monthly price for fermionic condensates and the monthly volume traded in all of EVE from September 2005 through December 2008.
Figure 2: The monthly trade of fermionic condensates. The red line shows total units traded on the market and the blue line shows the average per-unit price that it was traded at during that month.
The graph shows that total trade for ferminoic condensates rose steadily from September 2005 to November 2007 when there was a surge in quantity traded and an increase in price at the same time. This price increase was due to increased demand with the release of Trinity in December 2007. An interesting development happens around June/July 2008. Prices start to decrease and quantity increases as well. This is at the same time as some of the corporations established the larger operations with between 70 and 80 reactors online by the end of October.
A similar pattern is seen for the ferrogel. Ferrogel is also an advanced moon material used mostly for Tech II production. Figure 2 shows the average monthly price and volume traded for ferrogel from October 2005 through December 2008.
Figure 3: The monthly trade in ferrogel. The red line shows total units traded on the market and the blue line shows the average per-unit price that it was traded at during that month. There is an increase in price just after the Trinity expansion in December 2007.
We see the impact of the Trinity expansion clearly in this graph when the price increases in December of 2007. Quantity continues to increase but at similar level as before, even stabilizing in the second quarter of 2008. However, quantity starts to increase rapidly again in June/July which coincides again with the build up of bugged POSes. Prices are declining at the same time, until the discovery of the exploit in December and prices start increasing again.
Looking at this in combination, i.e. quantity for ferrogel and fermionic condensates with number of bugged POSes online, the trend is quite clear as is shown in figure 3.
Figure 4: The monthly trade in ferrogel, fermionic condensate and the number of bugged reactors online. The values for ferrogel and fermionic condensates are indexed to 100 on June 1st 2007. The value for POSes is indexed at 1 on June 1st, 2008
The rate of growth in the quantity of ferrogel and fermionic condensates is very similar between 2005 and 2008. This happened at the same time that demand for these materials was ever increasing and hence it was difficult to detect any anomalies through the market data. However, looking at individual items and the share of trade which characters using the exploit had we can see more accurately the impact that they had on individual markets.
Ferrogel was the material that was produced in greatest numbers by the exploiters. The following chart is the same as Chart 2 above but with the net trade from the top 10 exploiting characters added.
Figure 5: Monthly trade in ferrogel along with monthly average prices and quantities traded by the top 10 exploiters.
Figure 5 shows clearly the impact of the top 10 exploiters on the market. The quantity they traded from October 2005 through June 2008 was increasing but at a slower rate than the overall market trade. That changed however in June 2008 when the larger exploit operations became operational. The quantity traded by the exploiters increased significantly resulting in a continued downward trend in the price of ferrogel.
The same result can be seen for the fermionic condensates. Figure 5 shows the monthly trade for fermionic condensates, monthly average price and the trade by the same top 10 exploiters.
Figure 6: Monthly trade in fermionic condensates along with monthly average prices and quantities traded by the top 10 exploiters.
The same pattern emerges as with the ferrogel. In June/July 2008 we see an increase in trade by the top 10 exploiters and immediately a decrease in fermionic condensates prices. And once the exploit is discovered, prices start to increase again.
What we can learn from this is that after the Trinity release there was an increased demand for Tech II and capital ship inputs. This lead to a price increase which made using the exploit even more profitable than before. The exploiters scaled up their operations in the first half of 2008 increasing the available supply considerably resulting in lower prices for these inputs. As a matter of course, this impacted the market for Tech II items in general, the impact of which we can best see by looking at the price development of several Tech II items after the exploit was stopped.
The wider impact
The first one is ferrogel (Figure 5). The figure shows the average price (5 and 20 days) and median price of ferrogel in 2008 and into the first weeks of 2009. Prices start to drop around April 1st, and continue to drop throughout the year. Once the exploit became known to the general EVE player, prices increased instantly but are still around similar level as where they peaked in April.
Figure 7: Snapshot from the market history view in EVE. Ferrogel prices in Jita.
The next graph shows a very similar pattern for fermionic condensates. Prices drop early in April and continue to decline over the time period. Once the exploit is discovered, speculative trade kicks in; however, what is interesting is that initially the price increase was moderate but then in the second phase it really jumped to new historical highs.
Figure 8: Snapshot from the market history view in EVE. Fermionic condensates prices in Jita.
Similar patterns can also be found for other high end minerals; just look at ingame market data for minerals like dysporite or prometium.
The increase in supply of the high-end moon material does of course impact the prices for Tech II end products using that material. Looking first at the Deimos, a popular Tech II cruiser.
Figure 9: Snapshot from the market history view in EVE. Deimos prices and volume in Jita.
The price for a Deimos was relatively stable after an initial price decline early 2008. Prices fluctuated around 85 million ISK with a downward trend in the latter half of the year. As soon as the exploit was discovered, speculative trading started and price started to increase. The price seems to have stabilized for now around the same value as in the beginning of 2008, or about 95 million ISK per unit. That the price increases as soon as the information is available, and before the new production cost really kicks in, is a measurement that shows how efficient the EVE market has become. It basically reacts to new information instantaneously, making sure that the right price message is put through the system to all players in EVE.
Similiar price patterns can be found for other Tech II ships, as illustrated in the example of the Zealot which is shown in Figure 12 below.
Figure 10: Snapshot from the market history view in EVE. Zealot prices and volume in Jita.
Prices for the Zealot started increasing again the same days as other Tech II ships. However, Tech II items not using these high end minerals for their production did not have the same price pattern. Examples are the Cap Recharger II, Cruise Missile Launcher II and Tachyon Beam Laser II.
Looking at the Cruise Missile launcher II in Figure 13 shows that though prices increased somewhat in December, they soon decreased again.
Figure 11: Snapshot from the market history view in EVE. Cruise Missile Launcher II prices and volume in Jita.
So only the Tech II items needing the high-end moon materials have increased in price.
As we can see from Figures 9 through 13, the market reacted quickly to the news of the exploits. Prices spiked as speculators purchased large volumes of moon mining related materials and/or Tech II items requiring these inputs. Price of Tech II items that needed these materials increased rapidly. Currently prices are still increasing for ferrogel and fermionic condensates but other items seem to have reached relative stability.
Our estimate is that the exploiting parties made between six and 12 trillion ISK from this exploit. Taking the data available at the time of discovery, which showed the length of time the reactors had been used for the exploit, and looking at the per-unit value of the exploited items, showed us that the minimum was six trillion. Taking into account reactors that have been used historically and considering higher prices in the past, we believe the upper limit of how much could have been made to be around 12 trillion.
From the technical part earlier in this blog, we basically can assume that the exploit has been possible from the beginning of player owned structures in EVE. After digging through terabytes of data we know that the exploit existed on a small scale until 2007. Until then, the cost of setting up POSes and operating them probably outweighed the benefit of large scale operations and hence these operations were able to stay under the radar. With invention for Tech II items introduced in late 2006 things changed rapidly. Increased demand for Tech II inputs due to innovation led to increased prices and corporations that were aware of this exploit started to build up their operations, though they kept the secret within their group so this did not become widespread knowledge. The operations expanded throughout 2007 but we really started to see them build up to industrial scale in late 2007 and early 2008. So the greatest impact of this exploit was during the second half of 2008.
The important news here is that due to the single shard approach and the very efficient market structure this exploit did not threaten the EVE economy overall. It has impacted the market for several important items but as time passes this will simply be seen as a dent in the general price development for those items. EVE is strong.
However, that does not change the fact that this is a serious violation of the EULA and is dealt with as such, which brings us to...
Crime and Punishment by CCP Grimmi
As clearly stipulated in our rules and policies, exploiting is strictly prohibited. In our Suspension and Ban Policy there is a special clause about so-called "duping" exploits. Employing this sort of exploit will lead to permanent bans for anyone directly involved as well as possible reprimands for players who benefit from such exploits from removal of the items in question up to, and including, banning of their accounts.
A "duping" exploit is basically an exploit by which some bug in the system is used to create items or ISK out of nothing. The POS reactor exploit was an example of a "duping" exploit and it was handled accordingly.
At the time we discovered the exploit in early December, there were 232 reactors running in the bugged state. Those were installed at 178 POS complexes owned by seven corporations. The scale of the operations differed quite a bit, with one corporation running 81 bugged reactors and another with 3 reactors active.
The opening action on our part regarding the exploit included the total destruction of all the POS complexes involved. This entailed flying to each one and basically nuking everything in sight - a fireworks show of epic proportions but with no witnesses except the GMs in the demolition team.
Users directly involved in the exploit were permanently banned. Direct involvement meant that the character had a director role in the corporation using the exploit or was directly involved in servicing the POSes in exploited state. Others that were found to be involved in moving the exploited goods and laundering the ISK also received bans for their part. A number of players who had benefitted directly from the exploit were also banned. The total number of users banned in relation to the exploit of POS reactors is 134.
The purpose of the exploiting was quite different for the corporations involved. Two of them, including the one with the largest operation, were found to be involved in RMT, selling off the proceeds for real money to random players. Another two had funneled what they had gained into expanding their operation and at the time of discovery had not made much more from the exploit than what they rolled over back into the operation. The last three were exploiting the reactor bug in order to gain unfair advantages for the users involved, meaning that ISK and assets were moved to their other characters.
The assets removed from the game by our actions because of this exploit consist of large numbers of capital ships including some motherships and titans, over 30 Tech II BPOs and other valuable items as well as large amounts of ISK.
How did we not see this?
To date, we have found three petitions regarding the reactor bug in our systems. Two of those are since late October 2005 and one since late October 2007. In all cases, the issue was handled as an isolated bug for the players reporting it and the work done was geared towards fixing that particular issue.
In one of the older cases, the user petitioning was asked to file a bug report but the resulting report was closed after the bug hunters were not able to reproduce the issue. At the time, no procedures were in place on our end to ensure that reproduction steps were included. Nowadays, our bug hunters will contact the player submitting the report and request that the steps be added if they are missing.
Quite frankly, it must be said that at the time, the documentation and logs available for POS-related things left much to be desired and anyone involved in handling such issues would have been facing a very difficult task indeed. Very little information was available to staff and players alike about how things were supposed to work and what little logs existed were in no way sufficient to provide information needed to successfully tackle problems with POS mechanics. The usual stopgap fix was to simply repackage the structures and hope that it would take care of the issue at hand.
The last petition was correctly filed into the exploit category but it was simply handled as an individual problem for the player reporting it. Thus it fell through the cracks and did not raise the flags it should have and no exploit investigation was launched.
All the staff members involved in handling the cases have been thoroughly investigated and cleared of any involvement in the exploit by our Internal Affairs department.
Internal Affairs have also investigated other staff members for involvement in the exploit and have found no links to the exploiting corporations or characters involved in the exploit. CSM members were checked and cleared as well.
How will this not happen again?
Our systems for detecting issues failed and this brought it to our attention that they are due for an overhaul. The way exploit petitions are received and handled is currently being restructured. The same is true for the bug reporting tools and work procedures in regards to how bug reports are handled.
The QA and Customer Support departments are working on these matters together and new updated systems and procedures will be implemented as soon as they become available. In some cases, the necessary changes have already been implemented.
Active monitoring of individual items on the market will also be part of our line of defense against exploits and handled by our Research & Statistics department. This means that instead of looking at general trends based on our interests in researching specific markets we will focus more on automated detection of anomalies In the market data.
So in the end.....
There will be exploits in the future and we will do our best to discover them in their early stages and minimize their effect on EVE with new and proactive procedures. We do hope that the EVE community will accept our way of handling these once they are found. The procedures and rules used against those using these exploits will also be reviewed on a regular basis. We have already had discussions with the CSM on fines and other tools to punish those that directly, or indirectly, reap the benefits from illegal activity within EVE. No final decisions have been made yet, but now is your chance to contact your CSM representatives and let your voice be heard.
