• We are looking for you!
    Always wanted to join our Supporting Team? We are looking for enthusiastic moderators!
    Take a look at our recruitement page for more information and how you can apply:
    Apply

The reality of probability...don't let your brain warp it

Sharmon the Impaler

Well-Known Member
So you are a customer and want to be treated as a customer. What would be the logical approach to take? Contact customer support, or start arguing with other customers?
You look in a product’s forum and ask questions to ensure that it is a problem before contacting support. I am not arguing with anyone , if someone needs to be right and argue so they feel special then they can go ahead and be right and argue.
 

Johnny B. Goode

Well-Known Member
As luck would have it, I‘ve been collecting ToR samples from several cities since CDMark posted their data. I’ll even share it for you. I was going to save it for a Christmas present once it reached 10K, but it’s close enough.

View attachment 19300

I’ve added the 99% confidence intervals (lower, upper, width), as well as tracked how each entry performed relative to expected value after Lord Pest‘s assertion that their outcomes weren’t being evenly distributed above and below expected value. There are several troubling findings (Skip this part if you are a statistics expert and want to draw your own subjective opinions about the data).

- Three of the tracked levels are already performing outside of the 99% confidence interval (note that most data analysis is only run against 95% confidence). Several others are approaching the outer spectrum and just need another month or so of values to shrink the width.
- The overall total value is already outside of the 99% confidence interval.
- Only one level is exceeding expectation.
- The distribution of the results versus expected value should roughly be about 11% hitting expectation, 44.5% below, and 44.5% above. The actual results are heavily weighted to underperformance with about 62% of the results being below expectation and only 24% above.
- The average relic gap for the misses is 3.5, while the average relic gap for the overages is 2.5.
- The total performance is 2.4 points under advertisement (in real value not as a percentage), well outside of statistical variance for this quantity (basically double), and it’s caused a 7.5% reduction in relics.

Coupled with CDMark’s data and even Lord Pest’s assertions, there is clearly something up with relic generation. Maybe RNG, maybe a table, maybe bad code, maybe something else.

Now, what I can’t predict is which worn-out response will be used to rebut this data, maybe:
-what about the millions of other ToR hits
-you need to sample at least 100,000
-this is just a bunch of bad luck on top of more bad luck and more bad luck that just happened to be tracked
-what about the one level that is overperforming
-its only off a couple percent, that doesn't prove anything, it will get closer if you just keep tracking
-obfuscation!
-gee, maybe there is something to this probability theory stuff, I think I might take the time to learn more about that and see if it could be used to explain why things happen in chance events and provide more constructive feedback to misguided individuals in the future

Ok, so one of those is off the table…….
Despite all the numbers in that chart, there are only two relevant ones. Published % and Observed %. And if you look at those two numbers, they are in line with CDMark's tracking. Less than 2.5 percentage points below expected. And CDMark's conclusion was that it wasn't a big deal. Will it get closer if you keep tracking? Who cares? I certainly don't care about it enough to track relic spawning.

By the way, Lord Pest's assertions mean absolutely nothing. They are just that...assertions...not data.
You look in a product’s forum and ask questions to ensure that it is a problem before contacting support. I am not arguing with anyone , if someone needs to be right and argue so they feel special then they can go ahead and be right and argue.
That's funny, because you are here arguing that it is a problem. You've already found a couple of people that agree with you, so go ahead and contact support now. Or continue to "not argue" about it here.
 

Pericles the Lion

Well-Known Member
Despite all the numbers in that chart, there are only two relevant ones. Published % and Observed %. And if you look at those two numbers, they are in line with CDMark's tracking. Less than 2.5 percentage points below expected. And CDMark's conclusion was that it wasn't a big deal. Will it get closer if you keep tracking?
CDMark has two cities, two TORs (one @57 the other @62). Using 60 as the average, there is a 30.25% chance of a relic spawning. If his results show that the spawn rate was 2.5 percentage points under the published rate then this is an 8.3% deviation from the published rate and that is a big deal.
Who cares? I certainly don't care about it enough to track relic spawning.
If the actual spawn rate is 8.3% less than the published rate then I care as should any player.
 

Agent327

Well-Known Member
I suggest that you read back through this thread starting with #60. Methinks that you've lost track of the subject matter.

You asked if Support would answer the question. I told you they would. You made the distinction between an answer and a response. That must mean you asked the question, so I am eager to know.

Now don't tell me you are just assuming what they would tell you? If so, just say so. You are not very transparent at the moment.
 

Pericles the Lion

Well-Known Member
You asked if Support would answer the question. I told you they would. You made the distinction between an answer and a response. That must mean you asked the question, so I am eager to know.

Now don't tell me you are just assuming what they would tell you? If so, just say so. You are not very transparent at the moment.
Did you read back through the posts? I think not. I did NOT ask if Support would answer the question. I specifically asked you if you were implying that Support would answer his question. Look at post #60.
 

Agent327

Well-Known Member
Did you read back through the posts? I think not. I did NOT ask if Support would answer the question.

Are you implying that contacting Support will result in an answer to the question?


I specifically asked you if you were implying that Support would answer his question. Look at post #60.

Are you implying that contacting Support will result in an answer to the question?
 

iPenguinPat

Well-Known Member
As luck would have it, I‘ve been collecting ToR samples from several cities since CDMark posted their data. I’ll even share it for you. I was going to save it for a Christmas present once it reached 10K, but it’s close enough.

View attachment 19300

I’ve added the 99% confidence intervals (lower, upper, width), as well as tracked how each entry performed relative to expected value after Lord Pest‘s assertion that their outcomes weren’t being evenly distributed above and below expected value. There are several troubling findings (Skip this part if you are a statistics expert and want to draw your own subjective opinions about the data).

- Three of the tracked levels are already performing outside of the 99% confidence interval (note that most data analysis is only run against 95% confidence). Several others are approaching the outer spectrum and just need another month or so of values to shrink the width.
- The overall total value is already outside of the 99% confidence interval.
- Only one level is exceeding expectation.
- The distribution of the results versus expected value should roughly be about 11% hitting expectation, 44.5% below, and 44.5% above. The actual results are heavily weighted to underperformance with about 62% of the results being below expectation and only 24% above.
- The average relic gap for the misses is 3.5, while the average relic gap for the overages is 2.5.
- The total performance is 2.4 points under advertisement (in real value not as a percentage), well outside of statistical variance for this quantity (basically double), and it’s caused a 7.5% reduction in relics.

Coupled with CDMark’s data and even Lord Pest’s assertions, there is clearly something up with relic generation. Maybe RNG, maybe a table, maybe bad code, maybe something else.

Now, what I can’t predict is which worn-out response will be used to rebut this data, maybe:
-what about the millions of other ToR hits
-you need to sample at least 100,000
-this is just a bunch of bad luck on top of more bad luck and more bad luck that just happened to be tracked
-what about the one level that is overperforming
-its only off a couple percent, that doesn't prove anything, it will get closer if you just keep tracking
-obfuscation!
-gee, maybe there is something to this probability theory stuff, I think I might take the time to learn more about that and see if it could be used to explain why things happen in chance events and provide more constructive feedback to misguided individuals in the future

Ok, so one of those is off the table…….

Beautiful data set. Also, regardless of the total population size, your sample appears to be large enough to validate a statistical issue (as you already mentioned).

Yep, pretty much. You take one player's data and assume it's applicable across the board. If you don't know that you're wrong in that, you're not as smart as I thought you were. Unless there's something else in that data that isn't being shared. It's kind of hard to know unless someone shares a link to that post.

EDIT: Nevermind. I found the post. Another one where you argued ridiculously for a tiny set of data to indicate a system wide problem. Even the player that tracked and shared that data stated that he didn't think it indicated anything. And remember, the thread that triggered this one was over 4 level 9 Pirates Hideouts tracked over a measly 3 days (or maybe it was 3 of them over 4 days, in either case it was 12 collections) that didn't pay out FPs. All you conspiracy theorists jumped on it as an indication of a problem with game mechanics. Same old same old.

I have collected Data and reviewed players' data as well. Samples were collected over a wide range of time and not likely a knee jerk emotional response to a string of bad luck. Samples range from 1200? To 10,000... They were also different aspects of the game (tor and gbg).

All have had the same experience -- 1-3% off with the larger sample looking less favorable to players.

Given the collect of data, I think I would be dishonest to not question if something is off.

1-3% doesn't make a big difference with small numbers. When you're dealing with $1 BILLION, that 1-3% is a lot of money.

I'm ready to see evidence that these combined 25,000 samples are just a fluke.

At this point, there hasn't been data presented that refutes our concerns. No amount of name calling changes the only facts presented look bad for inno.
 

Emberguard

Well-Known Member
I'm ready to see evidence that these combined 25,000 samples are just a fluke.
This needs to be answered then:
Did either of you ensure you pick up every relic at the time of it spawning? Or did you leave them spawned and only pick up when you completed the level?
If those samples had the player picking up as they spawned then we can be certain the system isn’t blocking new spawns from lack of spawn points. If however they’re leaving them until they have several relics accrued then that just supports my own testing before that seemed to suggest spawning limits on types and amounts of relics on the screen at any one time.

The RNG could be trying to spawn and due to all spawns currently occupied be unable to spawn another until those on the map are picked up. Was the dataset with or without picking up relics as they spawn?
 
Last edited:

UBERhelp1

Well-Known Member
This needs to be answered then:
If those samples had the player picking up as they spawned then we can be certain the system isn’t blocking new spawns from lack of spawn points. If however they’re leaving them until they have several relics accrued then that just supports my own testing before that seemed to suggest spawning limits on types and amounts of relics on the screen at any one time.

The RNG could be trying to spawn and due to all spawns currently occupied be unable to spawn another until those on the map are picked up. Was the dataset with or without picking up relics as they spawn?
I think you're right but slightly wrong. I'm almost 100% sure that there are 16 spawn locations on the map so that you can't completely fill up all the spots. The game, though, tends to put rarer relics towards the end of the map than the front. There might be set values saying spot A can only have gold or jade relics, and that's how it becomes harder to fill it up. Or, the game could do a check and attempt to spawn a relic at a certain spot and if it's filled the attempt fails.
 
Did either of you ensure you pick up every relic at the time of it spawning? Or did you leave them spawned and only pick up when you completed the level? If there are any limitations in how many can spawn then it won't matter what the % is (just like how there are limits on Incident spawns: if you don't have a spot available for it to spawn then it won't spawn until a spot opens up)

I can’t say for certain on all of the data on my sheet. Most of them are immediately collected. I can vouch that 2 of the levels that are already out of bounds are immediate collects (assuming they aren’t in fog, of course). It’s an interesting thought, though. One of those “not RNG issues that present like RNG issues”. I used to let my relics sit and can remember getting maybe 8 or 9 on a single level before collecting. In the data that I directly collected as part of this data set, I only exceeded a total of 6 relics on a level twice.
 

Johnny B. Goode

Well-Known Member
Beautiful data set. Also, regardless of the total population size, your sample appears to be large enough to validate a statistical issue (as you already mentioned).



I have collected Data and reviewed players' data as well. Samples were collected over a wide range of time and not likely a knee jerk emotional response to a string of bad luck. Samples range from 1200? To 10,000... They were also different aspects of the game (tor and gbg).

All have had the same experience -- 1-3% off with the larger sample looking less favorable to players.

Given the collect of data, I think I would be dishonest to not question if something is off.

1-3% doesn't make a big difference with small numbers. When you're dealing with $1 BILLION, that 1-3% is a lot of money.

I'm ready to see evidence that these combined 25,000 samples are just a fluke.

At this point, there hasn't been data presented that refutes our concerns. No amount of name calling changes the only facts presented look bad for inno.
So all of a sudden you have 25,000 samples on all aspects of the game? And through all the discussions of this over the past couple of months you haven't mentioned them at all? Yeah, right. And they're all "1-3% off"? Yeah, okay. Seems awfully convenient.
 

Pericles the Lion

Well-Known Member
So all of a sudden you have 25,000 samples on all aspects of the game? And through all the discussions of this over the past couple of months you haven't mentioned them at all? Yeah, right. And they're all "1-3% off"? Yeah, okay. Seems awfully convenient.
Considering that @iPenguinPat has 11,600 followers and collaborates with @MooingCat I doubt that he's being disingenuous. If he says that he has collected 25K samples then I believe him. I've seen his spreadsheets. It is my opinion that he is no stranger to mathematical/statistical analysis.
 
Last edited:
Top