The End of the Recruiting Bonus
Over the past few months we have been seeing an increase in recruiting cheaters. This caused us to completely overhaul our medals system last month, only awarding the recruiting achievements for recruits that had completed training (level 100), and only awarding recruiter cube bonuses once per device.
Today, I’m saddened to report that we have suspended all future referral bonus cubes for players that recruited their friends. Ultimately it is the cheaters who caused this problem to bring us to where we are today, but it was Apple that made this necessary.
Without going into technical details, we have found that Apple’s imposed limitations on applications in iOS7 makes it impossible for us to reliably stop cube farming programmatically, and we lack the human manpower necessary to continue to combat this problem on an ongoing basis. So far, the problem has only manifested itself in a handful of players, but recent developments with an abusive player and his friends have forced us to take action today.
We continue to work hard to keep QONQR fair for all players. We receive numerous reports of bad players every day. It is a frustrating and demoralizing part of our day to say the very least and unfortunately takes us away from adding new features to the game. We thank those who play QONQR with honor and integrity. We hope you understand our decision today. Had there been a reliable technical solution, we of course would have endeavored to keep the bonus in place. Your recruits will still receive 2 cubes for using you as a recruiter, at the time of their enlistment.
Thank you for your continued support of QONQR. You make this game great, we are working hard to keep it that way.
-Scott (aka Silver)
The Passing of MrBizzy
Last week, I received a shocking and surreal support email from one of our players in Michigan. A fellow QONQR player, MrBizzy, had died unexpectedly. The person contacting me had asked if we could offer any clues for his friends and family, because MrBizzy was an avid QONQR player and was probably playing much of the day.
It was a bit shocking to see deployments many times an hour, for 6 hours straight, and then nothing. I didn’t expect the feeling of emptiness I felt looking at that last log entry. I’ve read his obituary and wish I had known Vuthy Pen. He was my age, was a co-founder of an IT firm, and leaves behind 4 children. I didn’t know Vuthy, but just the same, I’m saddened by his passing. The entire QONQR family extends our deepest sympathies to Vuthy’s family and friends.
A fellow QONQR player sent this, and asked us to post it.
On Monday, November 11th, Michigan Swarm lost an exceptional player named MrBizzy.
He was all one could ask for in a teammate: kind, humorous, helpful, and never hesitant to rain down missiles or stack a zone to a million bots. He welcomed new players, frustrated the competition, and kept us all laughing. We lost a true friend and a great leader.
However, Swarm does not mourn him alone. An outpouring of support was shown by Legion and Faceless in a temporary cease-fire that began almost as soon as the news of his death was publically announced.
Many players, from all factions and multiple states, changed their in-game avatars to one honoring Bizzy.
The Qonqr forum had a thread this morning discussing a nation- or world-wide way to pay him tribute.
Our team members have received many messages of condolence and support… and all of this means the world to us.
Michigan Swarm feels like a family, yet we recognize that Bizzy’s actual family has lost a father, son, husband, sibling, cousin, and uncle. His game-play was a true reflection of a life filled with caring for others, leading by example, and being a joy to those around him.
Bizzy, you will truly be missed and we are all better players and people for having known you.
The QONQR players on the forums have suggested we make Thanksgiving Day in the United States (November 28th) a day of remembrance and thanks in Honor of MrBizzy. The players have suggested that we “Stack to the Heavens” for MrBizzy, deploying only defensive bots and avoid battling for the day. We agree this is a great way for the community to give tribute to a fellow QONQR player, and show how thankful we are for the new friends we have all gained through QONQR. I will be displaying the MrBizzy tribute image as my profile picture through Thanksgiving, and I hope you will too.
-Scott (aka Silver)
Faction Leader Character Design
We received an email recently from a player expressing disappointment about Kimyo’s look in the new comics. This player had chosen to enlist with Swarm partly because Kimyo was a normal looking girl “who was super smart and didn’t dress like an idiot”, and she felt that the updates to Kimyo were intended to sexualize her for marketing reasons.
I responded right away because this is a something I take very seriously as the Chief Creative Officer, and now I want to share some of the resulting discussion here on the blog to give a behind-the-scenes look at some of the discussions that go into character design.
Scott (our CEO) and I both have young daughters who are very into tech. They build their own computers, write code, are active gamers, and dress as Star Trek characters for Halloween. Our only full time employee is a passionate gamer with two young daughters of her own. When our kids get together, they trade Star Wars trivia and discuss Minecraft. QONQR also sponsors our local Girls in Tech organization. In short: this matters to us.
When I was designing the faction leaders, I intentionally made two of them female. And my instructions to our illustrators from the beginning has been to not sexualize either of them. We want these to be strong female characters who we can create stories around. My daughter is an avid comic reader, and I wanted her to be inspired by Kimyo and Sunday.
Our new illustrator is amazingly talented, but comes from a traditional comic background. I have had him modify concept sketches of Kimyo when I felt she looked too alluring. Here is an early concept sketch of Kimyo. I felt she was too voluptuous and asked him to make adjustments.
Here is the revised version which retains our illustrator’s slightly cartoonish style that I love, but makes Kimyo’s proportions more realistic and her top less revealing.
I also asked him to redraw several panels of the most recent Swarm page because I felt Kimyo’s position while digging into the Fab Lab tech was too suggestive. Here is the sketched layout for those pages initially. You can see the final page with modifications here.
Kimyo’s slightly different look isn’t because we felt the old illustrations were inadequate, it was because we hired a new illustrator, and every artist has their own style. I love the new illustrations and how they are bringing the QONQR story world to life. I didn’t think the mid-drift tank top objectified Kimyo. She’s a young woman who is a brilliant scientist, runs a multinational corporation, and leads a movement that strives for technological advancement. But she’s also puckish — she dresses unconventionally, dyes a green stripe in her hair, and often ignores those around her when she’s digging into some fascinating bit of technology. She’s a less abrasive version of Tony Stark, but this time as a confident young woman.
These discussions about Kimyo are just an example; we’ve had many similar discussions about Sunday too. Discussions like this have been ongoing since we created the Faction Leaders, and they will continue for the foreseeable future. But rest assured that we are just as protective of these characters as you are.
-Justin (aka GungHo)
I’m excited to share a new reward system that became available today. The developers built a tool I can use to award players one cube for awesome twitter smack talk. We now have the ability to easily reward you for the awesome stuff you already do to promote QONQR.
From the beginning QONQR’s growth has been tied to you, the players. When QONQR launched the beta at SXSW in 2011, there was no marketing and no PR, only the ability to Tweet you had just captured a city.. Yet, within three days of the beta launch, players had captured cities in 20 states, 10 countries and 4 continents. Our players telling other players about QONQR has always been our primary method of growth, in fact, because of you we managed to cross the 100,000 player mark without even purchasing a Facebook Ad. Thank you!
Going forward, I (@Gadgerson) will be watching for awesome #Qsmack tweets and other things that evangelize for QONQR. Keep up the good work and maybe your tweet or YouTube video will be the one I pick when I give out some bonus cubes for the month. Be sure to tag your tweets with #QSmack to make it easier to find.
I’m happy to say I started awarding cubes today and it felt great! I’m excited to see what you Tweet next so I can give out more Cubes.
New Medals and More
Last night we released a new server update. The new Swarm pages are finally here! The third Legion page is also posted.
This update also includes the ability to exchange cubes for credits in bulk through the web portal. Take advantage now while the exchange rate it still 6500 to 1.
There are some new Medals at the top of your list of awards/achievements. “Complete Bootcamp” and “Complete Training”. You earn these medals respectively for achieving levels 10 and 100. These medals were added to incentivize new players to level up quickly, but more importantly they help with the vocabulary in the game. Now you will be “helping your friends through boot camp” so you can get the recruiter bonus and try to convince your enemies to switch factions “before they complete training”.
In our ongoing struggle to stop cheaters, we have revamped recruiting. In the past month, we have seen an influx of reports for people who are “farming recruits”. The old recruiting system created days upon days of lost time to dig through players’ recruit list to evaluate which are legitimate accounts and which were multi-scope accounts to either earn the recruitment badge or bonus cubes. We banned many people in the past month as a result of multi-scoping to farm the system.
This is how the new recruiting system works.
1. You still earn 5 cubes for recruiting a player who makes it to level 10, but if that award is for your son playing on an iPod, he gets bored, and you create an account for your daughter to play on the same iPod, you will not earn another 5 cubes for her. Recruit cube bonuses are now limited to one per device.
2. All the recruit medal/achievement counts have been lowered. Previously you needed to recruit 1, 5, 10, and 25 recruits to earn the four recruiter badges. Now you only need 1, 2, 5, and 10.
3. Your recruits must now complete training (level 100) for them to count as your recruit.
We did much analysis on this change. Our goal was to avoid penalizing people who recruited honestly, but that couldn’t be avoided 100%. We had people who moved both up and down as a result of the change. Unfortunately for those who were negatively impacted, the change was necessary to stop the cheaters.
In addition to the medal changes above, we added a few secret medals. They will not appear on your awards list until you have earned them. I’m sure it won’t take long for this community to find them. A year ago when we first introduced medals, we received feedback that the top levels were “impossible” because “it would take a year of obsessive playing to do that,” and should be lowered. We listened to the community and lowered the requirements for the top requirements. However, some of our “obsessive players” have been playing for over a year, have hit those previous marks, and now want those exclusive levels back. We chose a new top mark that had at least 2 players who had already earned the medal in each category. Good luck.
Finally I am proud to announce that we now have our first two Sergeant Majors in the game. Again, you will need to find them. I’m not going to risk making them a target by calling them out. These players earned points through the new medals to push them over the rank requirements to earn the highest rank possible in the game. For those that are wondering will there ever be anything higher. Yes. We used the enlisted ranks for version 1.0 of QONQR. We are holding the NCO ranks for the version 2.0 expansion pack, which we hope to start building the second half of 2014.
We continue to march down our list of features to add to the game and continue to fight against cheaters. Thanks for being part of our journey.
Yep. It is.
Why do we cheat? We all cheat at something. We cut in line, look at the answers on the kid’s answer sheet sitting next to us. These little “cheating” things we do as kids, we usually grow out of it at some point…right?
I love this game and some of the things I love about it most, is the passion with which the players play it, the community that those players have formed, and the dedication of the developers to make it the best game that it can be. For those of you that I just described…you are the reason I get up in the morning and go to a job I love every day, so thank you.
However, not everyone who plays the game, follows the rules of the game. Some of you do not seem to know what those rules even are, so I thought I’d break down some terms of service key points for you that most of you skimmed or skipped altogether.
- Deploying for a friend, spouse or child is against the rules and grounds to have your device banned.
- Starting a second account and playing it on a tablet along with your main account is also against the terms of service and punishable by having your device banned.
- Creating a bunch of dummy accounts in order to get free cubes and a recruitment medal is also against those same rules and can have the same consequences.
Number 1 actually surprises some people. We have some players who email us after their account have been banned claiming, “I have never cheated, ever!” Then when we respond, “Our algorithms detected with 90% certainty that you were deploying for [Player2] for over an hour on X/X/XXXX,” the player responds, “Holy crap, you can tell that? I didn’t know that was bad. I promise I won’t do it again.”
Yes we can tell, and while we don’t check every launch everyone does, you never know when we might look. Our algorithms don’t distinguish between you launching for your girlfriend every so often, and you playing on a second account from your iPod, which never leaves your side.
In order for all of the things I love about QONQR to stay intact along with the game’s integrity, we need your help in spreading a very important point around this community that we all love…
”Friends don’t let friends multi-scope.”
The Team Is Growing
Many of the QONQR veterans know a fair bit about QONQR’s beginnings, our struggles and our road to get where we are today. Here is the 30 second recap for the new people. Justin pitches the idea of QONQR at Startup Weekend, which we win with a proof of concept built in 48 hours. Successful HTML5 Beta follows. Three partners quit jobs live on savings for a year, work on QONQR for a year. Failed to find an investor. Two partners run out of money, leave to get “jobs”. Scott (Silver) runs the company for a year mostly solo. Company finally becomes profitable 2.5 years after Startup Weekend.
Today I’d like to share some big milestones that our players have helped to make possible. First let me share a photo of the people who have had a significant contribution to QONQR over the years.
From Left to Right:
- Jessy: Original Startup Weekend team member and Founder. Server Dev. Currently limited engagement beyond board.
- Donn: Founder. Android guru (as well as many other technologies). Part-time Android lead dev.
- Justin: Original Startup Weekend team member and Founder. Full time Chief Creative Officer. Server and iOS developer, Brand/Product manager.
- Joe: Android and iOS developer. Part-time.
- Scott: Original Startup Weekend team member and Founder. Full time CEO. Windows Phone and server dev. In charge of picking up garbage and making coffee.
- Joel: iOS ninja. Part-time work on QONQR.
- Andy: Original Startup Weekend team member and Founder. Returning full time soon. Original game mechanics designer.
- Jeff: Founder. Cloud Architect. Currently limited engagement beyond board.
- Dave: Founder. Graphic design. Currently involved on request. Generates amazing logos, icons, screen designs, etc
- Christie: QONQR’s first employee. Customer Engagement Manager. Head of support, Testing and Social Media.
- Adam: Former Army Intelligence officer (stationed in Baghdad), now checking for QONQR cheaters a handful of hours a week.
Not pictured: Ashley – contracted, original faction leader illustrator. Tori – contracted, music. Cole – intern, server dev. Becky – contracted, HTML developer. Joey – comic artist
Today marks 3 years since I walked into the Secretary of State’s office and filed our papers to legally create QONQR (the company), three weeks after having won Startup Weekend. Some amazing people have worked for QONQR over the past three years. Even when things looked the bleakest, and we were down to one person focusing on QONQR, we continued to have a great group of people who worked when they could to keep QONQR moving forward.
That persistence finally paid off earlier this year when we had reached the point where we had paid all the people to whom we owed money, I could take my first paycheck in 15 months, and the bank account was bigger at the end of the month, compared to the previous month. We were profitable.
It is important that we share our continued success with our players and fans as we grow. You are responsible for the success we have today, and we want to give you credit for recent news. Justin and Andy worked on QONQR for a year, but had to leave a little over a year ago due to the financial strain of living on savings. Now we have the funds to pay our partners a moderate salary. Justin is back full time. He is the father the QONQR idea and is back helping me run the company day to day. We are lobbying for the return of Andy, the mad scientist behind the Plasma Beam and much cursed “maximum overheat,” but the timing of his return is uncertain due to a critical role he plays at the job for which he left QONQR.
Most exciting for us is the announcement of our first full-time official employee. Christie is a long time QONQR player and serious hardcore gamer. I met Christie through a mutual friend, and later discovered that she was causing all sorts of havoc for her QONQR opponents through her aggressive conquest of nearby zones. I knew Christie had some experience with social media and asked if she could help manage our Facebook and twitter. Christie has posted every “Where in the World is QONQR” post on Facebook without missing a day for more than 6 months.
Christie is our new Customer Engagement Manager. Her focus will be support, quality assurance, and social media/marketing. We are very excited to have Christie on board and look forward to her help. Based on her initial blog post, we think she is pretty excited too.
Until now we have been a startup largely scraping by on the generosity of our partners and subcontractors who were willing to work extra hours after doing their day job. 60 days ago we were pretty much a 1 person company, 30 days from now we will be a 3 (possibly 4) person company with the partners and Christie all working full time.
We would like to sincerely thank our players and fans for their support. I am very excited to have Justin back (maybe Andy soon), and Christie’s contribution will make a huge difference in our daily workload. You made this possible. Thank you!
-Scott (aka Silver)
Yesterday we released quite a bit of new code. An update was submitted to Apple containing stability and bug fixes. Since the release of iOS7 SDK, we have been working to rewrite much of our application, which was originally written for the iOS5 SDK. This most recent update is focused primarily on performance and memory management, as well as some bug fixes. It should be available in a few days.
An update to the Android beta was released yesterday. From a functional point of view, we believe Android is very close to having all the features and a reasonable level of stability that would allow for a general availability release. However, we continue to be plagued by the “openness” of the Android environment. Despite exhaustive attempts, we have found that location spoofing is nearly impossible to stop on rooted Android devices. As a result, we now need to write code on the server to identify and penalize people who hop locations. This will take quite a while to create and will cause a further delay.
A fairly large web portal update was released. The first two Legion and third Faceless web comics were published on the portal. FYI, our illustrator is starting on Swarm comics on Monday. New leader boards for zones captured, zone leads, bot kills and credits harvested were released. If you like to send messages in the game, but hate the tiny keyboard on the phone, you can now use the Wire on the portal from your computer. You can change your password from the portal now, so if you have support reset your password, you can now go change it to something you can remember.
Finally, the dreaded “Comm Channels Jammed” error should be seen much less often by players. This is our generic error. This error is nearly always encountered by players when their scope is out of sync with the server and has the wrong bot or energy count. Windows Phone and Android users find themselves in this situation often when they hit launch, then the hardware back button, then launch again in rapid succession, emptying their bot tank in a couple seconds. If the multiple launches that happen at almost the same time are processed out of order by the multiple web servers, the phone will probably get the launch results out of order and the last one it processes will probably have the wrong bot count. Often the last one processed will be a “Comm Channels Jammed” error, which resulted because you launched too fast and your last attempt failed because you didn’t have enough bots left. Veteran players have figured out that harvesting your bases will re-sync the scope, but it was a frustrating workaround. We have now fixed this issue on the server by sending back a more detailed error, stating you are out of bots, energy, missiles, etc. There is also extra data included in this error response that will force your phone to re-sync with the server.
We had a very good planning session yesterday following the releases and have made plans for some much-requested new features. I’m not going to share our roadmap though, because if QONQR has taught me anything it is that priorities shift quickly. However, I will give you one hint, since the feature is nearly complete and it is safe to assume it will appear in the next few weeks. Bulk Purchasing.
Pay To Win
There has been much discussion on the forums and in a few emails lately about the players in QONQR who are “ruining” the game by spending too much money to dominate their areas. I received such an email earlier this week and thought I would share my response for the larger audience.
Game balance is something we struggle with every day. It is one of the most complicated computer problems I’ve ever worked on and I’ve worked in the cancer research and insurance fields.
We love our free players and we need them. But we have to give the players that spend money the ability to win (in the short term) otherwise there would be no reason to spend money. The balance we try to strike in QONQR is that a players spending money should be able to overcome a smallish group of players 3-10 players, but would be unable to spend (within reason) to overcome many more than that.
In mobile gaming only a small number of the players spend money. For example if 10% (1 in 10) spend money (the industry average is closer to 3-5%), then it would make sense that 1 player spending money should be able to hold his/her own against 9 free players. QONQR’s numbers are slightly better than industry average, but not 10%. Honestly I think we favor the free player on the whole. I know there are a few players that held off major attacks mostly on their own, but holding off a 20 person attack solo seems like a difficult expectation, especially if the attack is sustained for more than a day. I don’t know many paying players who could beat more than 10 free players, indicating our formulas may skewed in favor of the free players by a small margin.
We know pay to win happens in QONQR, but it is only winning the battle, not the war. Even those that have spent thousands in the game are often getting their butt handed to them in daily skirmishes and are forced to spend again and again. The little guys get beat down over and over, they retreat, regroup, reorganize, recruit and go back on the offensive. We have seen this happen many times. Six months ago the Legion were nearing an unsustainable level in most places around the world and we were concerned that we may soon see a two faction game. But the underdog was put in a situation where they needed to recruit and be better organized. The Legion have seen a resurgence in the past few months and they are rapidly advancing in many areas. As a whole, the percentage of Legion players spending money has not increased. The number of players spending money has grown at an even pace with the total player growth, which reflects their success is mostly due to better organization of the free players.
We know we will have players who love QONQR, but quit out of frustration. Unfortunately it is a necessity of the game to keep things competitive. We have plans to add game expansions that will create more reasons to play QONQR. Right now, if you can’t be successful battling, there is nothing else to really focus on. Over the next year we hope to give players the ability to say, “I’m never going to beat him head to head, but while he is focusing on battle, I’m going to go build up (new features) and I’ll dominate this part of the game.” We know this doesn’t help you and the others today.
All I can say is that others who have been in your position have shifted their philosophy in the game. Those that spend money can’t spend forever. In every real war you have two options to win. Either you march over your enemy with superior forces, or you create a war of attrition and ensure you have the greatest stamina. In this regard QONQR matches real warfare very well. We have seen our very big spenders spend huge sums of money for many months, then leave. Some big spenders stay or come back, but continue as a free player. In the end, the free players always win the longer term battle. Spending money is for near term bragging. Playing for free requires long term commitment when in the heavily contested areas, but leads to legendary glory. Both ways of playing work, but you need to make sure your play style matches your goals.
I started this topic stating we love our free players. Not many gaming companies can say this. For many fermium games, free players are extra overhead that the gaming companies want to encourage to leave. For them, free players require more servers, more cost, and no value. Those companies create game mechanics that make it impossible for players to advance beyond some limits without spending money. In QONQR, our free players are content creators. QONQR could not exist on only the small number of players that spend money. That would be a very boring game. Our free players provide tremendous value to the ecosystem. That makes balance very important for us.
The balance of revenue and ecosystem is one of the hardest challenges we face. For every 10 free players saying I can’t win unless I spend money, there is a paying player somewhere saying, the free players are too strong and I have to spend too much to fight off the groups. While we don’t think we are perfect, we think this is a good indication we are close.
I hope this helps. We do understand your problem, and it is something we think about and plan for on an ongoing basis.
-Scott (aka Silver)
The Pain of Growth
Two weeks ago QONQR experienced several issues, most notably problems with launches and harvests failing. I would like to sincerely apologize for the issues everyone experienced. I was playing the game hard during this time as well, trying to get a feeling for how badly the servers were failing to keep up with our traffic, and I know the frustration you experienced.
As with most things at QONQR, we have been very open with our community and share many of the technical details of QONQR. We are nerds and geeks and many of you are too, so this post will be very technical and a recognition of where we are correcting mistakes and making improvements in our system. Many companies may think such open sharing is a mistake, but we think it is what makes QONQR better.
Here is what we have learned (and a few things we already knew but want to share). If you get hung up in the nerdy parts, or just can’t read the tl;dr; that follows, just jump to the end.
It is growing more and more likely that people switching factions may have been the greatest factor in the recent issues. It is the one action a single player can take in the game that causes thousands of changes to be made to the data in seconds. Destroying a player’s nanobots and recalculating the zone totals for players who have been in the game for many months is something we can no longer do in 10-30 seconds. Therefore we had to temporarily suspend faction switching while we worked on a new off-line method to do this. Today we released a new feature at http://portal.QONQR.com, which runs behind the scenes to self-destruct your nanobots more slowly (reducing the stress on the server). The result may mean faction switches could take many minutes. We apologize if you experience increased trauma enduring the agony of millions of tiny nanobot screams, as they self-destruct over an extended period of several minutes. You can now be witness to their demise by watching your zone and bot counts slowly diminish on the new portal web page, which auto-refreshes through the self-destruction process.
We wrote the original system for Qredit collection to fill your bases with Qredits on an interval. A little program would run to put some credits in everyone’s bases every 20 minutes. It is very logical and straight forward, but a killer when it comes to performance. This is another big area that may have contributed to so many outages. We have completely rewritten the Qredit collection and harvesting code to be mathematically calculated based on your last harvest and recent zone switches. No more programs running all the time. The solution required us to approach the problem from a completely different and more complicated point of view. However, in the end, we removed hundreds of thousands of base-filling updates from the database every day, and a harvest now involves a single database update instead of twenty-two. This also means harvest either works or it doesn’t. No more issues where the Qredits are removed from your bases, but fail to make it into your accounts. I’m particularly happy with this change, and hope we can find other areas where approaching a task from a completely different angle can mean huge boots in performance and reliability.
If there is one thing that cannot fail in the game it is giving people the cubes they purchase. In the next release of the iPhone, we will implement retry in the client if the phone fails to get the Apple receipt to the server. This has only happened about 30 times in the past year, and 25 of them were in a span of a few days during the outages. We should have implemented the retry from the start, but this was something we missed until recently. We apologize. We have the fix in place and will submit the update to Apple soon. Retry already exists in Windows Phone (the implementation requires it). You can manually force a retry in Windows Phone by opening up a cube to purchase and then hitting cancel. When QONQR resumes it will check for pending transactions and attempt to fulfill them again. PayPal already does retry as well and it is automatic.
No Refunds for Apple (or Microsoft or Google)
Apple does not give us the ability to process refunds. If you click Buy Cubes, and your cubes don’t show up, please don’t hit the button 10 more times. Apple will keep charging you. Go find your receipt and send us an email so we can add your cubes. The only place we can do refunds is PayPal. If you accidentally buy cubes and buy them again because you don’t think it worked, we can actually process a refund through PayPal if you haven’t spent the cubes yet and the transaction is in the past few days. Buy your cubes online. http://portal.qonqr.com (also you should know, PayPal takes 3-5% of the transaction, Apple takes 30%)
Disable Game Features
We have learned that it is important for us to have the ability to disable game features. The old iOS offerwall and recent Apple checkout issues have shown us that we need to have the ability to turn off those features so when things go wrong, we can stop the players from getting hurt. It was very painful to watch a player continue to hit the Buy Cubes button, knowing that we had an Apple bug and they were buying more cubes than they wanted. No email on their account and messaging disabled. The same was true for switching factions. Once we had determined that switching factions was killing the server, we didn’t have a switch to disable it. Going forward we will be adding these features as a safety mechanism. We may also move some features to the web portal so we can have more control over enabling and disabling it. For any developers out there, learn our lesson and put in an emergency shut-off valve in critical parts of your app, to avoid taking the whole thing offline. We have also heard from users that they would like us to completely turn off the game in periods like the last few weeks and let them know it will be down for “X Hours” so they can go to sleep and not worry that their bots will be destroyed when they wake up. We will consider this as well, but often we don’t need scheduled down time and don’t know how long it unplanned issues will last. Just the same, we will consider that option.
We have been moving more and more things into memory cache over the past month to take load off the database, but we need to move more and faster. We are working on taking the few cache items we have incorporated already and building a comprehensive Caching engine to manage as much as we can without hitting the database.
Logging is a Double Edged Sword
Recording what is happening in the system is critical to making sure we can track down problems, and give people good customer support. But logging every connection to the server also slows everything down. We are evaluating how to reduce and/or optimize logging. Striking the right balance of information and performance is something we will need to work on over the next couple weeks.
NoLock is your Friend
For the SQL Server database developers in the crowd, get to know NoLock.
Select * from People with (NoLock) where FName = ‘John’
NoLock is basically a dirty read of the data, which means the programs don’t have to wait until everyone else is done writing before it can read the data. When counting up how many nanobots are alive in the world, we don’t really care if a few hundred thousand of the many trillion bots are killed or added between the time we start counting and the time we finish. There are many places in QONQR where an estimate is good enough, which really speeds up our processing. Locking is most important when something puts data into the database and says, “Woops, just kidding” and pulls it back (aka rollback). That isn’t really a problem for us so we can remove the lock in most cases.
Flatten that Data
Over the past 6 months we have been moving our data slowly towards a very flat and wide table schema. Third Normal Form is pretty, but when it comes down to it, a big ugly wide denormalized table will perform much faster. This restructuring also moves us closer to a NoSQL solution making a future migration, which will almost certainly be necessary, much easier. One of our big tables got a little wider in the past two weeks as we rolled another child table into its parent. The speed increases was immediately noticeable.
For the truly hardcore geeks, avoid database transaction locks whenever you can, especially in very high volume systems. Log the failures and clean them up later. We had a couple transactions in our system and they deadlocked the database too much. Do your best to architect the database so things can be updated in a single statement, and if you allow for failures across multiple statements, processes the statements in the order of importance that hurts the user the least. In QONQR, we destroy the enemy bots in a zone, before we update your XP and remove your bots on a launch. If things go wrong, as they did during the outage week, you may have gotten some free kills if we updated the zone but failed to update your player data. That would not be OK at a bank or hospital, but this is war game, and when things go a little crazy, we can shake our fist and say it was the “fog of war”. As long as we don’t mess with people’s real money, and that always works reliably, we hope people will forgive a quirk every now and again if it means it helps keep the servers from crashing.
It turns out 5am central USA is a better time to update the server than 4am USA. There are more people playing at 4am.
QONQR Players Are Amazing
This has been a very hard two weeks for us with little sleep and tons of support emails. Through all the frustration, most of the players who I replied to on support returned my message with encouragement and understanding. Thank you for supporting QONQR. Your kind words and patience is amazing and we cannot say enough how often it makes a difference to us as entrepreneurs to have such positive feedback when things are going so wrong.
Finally, I need to acknowledge the honesty of our players. When things were at its worst, and I was in panic mode, we had some angry users that didn’t get their cubes. I knew the auto-retry would kick in and they would eventually get the cubes (PayPal), but two users were particularly stressed over 5 and 10 cubes, so I added them to their accounts anyway. 15 cubes wasn’t going to fix my server problems, and I wanted to close their support requests, which were already two hours old. Within an hour after my response both replied back asking me to remove the cubes from their account, because the originals showed up and they didn’t want to take what wasn’t theirs. That is awesome! What amazing gamers! I told them to pay it forward and buy cubes for some noobs.
Thanks for being patient. We are getting better.
-Scott (aka Silver)