Dyescape - Where Adventures Begin

Initial Plan
The initial plan of the v0.2 update was to populate the forest and the mining region. Moreover, it would feature a class release and plan to fulfil promised donator features such as banking space. Quality of life fixes would be performed by software changes and more quests would be added to the aforementioned region. The level cap would have been expanded by five levels and we would call it a day. The tally would be around 20 tickets for the content team to tackle and about 50 for its software counter part. All looked well, we tackled feedback and addressed it. But one comment in a video stood out to us. The content lacked personality.

Personality
We realised that this remark was correct, but we were in alpha stages after all. The map was not populated or lively because making any MMORPG is time consuming. We wrote down all player feedback posted in #alpha and started development for v0.2. This update was originally planned to release in the first quarter of 2022. But once juggling around content something clicked and the remainder of the aforementioned remark rang true. The comment concluded that the content was not engaging and it could not be fixed by polishing out the current content. "We needed someone that could make engaging content". We dismissed this at first, but the player was right. The content was not engaging. The MQL fell flat and the combat (partly because of the mana drought) felt like a chore. After all, we did not need someone new. We could learn. What we did need was to start over.


New Beginnings
So we did. We started over. Clemens was unorganised, bland, and repetitive. It was a ghost town. The quests were spread out but required to traverse the city over and over. The MQL had no emotion and felt like a task list that needed to be checked off. Walking distance was feedback we should easily be able to solve... right? Maybe, but there was one major thing blocking most change. Clemens. After careful consideration, it had to be remade from the ground up. This felt right, it played well. But, every now and then something would block us. We would ponder how to solve a current issue and more often than not, we realised it could not be solved. It had to be redone. So we did. We started gutting the quests. Removing a few, reworking others. We were left with the MQL... The prime quest line that felt aged and dated. We rang the alarm bells, to the butcher you go. The ticket count kept rising and rising, totalling more than 300+ combined tickets.

The v0.2 release, New Beginnings, reworks Dyescape's content and systems from the ground up. We dissolved our own pride and listened. We worked through many issues. We rebuilt and restructured. For these specific reasons the release has grown upon us and felt a higher attachment to the current material we started producing. It felt fantastic compared to the initial release. We were proud of what stood before us. It is a reflection of our passion and willingness to learn. Initially this pride wanted us to show it off to the world. A marketing campaign was planned, but it did not feel right and we cancelled it. The update feels more personal. And we want you to experience the same. The community of Dyescape has supported throughout these years. With their feedback we were able to develop v0.2 to its current standard. We thank you.

Release Date
The release date will become apparent in 2 days, on YouTube.

Dear community, as some of you may have noticed, we've suffered from several infrastructure outages over the recent weeks. This caused our website, Minecraft servers and internal tooling to be unavailable. We wish to be transparent with everyone on what is going on, why these outages happened and how we plan to tackle them moving forward. The post below will involve some technical details. You can skip to the summary at the bottom for a short, simplified version.

Storage
Within the hosting, and especially the cloud hosting industry, storage comes in many different shapes, sizes, pros and cons. Some setups value simplicity, some value scalability, others value performance or integrity. When setting up our infrastructure, we had to choose between these options and decided to go with a storage solution that had integrity and was scalable. As such, we are running an internal block storage solution called Longhorn, with volume replication across all of our servers in geographically different data centers. This means that all machines are constantly replicating each other's volumes. This is great for keeping data integer and fault-tolerant, as well as being able to quickly move software deployments such as databases to new machines without having to wait for file transfers.

That sounds great, so why bring it up? Everything works great as long as everything remains connected. As soon as these nodes lose connection with each other, the machine entirely will be marked as unhealthy. This is okay because then we can deploy software on different machines and quickly recover. The problem, however, is that the node becoming unhealthy, results in the storage solution shutting itself down on this machine. This means the replica status is lost and no health can be communicated anymore. As a result, the data replica on the unhealthy node is immediately considered degraded. This doesn't mean data loss, nor does it necessarily cause any damage, but it does cause the storage solution to rebuild the entire replica on that particular node from a node that was healthy. We have got data replication across our American and European servers as some of you may know. Repairing a volume across the Atlantic ocean comes with a big drop in network capacity. To save budget and reserve extra capacity for alpha and beta development, excessive hardware has been removed and thus we cannot have replicas for these volumes be reliably placed across multiple machines in the same continent, and we're forced to replicate across continents. A full replication rebuild thus causes time. Doing this with dozens of volumes puts considerable strain on our connection between the two continents. In some cases, volumes cannot be attached to workloads before they are fully healthy. This is what was the cause of lengthy outages the past few weeks. Why machines lost connection in the first place will be disclosed later in this post.

The above wouldn't be a concern if a short connection drop wouldn't immediately kill the replica instance on the other machine that loses connection. The developers of said storage solution agree, and a bug/feature ticket has been created on their Github as a result. We were aware of this issue, and have been tracking it for some time. Unfortunately, the priority wasn't high enough initially and a resolution has been pushed back to the next major update. Since the outage last weekend, we have tweaked several settings in this storage solution and are monitoring the results. We have an alternative storage plan drawn out in case this solution remains problematic.

Nodes losing connection
As discussed above, issues occur when nodes lose connection. This by itself has several causes, and we'll be sharing three of those causes here.

First, a worldwide OVH outage on the 13th of October caused a connection loss between all of our servers. Exact details are missing, but the general understanding is that OVH failed a large scale BGP routine update, causing all routes to disappear. The same happened with the worldwide Facebook outage not long before that. In a case like this, we are powerless, and with outages like this, the Minecraft industry as a whole takes a beating. The outage was solved by OVH within a reasonable time, but it caused our storage solution to panic and all volume replications had to be rebuilt as a result.

Secondly, we only got an understanding of the magnitude of the bug (or at least, limitation) of our storage solution somewhat recently. We've performed several rolling updates on our orchestration platform in production to keep everything up to date. This can normally be done without any downtime, but because of the storage limitation, does cause downtime. We weren't aware of this until it happened a few times.

Thirdly, the most recent one was a problem with our CNI (Container Network Interface) plugin. We are still unsure of the exact cause, but a sudden crash occurred on the plugin on one of our machines. This caused a connectivity loss between processes on that particular machine. Rebooting & reinstalling part of the systems did not help. We could not trace the cause but eventually tackled the issue by updating the kernel and operating system on the machine. We suspect that an unknown, undiscovered bug occurred as a result of an edge cause of our specific kernel, operating system and CNI plugin versions.

Summary
In short, a combination of sudden connectivity drops and a current limitation of an internal storage solution caused several unexpected, lengthy outages. Moving forward, we are adjusting settings appropriately to get the desired behavior of our systems and are continuously monitoring the results. An alternative plan for storage is ready in case this setup remains problematic.

We sincerely apologize for the outages. We are still actively learning from all of the feedback we've gotten since launch; on a functional and technical level. Thank you for your continued understanding and patience.

After a tremendously long development period, Dyescape has finally been able to put out a first release; v0.1.0. Those who were active in the Discord or the Twitch livestream yesterday will have already seen that we ran into a few technical struggles. This thread serves as transparency towards the community to explain what happened, why it happened, how it could happen and what has been done to address it. On a positive note, we'll also list a few technical things that did go well.

Database cluster
In a project like this, being dynamic with content changes is crucial. Software and content are two completely separate complexities, and in order to ensure that it doesn't become a single, even larger complexity, we've made everything configurable. From items, to skills, creatures, quests, random encounters; everything is configurable. In order to facilitate for this, a JSON document based file storage system was set up. For the past years during development, this has been done through flatfile & SFTP to allow the content to quickly make changes, test these and commit them to version control. A few months ago, a MongoDB cluster was set up to make it production ready.

Having set up the database cluster, everything was operational. We could import our content, we could play the game, save characters, swap to a different server, and everything would be there. However, before launch, we still had a few important content & software changes to process. While doing so, something in the database got upset, causing cluster sharding initialisation to fail and the cluster to become unusable. This is the first major technical downside we ran into, which is what caused the initial release to be two hours late. Thankfully, the team quickly jumped to support and we managed to solve the issue.

GEO load balancing
GEO load balancing is a technical setup that automatically routes users to the nearest server. This is setup by our Anti-DDoS & Load Balancing provider. From what we could see, this load balancing worked for most users. North American users see on average roughly 40 to 60ms ping from earlier playtests, and west European users get a ping of around 15 to 25ms. This GEO load balancing is setup for two main reasons; to give users the best possible connection and reduce lag, and to limit cross-continent bandwidth usage.

However, due to a misconfiguration on the proxy service discovery side, players were often sent to a fallback server in a different region. They would connected to a North American proxy, but a European fallback server for instance. In some unlucky cases, some people didn't get the expected GEO load balancing on the proxy to begin with. This caused for example American users to connect to a European proxy, and then to an American fallback server; doubling the already bad latency as a result. However, this was only for a handfull of people and was likely caused by some fallback servers crashing (more on this later).

In order to fix a few latency issues some players were having, two new domains have been created. These domains are eu.dyescape.com and na.dyescape.com created. In order to keep things as stable as possible for the time being, the proxy & fallback setup has fully been split into two separate networks now, so there's no chance of ever being routed to a fallback server of the wrong region. The play.dyescape.com domain should still provide accurate GEO load balancing. If not, please contact the team.

Cross-continent bandwidth
We have explicitly set up the infrastructure to have considerably strong network bandwidth. At our hosting provider, a large private network consisting of 4 solid dedicated servers with 8 physical cores, 64GB memory each, and a 2gbit connection. Despite initial thought yesterday, we currently see no signs of bandwidth coming short. The timeout errors we were seeing yesterday was actually caused by an issue on the software. The resulting issues for players however made us think it was a network related issue.

Crashing fallback servers & timeouts
While the Dukes were playing, we were notified of numerous timeout & crashing issues. After some investigation, it seemed to be caused by a recent software change in our interactive chat plugin. A code issue caused an infinite loop, exhausting the CPU capacity and killing the instance. This issue was fixed at around 02:15 AM our time. Afterwards, the Dukes could continue playing.

Remaining issues & alpha queue
The question which we see posted in Discord extremely often since the release, is why the queue is not processing. There's a good reason for this; although we've fixed any infrastructure & fatal software related issue we see, there's still a few in-game issues causing quests to become stuck, blocking progression. These issues are scheduled for a v0.1.1 patch release, which is planned to go live in the upcoming days. The team is working round the clock to get this patch out.

Because these remaining issues prevent gameplay progression, we've decided not to progress the queue until the v0.1.1 goes live. We deemed the best decision would be to have only Dukes test the game. It's a small group of people that can help us efficiently identify critical issues.

Positive notes
Down to some positive comments, because despite the technical issues, there are also multiple compliments worth mentioning. I'll go over some of these below;

Gameplay feedback (from Dukes); while currently only Dukes are able to play, we've received very positive feedback from them. The game is smooth, skill usage is a bliss, content is interesting & understandable, and there's multiple qualify of life features that are very well appreciated. After having fixed the regional connections & freezes, ping seems to be good, and the average milliseconds per tick seems to be healthy. Once the in-game issues are fixed in the v0.1.1 launch, we can likely start processing the queue.


Conclusion & thank you
We want to close this off with a massive thank you. Albeit rough around the edge; Dyescape has launched. It has taken us well over 4 and a half years to get to this point. It has been an incredible ride to reach this state. We've had our ups, we've had our downs. We've seen team members come and go, we've seen content be revamped, software be overhauled and we've seen a community grow.

Despite the messy infrastructure launch, in all of our years of playing Minecraft, we have not seen a more considerate, friendly & heartwarming community. The support is incredible. We will continue to work hard getting the fixes out and to have everyone be able to join. Hope to see everyone on the server in v0.1.1!

Alpha
While some still cannot believe it, we are actually launching on the 7th of August! Note that there are only limited alpha slots available. Sign up before it's too late. And join the discord to receive all kinds of unique sneak-peeks!

Alpha Release Date
The ancient old question; when will Dyescape release? The wait is soon over. Please read the post below and sit tight for just a little longer!



Alpha Teaser
While some still cannot believe it, we are actually launching! On the 17th of July, we are releasing an alpha teaser. Note that there are only limited alpha slots available. Sign up before it's too late.

And after this Alpha Teaser, we will be hosting a QnA among other things in the discord! Questions for the QnA can be submitted in the discord. The full schedule of Saturday the 17th of July is:


Class Descriptions
We have revealed some new imagery in regards to classes! These images and descriptions can be found on the wiki thanks to some community members! If you want to always stay up to date on any development progress or sneak-peeks join the discord!



Progress Percentage
We have introduced progress percentages a few months back. This percentage shows the ratio of open tickets vs closed tickets. These tickets are not weighted, which means that the percentage could quickly go up or stay stagnant for days on end. Even worse, it can go down if we find new issues. The current percentages are:

• Content = 77%
• Development = 93%

Release Date Contest
Lastly, there is a small contest going on. Predict the release date! If you have the closest guess to the actual release date, you will receive the Noble rank! Fill in your guess in the discord!

We hope to see everyone on the 17th of July! Thanks, everyone for years of support!