Why the Heck are all the Games late? The trouble with estimation in Web3 gaming
If you have been following the Web3 gaming space for a while, you may have noticed a concerning trend, almost every game has been late compared to their initial estimates. This article aims to explain why this is so common, and explore if this is a problem only in the Web3 space, or does it happen in traditional games also. The article is the first that expands on the topics in my previous article on the ‘Threats and risks to NFT and Crypto games’. A future article will look at solutions teams can utilize to minimize both the probability, and impact of unrealistic estimates, and also advice for investors.
Table of Contents
Examples of delayed games in Web2 and Web3
Are Estimation Issues and Delays Also Common in Web2 Games?
Reasons for poor estimates/delivery
Project Management Methodologies
Examples of delayed games in Web2 and Web3
This table is taken from a previous article where estimates were covered. It should be noted that Axie Infinity Land Alpha has since been released on the 28th December 2022.
It should be noted that the above table is not here to shame the above projects, but to show how easy it is to underestimate complex tasks, and that even skilled and dedicated developers can still fall short of their initial estimates.
Are Estimation Issues and Delays Also Common in Web2 Games?
As the table above shows, there are Web2 games that also suffer from estimate issues and delays. A common phrase that players/holders will ask is ‘Wen launch?”, and a common reply by the team is “Soon™”. Soon™ is an old phrase first said by the World of Warcraft (a popular MMORPG game) developers, in reference to when the new updates will be out. Crowd funded games (through Kickstarter etc) are probably the closest parallel to Web3 games in that they are often funded upfront by the community, can have inexperienced developers, and list the launch milestones upfront. Crowdfunded games are notoriously late, and many never launch. Even large gaming studios can see massive delays, CyberPunk 2077 first was announced in 2012, and full development started in 2016. Its early 2020 release date was further delayed 3 times before it was released late 2020, where it was heavily criticized for being riddled with bugs, in reality it probably needed a 4th delay. Therefore estimate issues and delays are not unique to Web3 games.
Reasons for poor estimates/delivery
The following is a compilation of reasons why Web3 games are often underestimated and delayed.
Scope Creep
Once development is underway, it can be very tempting to add in new features that were not initially thought of at the start of the project. Unfortunately, the initial estimates were calculated based on the features that were planned at the time, and therefore adding new features will usually add duration to the delivery of the game. Careful consideration should be given to whether the added time required for the feature is worth the delay in the game's release, or if it could be included in a future update instead. Alternatively, it may be necessary to consider removing another feature to make room for the new one." For example, the Illuvium team's initial estimates did not take into account the Illivum Zero land game, which was conceived of well into the development cycle.
Inexperienced Teams
The unfortunate truth is that many Web3 gaming teams lack experience in game development. Many teams jumped onto the Metaverse and Play-2-Earn(P2E) bandwagon, either directly, or as a pivot after making a PFP NFT project. Estimation is already difficult enough for experienced developers, let alone new teams. While the Web3 gaming space does have some experienced game developers, it's important to note that there are also many new or amateur developers and founders who are new to the industry. This may lead to them having unrealistic expectations and over-promising both in scope and time-frame.
Adding more resources to a late project
When a game is behind schedule, it may be tempting to add more people to the development team in an attempt to speed up progress. However, this approach is often ineffective and can even make the situation worse. This phenomenon is known as "Brookes Law," named after a programming book called "The Mythical Man Month." Although the term "man" in the title is now outdated, the law remains valid. Brookes Law states that "adding manpower to a late software project makes it later." This may seem counter-intuitive, as adding more resources to a project should theoretically speed it up. However, there are several factors that contribute to this phenomenon, including:
The project can only be divided into a limited number of individual tasks.
Training new workers takes time and can decrease the productivity of current workers who have to devote time to training and correcting errors. In some cases, a new worker's productivity may be negative overall due to the time and resources required to bring them up to speed.
As the number of people and sub-teams on a project increases, the number of communication lines that need to be established grows exponentially. For example, while there are only 6 connections between 4 people, the number of connections between 6 people increases to 15.
To further demonstrate this point, let's consider an extreme example of a large project that would take one developer one year to complete. Even if 364 additional developers were brought on board, the project would not be finished in just one day.
Ninety- ninety Rule
The 90-90 rule states that 90 percent of the code takes 90% of the effort to develop, while the remaining 10% accounts for the other 90% of development time. This obviously adds up to 180%, and suggests that not only are estimates often wrong, but the final part of the project tends to be the most difficult and time-consuming. According to the 90-90 rule, when a team thinks they are 90% complete, they may be closer to only 50% complete in reality. A team may publicly announce they are 90% complete, leading to disappointment when it takes longer than expected. There are several reasons why the final stages of a project may take longer than expected. For example, additional tasks that were not anticipated may be necessary to complete the code and get it live. Installing the code in the live production environment may introduce new bugs that were not present in the development environment, and integrating the code from multiple developers may require extensive rework. All of these factors can contribute to the difficulty and duration of the final stages of a project.
New Technology
Accurate estimates depend on familiarity and experience with the tasks at hand. As blockchain, smart contracts, and NFTs are relatively new technologies, the average level of experience among developers in this field is lower than in more established industries like web development. This can make it harder to make accurate estimates for projects in the Web3 space. Even for game developers with decades of experience, incorporating Web3 technology into a project can present a steep learning curve. Additionally, the immutability of the blockchain (there is no easy way to rollback) makes it more difficult to resolve and recover from code bugs. Add in the potential for significant loss of financial assets, and the effort and care needed to develop and audit the code will often take far longer than expected. Third Party auditing is often recommended, which is in high demand, often requiring significant lead time to book in, and any issues found will result in re-write and potentially another round of audits.
Time Thieves
The 5 Time Thieves are outlined in the book "Making Work Visible," which advocates for agile project management practices. The Time Thieves are:
1. Too much Work In Progress (WIP):
Starting new tasks can be exciting and enjoyable, but this can also lead to teams starting too many things without completing them. Picking up an abandoned task again requires significant mental effort, as the team must remember where they left off, re-familiarize themselves with the code, and resume work. Developers may also leave a project before a feature is complete, forcing new developers to start from scratch. Additionally, individuals may be reluctant to disappoint others and say yes to every request, leading to an unrealistic number of commitments. Context switching between multiple tasks significantly reduces output. There is also a relationship between Work in Progress and throughput, where the average cycle time is a ratio between these two factors, and the more work in progress, the more wasted time there will be.
2. Unknown Dependencies
Having too many dependencies can significantly increase the chance of a release being delayed. Unknown dependencies are those which have not been planned for, but end up being discovered too late, and result in delays. For example:
People are not available when needed to make decisions or provide assistance.
A change in plans or code results in something else breaking and requiring attention.
3. Unplanned Work
Unplanned work refers to tasks that arise unexpectedly and require attention from the team, but were not included in the original estimates. An example of unplanned work might be when the team is working on a new release and the current game suddenly breaks due to a full hard drive caused by log files. The developers would then have to drop everything to restore service. Another example of unplanned work might be when the game unexpectedly becomes extremely popular and the servers become overloaded, requiring the team to identify and address bottlenecks to scale the game. Alternatively, the team may need to quickly fix an infinite gold creation bug to prevent further damage and come up with a plan for resolving the issue. Unplanned work can lead to an excessive amount of work in progress, reduce the predictability of releases, and increase costs.
4. Conflicting Priorities
Developers often have multiple tasks that they could be working on at any given time, and it can be challenging to determine the best course of action on a daily basis. Should they work on the task that is most enjoyable, the one that is easiest, the one that the project manager is requesting for an investor demo, or the one that is boring but will help launch the final game faster? There are also tasks that may not seem urgent at the time but can later become emergencies, such as improving the security of a server that stores customer details. This issue can be compounded by too much work in progress, leading to conflicting priorities. For example, the Ledger marketing database hack highlighted the importance of secure servers for storing customer information.
5. Neglected work
There are some types of work that are not urgent and may struggle to be prioritized by the team and management. Examples of this type of work might include decommissioning old servers, refactoring the code to improve its structure and readability, or increasing the capacity and performance of servers. "The Axie Infinity Ronin bridge hack that occurred on March 23, 2022 was caused by neglected work. Hackers from North Korea were able to access a server that should have been decommissioned and had access removed in November 2021. However, due to neglect and possibly being seen as non-critical infrastructure, the hack resulted in a loss of over 650 million dollars' worth of assets.
Lack of Automated testing
Automated testing is the process of using test code to verify the functionality of the main code. For example, a test might be written to check the login process by entering a valid username and password and verifying that the code works as expected. Automated tests can also include negative tests, where an incorrect password is entered to ensure that the account cannot be accessed. These automated tests are grouped into a ‘test suite’ and can be run regularly to ensure that all current and previously written code is still functioning as intended. Automated testing helps prevent regression bugs, where new code breaks previously working code, which can be difficult to detect without regular testing. In the gaming industry, for example, an automated test might be used to ensure that the initial quest in the game still works after the second quest has been added. The second quest could also be tested to ensure it functions correctly for both new players and those who completed the first quest before the update.
Lack of Sufficient Environments
Environments refer to the combination of software and hardware required to run a game. At the beginning of a project, a team may only have one shared development environment, particularly if the team is small, the code complexity is low, and there are only a few early tests being conducted. However, as the project progresses, teams will typically need multiple environments, including:
Development (Dev): This is where the ongoing code for the current work is being created and tested.
Test: This is where the current code is sent to for testing, often by testers who were not involved in the development.
Pre-Production (Pre-Prod): This environment is designed to replicate the production (live) environment as closely as possible and is used as a final check to ensure that nothing breaks between test and production.
Production (Prod): This is the live environment where the game is currently running and being used by players.
Local Dev: Individual developers may also have local dev environments on their own machines that they use to test out their code, although these environments may not have all the features of the other environments depending on the architecture.
Unfortunately, many teams do not have all of these environments and may only have a development and production environment. The lack of these environments can lead to new bugs being found in production that were not present in development. It can also be helpful to have multiple Dev or Test environments so that different teams do not have to schedule time in the same environment and risk conflicts. Automated testing can further help by only allowing code to be deployed to the next environment if it passes all tests.
Managers ask the wrong questions
When a manager asks a developer how long it will take to implement a feature, the developer may feel pressure to provide a single estimated time-frame, such as 2 days, 2 weeks, or 2 months. Unfortunately, this is the wrong question to ask because it relies on a single estimate. Even if the developer tries to break down the feature into smaller parts and compare it to previous work, they are still only able to provide a single date. A better question might be "Can you give me an estimated time range in which you think there is a 90% chance you can deliver the feature?" This allows the developer to provide a more accurate estimate of 2-5 weeks, for example, which may be a larger range but reflects a higher level of accuracy.
Investor Pressure
Before investing in a project, investors may require an estimated delivery date for the game. Without this information, investors may be hesitant to invest in the project. However, relying on a fixed delivery date can provide a false sense of security. Instead, it is more advisable for the team to use dynamic estimations and transparent progress reporting to ensure accurate and reliable project management.
Burnout
Game developers face pressure to meet deadlines and create high-quality products, which can be stressful. Pressure from investors and the desire for financial success can also increase the pressure to work hard and launch faster. Developers may also have personal stakes in the game's success through NFTs or tokens. The demanding work in the game industry has even led to hospitalizations for some developers.
Project Management Methodologies
Waterfall Model
The waterfall Model was created in 1976, and follows 6 phases:
Requirements gathering: all of the requirements of the system will be elicited by business people, and documented in detail, and approved by the stakeholders.
Analysis: The business rules, data, and models will be documented and analyzed.
Design: The systems software architecture will be designed.
Development: The development of the system will take place based on the Requirements and Design.
Testing: The system will now be tested, and fixed before going live.
Operations: The system will go live, data will be migrated, and ongoing support and maintenance will be provided.
The waterfall method has faced criticism because projects often experience delays and go over budget. Despite this, it is still used in large companies because management often requires a business case with a set ROI (Return on Interest) projection and time-frame.
Agile Methodology
The Agile Methodology, which was first described in the 'Agile Manifesto' in 2000, is a project management approach that involves short, 2-week "sprints" instead of distinct phases with fixed time-frames. While it has several variations like Agile, Scrum, and Kanban, the general principle is similar. At the start of the project, high-level deliverables are mapped out and often summarized on sticky notes placed on a physical (or digital for remote teams) wall. The team will then assign a relative effort value to each feature listed on the sticky notes. For example, creating the login form might be a 1, while creating the home page might be a 5. ". During an Agile sprint, the Product Owner decides which features to prioritize for the next 2 weeks. These are moved into the "In Progress" column and team members are assigned tasks. The goal is to design, develop, and test the features within the sprint and present them to the Product Owner at the end. Over time, the team can track how many story points they can deliver in each 2-week sprint and chart this on a burndown chart to estimate the finish date of the project. However, this estimate may change slightly every 2 weeks.
Agile Waterfalls
So now we have looked at the waterfall and agile methodologies of project management, how does it apply to Web3 games? I suspect that many teams are running what appears to be agile methodologies, they will have ‘scrum’ walls with ‘Backlog’, ‘In Progress’, ‘Testing’, and ‘Done’, they will run in 2 week sprints, and they may even have product managers. However, the issue is, the investors and community are demanding whitepapers and road-maps. This leads the team to publishing road-maps up front, with monthly or quarterly deadlines. Unfortunately, this is the waterfall method, with the developers and team having to commit to distant deadlines on work that has not been started. The “Agile Waterfall” phrase refers to the team on a day to day basis using the agile methodology, but they are working towards pre-set deadlines as per the waterfall method. The whole point of the agile method is to accept that you cannot put a finish date of a complex task, and therefore the project should be tracked against tasks that are measured on their relative effort, and a burndown chart used to estimate the current (and dynamic) trajectory to completion.
Conclusion
Web3 gaming projects face a range of challenges that can impact the accuracy of estimates, such as scope creep, inexperienced teams, adding resources too late, too much work in progress, unplanned work, conflicting priorities, neglected work, new technology, lack of automated testing/environments, and investor pressures. These factors can lead to delays and budget overruns, even when agile methodologies are used. In the future, it will be important for teams to take steps to mitigate the risk of unrealistic estimates and for investors to be aware of these challenges when considering investments in Web3 gaming projects. A future article will discuss what teams can do to minimize the chance and impact of unrealistic estimates, and also contain advice for investors in the space.
Great write-up, will be featuring it in this week's newsletter!