Decoding Engineering Effectiveness: Twitter's Journey

This blog post explores the journey of Twitter's Engineering Effectiveness group, their role in scaling software, and the challenges they face.
Create an image that depicts an engineer at a desk, surrounded by symbols representing software scal

On this page

The content provides an overview of the role of Twitter’s Engineering Effectiveness group and the journey of Twitter's software scaling. The Engineering Effectiveness group is responsible for providing tools and processes that help Twitter engineers work effectively. The author discusses the growth of software companies and the challenges of scaling both software and organizations. Twitter's software scaling journey, from its first line of code in 2006 to its current state, is detailed. The shift to a service-oriented architecture, the formation and challenges of maintaining two monorepos, and the efforts to consolidate them are highlighted. The author also addresses the challenges in measuring engineering effectiveness and the aim to improve quality, speed, and joy in engineering practices. The importance of time savings, maintaining a flow state for efficiency, and the hindrance of technical debt are discussed.

Let a 1,000 flowers bloom. Then rip 999 of them out by the roots.

How does it apply to you?

This knowledge can be applied to any software development environment to understand the challenges of scaling, the importance of effective engineering, and how to tackle technical debt. It also provides insight into Twitter's engineering practices, which could be useful for those interested in the tech industry or aspiring to work in it.

Applied Learning to Developer Enablement

Application to Software Development

The content discusses the journey of Twitter's engineering team in scaling their software and organization, which provides valuable insights for software development. Key takeaways include the importance of effective tools and processes, the transition from monolithic architectures to service-oriented architectures, the challenges of managing large codebases in monorepos, and the need for dedicated resources to maintain these systems. The content also highlights the importance of reducing technical debt and maintaining a 'flow state' for engineers to enhance productivity.

Developer Checklist

Engineering Effectiveness

Understand the Importance of Consensus in Tool Usage: Recognize the need for consensus when choosing tools and processes, and the potential complications that can arise from lack of agreement.
Learn About Engineering Effectiveness Metrics: Understand the challenges of measuring engineering effectiveness and the need for clear, quantifiable metrics.
Understand the Triple Aim of Engineering Effectiveness: Learn about the interconnection of quality, speed, and joy in engineering practices, and how improvements in one area can enhance the others.

Productivity and Technical Debt

Recognize the Importance of Time Savings and Productivity: Understand how improving tools and processes to save time can enhance engineering effectiveness.
Understand the Importance of Maintaining Flow State: Learn about the concept of flow state and how interruptions can negatively impact productivity.
Recognize the Impact of Technical Debt: Understand the concept of technical debt and how it can hinder engineering effectiveness.
Manage Technical Debt: Technical debt can accumulate over time, becoming more difficult to manage as the engineering team grows. Clearing technical debt can lead to significant gains in effectiveness.
Establish Dedicated Teams: Engineering Effectiveness teams often deal with tech debt, especially when it accumulates in tooling. Establish a dedicated team to manage and reduce tech debt.

Tools and Practices

Promote Good Practices: Engineering Effectiveness teams can help by promoting good engineering practices and discouraging bad ones. This could involve code reviews, testing, writing design docs, and more.
Provide Good Tools: Good tools can have a positive psychological impact on engineers, boosting overall effectiveness. They can make daily work more pleasurable and boost confidence.
Build a Model for Total Effectiveness: To understand the total effectiveness of an engineering organization, build a model that considers the quality of software produced, the speed of production, and the joy experienced by engineers.
Set Parameters for the Model: Setting the values of 's' and 'b' in the Engineering Effectiveness model can be challenging. Adjust these values according to specific circumstances.
Visualize the Model with Graphs: Graphs can be used to visualize the total effectiveness depending on how many engineers are devoted to effective work.
Understand the Implications of the Model: As engineering teams grow, the optimal number of Engineering Effectiveness engineers increases faster than the total number of engineers, and the productivity payoff is disproportionately large.

Maintenance and Improvement

Weed the Garden: Decide which tools or processes are going to be nurtured and which ones are considered 'weeds'. Balance individual team preferences with overall productivity gains.
Invest in Tools and Processes: Choose a set of tools and processes and support them wholeheartedly. This may require more investment than anticipated, but the focus should be on making the chosen tools and processes as efficient as possible.
Deal with Consequences of Falling Behind: If an organization falls behind, it may need to invest more to regain control. If official procedures are not efficient, teams may resort to finding their own ways to complete their tasks.
Maintain a Well-Tended 'Garden': Once the tools and processes are efficient and effective, they will be used willingly. If a team deviates from the norm, it should be seen as an opportunity to learn and improve.
Learn from Others' Experiences: Other companies' experiences, such as Twitter's, can provide valuable insights for managing engineering effectiveness.
Focus on Quality: Prioritize quality in every aspect of software development. This includes code quality, user interface, and overall user experience.
Increase Speed: Implement strategies and tools that can help speed up the development process. This might include automating certain tasks or using more efficient coding practices.
Create Joy in Work: Ensure that the development environment is positive and enjoyable. This can lead to more productive and creative work.
Maintain and Tend to the 'Garden' of Processes and Tools: Regularly review and refine the processes and tools used in development. This 'garden' requires ongoing attention to stay effective and efficient.

Key Points

Introduction to Engineering Effectiveness - The Engineering Effectiveness group at Twitter is responsible for providing tools and systems that assist Twitter engineers in their work. These include build tools, continuous integration systems, and source control.

Scaling Software and Organizations - The author highlights the industry’s understanding of scaling software and organizations. However, there is a lack of knowledge on scaling the intersection of engineering and human organization.

Twitter’s Software Scaling Journey - Twitter’s software scaling journey includes its start in 2006, the creation of the Monorail, and the introduction of Scala in 2008. The company also created a new repo called Science in 2009.

The Shift to Service Oriented Architecture - Twitter shifted to a JVM based service oriented architecture after the 2010 World Cup to manage tweet volume spikes. This led to the creation of a new Scala RPC library, Finagle, and the training of Ruby developers in Scala.

Successful Scaling of Software - Despite diverse engineering styles and practices, Twitter successfully scaled its software. By the 2014 World Cup, Twitter was able to handle a significant tweet volume without issues.

Formation of Monorepos at Twitter - Twitter's codebase has grown into two monorepos, Science and Birdcage. The decision to create two separate monorepos was influenced by technical constraints and differing views between Java and Scala proponents.

Challenges of Maintaining Two Monorepos - Maintaining two monorepos led to interdependencies and a time-consuming update process. The lack of a dedicated team to maintain these monorepos caused inefficiencies.

Converting Birdcage to Use Pants - Birdcage was converted to use Pants, a build system. However, due to lack of resources, the versions of Pants were not entirely reliable, leading to different strategies for making it work.

Inconsistencies in Thrift Usage - There were disagreements on Thrift usage, including which Thrift compiler to use and whether to publish Thrift IDL jars or already compiled artifacts. This lack of consensus increased complexity.

Efforts to Consolidate Monorepos - An initiative was launched in 2013 to consolidate Science and Birdcage into a single monorepo. The Engineering Effectiveness group was later created to expand the scope of work and support for Twitter engineers.

Challenges in Measuring Engineering Effectiveness - Measuring engineering effectiveness is challenging due to the lack of clear productivity metrics. Engineer productivity is subjective and harder to measure and improve compared to back-end software performance.

Engineering Effectiveness and the Triple Aim - Engineering Effectiveness aims to improve quality, speed, and joy in Twitter's engineering practices. Improvement in one area can lead to enhancements in the others.

Time Savings and Productivity - Time savings can enhance engineering effectiveness. This can be achieved by improving tools and processes, and by minimizing time loss occurrences, such as debugging confusing error messages.

Maintaining Flow State for Efficiency - Helping engineers maintain a flow state, where they can concentrate and solve problems efficiently, can boost effectiveness. Providing tools that support uninterrupted work can save time and increase joy.

Technical Debt as a Hindrance - Technical debt, a consequence of prioritizing speedy delivery over perfect code, can significantly hamper engineering effectiveness. It can slow down processes, cause unhappiness, interrupt flow, and create a sense of drift.

Accumulation and Management of Technical Debt - Technical debt, like financial debt, can accumulate over time and become more difficult to manage as an engineering team grows. Clearing this debt can lead to significant gains in effectiveness. Engineering Effectiveness teams often tackle this debt, especially in tooling, and can help other teams manage their own tech debt.

Promotion of Good Engineering Practices - Engineering Effectiveness teams can promote good engineering practices and discourage bad ones through activities like code reviews, testing, and writing design docs. To do this effectively, these teams need strong senior engineers who can make impactful decisions.

Psychological Impact of Good Tools - Providing good tools to engineers has a positive psychological impact, making daily work more enjoyable and boosting overall effectiveness. Conversely, using poorly engineered tools can negatively affect engineers' confidence in their own abilities and the company's competence.

Weeding the Garden - As an engineering organization grows, the benefits of making all engineers slightly more productive can outweigh the slight gains that one team might get from doing things their own way. Deciding which tools or processes to nurture and which ones are considered 'weeds' is important.

Consistency and Chaos in Engineering - In chaotic situations, engineers with strong technical judgment are appreciated for making decisions and streamlining processes. Improvements and visible results can build trust within the team, paving the way for further decisions and changes.

Investment in Tools and Processes - Choosing a set of tools and processes and supporting them wholeheartedly is crucial. This may require more investment than anticipated, but the focus should be on efficiency. Involving senior engineers early on can aid in making beneficial choices and securing buy-in.

Consequences of Falling Behind - If an organization falls behind, it may need to invest more to regain control. If official procedures are not efficient, teams may resort to finding their own ways to complete their tasks.

Maintaining a Well-Tended 'Garden' - Once the tools and processes are efficient and effective, they will be used willingly. Deviations from the norm should be seen as opportunities to learn and improve. A well-maintained 'garden' allows for new 'flowers' (ideas or methods) to be nurtured and incorporated if beneficial.

Quality, Speed, and Joy in Work - The ultimate goal is to provide quality, speed, and joy in work. This is achieved while maintaining and tending to the 'garden' of processes and tools.

Subscribe to SmartJots newsletter and stay updated.

Don't miss anything. Get all the latest posts delivered straight to your inbox. It's free!
Great! Check your inbox and click the link to confirm your subscription.
Error! Please enter a valid email address!