It's a sign of seniority and clarity.

Too many engineers and product teams think that more features equals more value. That's a costly mistake. The best product leaders and engineers know that strategic scope reduction is a deliberate, mature choice driven by a deep understanding of business objectives and user needs.

Great products aren't built by stacking features but by eliminating distractions and zeroing in on what truly matters. Scope reduction done right delivers impact faster without diluting the value.

This article will show you how to build this skill and confidently cut scope, delivering better outcomes for your users and business.

Scope Reduction as a Strength

There's a strange notion in engineering:

❌ Reducing scope signals failure or an inability to deliver.

Let's dismantle that immediately.

Strategically cutting features doesn't indicate weakness but demonstrates business awareness and clear decision-making. When you're willing to trim features that aren't critical, you're showing that you prioritize real impact over superficial complexity.

Leaders and stakeholders respect engineers who can confidently say:

✅ This isn't essential. Let's focus on what is.

In fact, scope reduction is often the best step to creating leaner, more focused, and more effective products.

Share

Identify Your Core User Journey

Before cutting anything, clearly define what value you're actually trying to deliver and for whom.

Too often, product teams lose sight of the core user journey, stuffing their products with secondary features that distract from their central purpose—the features users actually pay for and which they expect to work flawlessly.

The first step in smart scope reduction is mapping out a minimal user journey focused purely on outcomes.

Forget about fluff and superficial what-ifs. Think instead about the critical steps users must take to reach their primary goal. Every interaction or feature should clearly connect to delivering that value.

Actionable Tip

Create a simple flowchart or diagram visualizing your product’s core user journey. Any step or feature outside this essential path is a candidate for removal.

Consistently return to the question:

💡 Does this step directly help the user achieve their core goal?

Aim to keep the journey lean, intuitive, and impactful.

Feature Split

Once you've identified your core user journey, the next challenge is handling complexity. Large features are risky. They slow you down, increase uncertainty, and often lead to feature creep during implementation, diluting the core value you aim to deliver.

Such monolithic features need to be split into smaller, manageable pieces that still deliver clear, standalone value. The goal isn't to deliver everything at once but to provide meaningful increments that still feel complete.

💡 What's the smallest increment that still solves a meaningful problem for our user?

Everything beyond this minimum increment is a candidate for a feature split.

Impact vs. Effort Matrix

After splitting your features, you need a clear way to prioritize these smaller chunks. The Impact vs. Effort matrix is the perfect tool: simple and fast.

Plot each feature based on two straightforward dimensions:

• X-axis: Required effort

  • Consider engineering and product complexity, dependencies, rollout strategy, and risks.

• Y-axis: Anticipated impact

  • How much user value, potential revenue, or competitive advantage it brings.

Practical Tip

Don’t get too overexcited when estimating.

• Engineers are often great at being overly optimistic when it comes to effort estimates (estimating too low).

• Product managers are often great at being overly optimistic when it comes to impact estimates (estimating too high).

Summary

Scope reduction is about clarity and discipline.

Successful product teams know that strategically cutting unnecessary features by

• clearly defining the core user journey

• breaking complex deliverables into smaller independent chunks

• prioritizing them objectively

demonstrates maturity and streamlines product delivery.

I’d like to hear about your experience. What methods do you use for scope reduction?

📖 Read Next

Discover more from the Product Engineering track:

If you’re looking for a space where you can learn more about software engineering, leadership, and the creator economy, with Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve created the Engineering & Leadership discord community:

Join

📣 Recommended Reading

• The 13 software engineering laws by in

• What engineers get wrong about communication by in

• Starting Your Career with a Remote Job Can Hurt You by in

Long-lived branches pile up, and merge conflicts become a nightmare. Deployments are high-stakes events, and when something goes wrong, rolling back is a mess.

This isn’t just an engineering problem. It’s a business problem. Slow releases mean slow feedback, which means wasted effort and missed opportunities.

Shipping software should be like breathing—constant, natural, low-drama.

With continuous delivery being de facto mainstream, this goal is theoretically within the reach of any team or organization. Smaller changes mean fewer surprises. Frequent releases mean faster feedback.

However, continuous delivery only works if your team can integrate changes safely. That’s where Trunk-Based Development and Feature Toggling come in.

Trunk-based development keeps teams moving. Feature toggling keeps releases safe. Together, they make continuous delivery a reality. This article breaks down how to make both work in practice.

Trunk-Based Development

Many teams don’t struggle with writing code, but getting it into production without breaking everything.

For years, one approach for this has been long-lived feature branches, aka. Feature-Based Development (FBD):

Note: In this article I call the default branch in the git repository “main”.

A developer starts a feature, branches off main, and works in isolation. Days or weeks later, they try to merge it back. But by then, main has changed, and the branch is out of sync.

Merge conflicts pile up. The code review is a mess. The release gets delayed.

A situation known as merge hell—integrating code takes longer than writing it.

Trunk-Based Development (TBD) is the very opposite of FBD. Developers merge small, frequent changes directly into main via short-lived branches that last hours or days, not weeks.

The result?

• Fewer merge conflicts → Changes are small and integrate continuously.

• Faster feedback loops → Code gets reviewed and deployed quickly.

• Higher stability → main is always ready for release.

⚠️ There is a caveat: When you frequently merge to main, you can’t afford to introduce half-baked features that break production.

Feature flags help you integrate work early and hide unfinished functionality behind a toggle. This keeps main stable while allowing continuous integration and early testing in real environments.

Feature Toggling

Feature toggles (or feature flags) let you separate deployments from releases.

Code can be merged and deployed continuously, but features are only enabled when you’re ready. Some features need more testing. Others should only be available to a subset of users. Or if you are rolling out experimental change, you want the ability to disable it instantly (a kill-switch) if things go sideways.

At its core, a feature flag is a simple conditional statement:

Instead of using hardcoded values, feature flag services like LaunchDarkly or Unleash let you dynamically control which features are enabled, for whom, and when.

Feature flags are not just for enabling new features to users, but they can be categorized into different types:

• 🚀 Release Flags – Hide unfinished features until they’re ready.

• ⚙️ Operational Flags – Enable or disable features instantly in case of issues.

• 🧪 Experiment Flags – A/B test variations and measure impact.

• 🔒 Permission Flags – Control access based on user roles.

Example: Using LaunchDarkly in a React App

To see feature toggling in action, let’s integrate LaunchDarkly into a React application using the launchdarkly-react-client-sdk.

Let’s start with installing the package into your existing React application (or create one using this doc):

npm install launchdarkly-react-client-sdk

The next step is to initialize the client SDK with your project’s ID and any additional information, such as the currently logged-in user. To get the ID, you can register for the free Developer plan, which has a limit of 1,000 monthly active users (more than enough for testing it out).

With the initialization out of the way, accessing and using a feature flag (in this case called newFeatureEnabled is as easy as:

The last piece of the puzzle is creating the flag in the LaunchDarkly UI (or via Terraform).

When the flag is created, it is by default in the disabled (OFF) state. In the example below, you can see our flag enabled and scheduled for Progressive rollout, which in given intervals enables the new feature for more and more users.

You can use many strategies, such as percentage rollouts, region-based rollouts, customer-based rollouts, etc. The main benefit is that you can change who will see new features and when without deploying new code.

Best Practices for Feature Flags

• Keep flags short-lived: Don’t let old flags accumulate as tech debt. Remove them once a feature is fully rolled out.

• Use a feature flag management tool: Manually toggling flags via config files is a bad idea. Services like LaunchDarkly provide better control.

• Combine feature flags with analytics: Track how feature rollouts affect user behavior, key metrics, and system performance.

• Test features separately: Ensure that both the enabled and disabled states work correctly.

• Plan for quick rollbacks: Make sure toggles can be switched off instantly if something goes wrong.

Share

Common Anti-Patterns

Feature Flag Tech Debt

Feature flags are meant to be temporary, but teams often forget to remove old ones. Over time, the codebase becomes cluttered with dead toggles, making it harder to maintain and developers waste time navigating code paths that shouldn’t even exist anymore.

• ✅ Track your feature flags - Use a dedicated feature flag management tool (such as LaunchDarkly) to keep a record of all active flags.

• ✅ Set an expiration date - Every toggle should have a planned removal date. If a flag has been active for weeks with no change, it should raise a notification and the flag should be reviewed (and ideally removed).

Relying on Feature Flags Instead of Testing

Some teams use feature flags as an excuse to skip proper testing.

The logic is: “If something goes wrong, we’ll just turn it off.”

But this doesn’t prevent regressions. It just hides bad code until accumulated tech debt cannot be ignored anymore.

• ✅ Test both ON and OFF - Ensure that both flag states function correctly.

• ✅ Write automated tests - Don’t assume toggled-off code is harmless.

Poor Observability

A new feature is rolled out, but the team has no way to track how is it used, who is using it, or how reliable or performant it is.

• ✅ Use feature flag analytics – Track who has the flag enabled and monitor reliability and performance metrics for the new feature (in tools such as DataDog).

• ✅ Log feature flag state – Include flag values in your logs, or create separate log entries per flag states, so they are easily filterable in your observability tool.

Overcomplicating

Some teams create deeply nested toggles, making it impossible to understand what code is actually running. Too many interdependent flags make debugging a nightmare.

if (featureAEnabled) { 
  if (featureBEnabled) { 
    if (!featureCEnabled) { 
      doSomething(); 
    } else { 
      doSomethingElse(); 
    }
  }
}

• ✅ Keep flags simple – Each flag should control a single feature, not multiple behaviors.

• ✅ Avoid deeply nested logic – Don’t mix multiple feature flags together.

if (featureAEnabled) { 
  doFeatureA(); 
}

if (featureBEnabled) { 
  doFeatureB(); 
}

Summary

I hope this article gave you an actionable overview of benefits of Trunk-based development paired with Feature flags. In short:

• Trunk-Based Development eliminates the pain of long-lived feature branches, keeping teams in sync and reducing merge conflicts.

• Feature Toggling ensures that new features can be integrated early and safely, allowing controlled rollouts and instant rollbacks without risky deployments.

I'd love to hear about your best practices for shipping code with confidence!

📖 Read Next

Discover more from the Product Engineering track:

If you’re looking for a space where you can learn more about software engineering, leadership, and the creator economy, with Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve created the Engineering & Leadership discord community:

Join

📣 Recommended Reading

• 5 lessons learned setting up a global engineering org at Google by and

• 20 Business terms every Engineering Manager should know by in

• I Became a Monk for 10 Days so You Don't Have To by in

While you can say this is a business or product management concern, it’s also an engineering challenge.

Our work is wasted if users don’t find value in what we build. If we fail to measure and iterate on user adoption, we risk shipping features that never gain traction.

Considering Product-Market Fit (PMF) means considering the bigger picture. It means asking questions like:

Does what we’re building actually matter to users? Are we solving a real problem in a way that keeps them coming back?

Let’s look at how to connect your engineering work to real product outcomes by understanding Product-Market Fit and using tools and techniques to measure key metrics.

Is What We’re Building Actually Resonating with Users?

Product-market fit is the moment when a product clicks with the market. This happens when users adopt it and love it enough to keep using it, pay for it, and recommend it to others.

It means to have:

• ✅ High engagement – Users actively interact with key features.

• ✅ Strong retention – They keep coming back over time.

• ✅ Organic growth – People share it with others.

• ✅ Low churn – Users don’t abandon it after a few tries.

• ✅ Feature adoption – New capabilities are actually used, not just shipped.

The engineering challenge in PMF is going beyond tracking traditional engineering metrics by holistically assessing the end-to-end user experience:

• ⚙️ Every technical decision—architecture, performance optimizations, feature prioritization—affects how easily users can adopt and engage with the product.

• 📉 If we don’t measure how users interact with what we build, we risk shipping features that get ignored or frustrate users. Without data, we’re just guessing what’s working.

Share

3 Techniques for Collecting and Analyzing PMF Data

Luckily, PMF is reasonably easy to measure and validate with data. However, not all data is useful. Too often, teams track vanity metrics (like how many times a sign-up page was displayed) instead of real indicators of product adoption, retention, and engagement.

1. User Behavior Tracking: Are People Using What We Build?

🎯 Why This Matters

Building a project just for the sake of building it happens very rarely. Most of the time, we want other people, aka users, to use it. While as engineers, we are excellent at shipping features, we, at the same time, need to make the next step and measure if users engage with them.

The engagement consists of users discovering the feature, actively using (or abandoning) it, and converting in ways we expect them to.

📈 How to Measure It

The first step in any analytics is telemetry instrumentation. For PMF, this can be any event-based analytics tool, such as Amplitude, which allows tracking of how users interact with our product.

Then, we need to define key user actions to track, such as "User completed onboarding", "User enabled notifications", or "User saved first document".

In Amplitude, tracking these events from code is as simple as (for JavaScript version):

With event data being sent to the analytics tool, we can build and track conversion funnels to see where users drop off. For the event above, it can be: Signup → Onboarding started → Onboarding finished.

Here is an example of how a ticket purchasing funnel might end up looking:

With such funnels in place, we can ask questions like:

• Which step has the biggest drop-off in the funnel?

• How can we improve the UX in that step so users can continue the flow?

• Give the current conversion, is the feature viable?

An effective strategy while developing a feature is to A/B test several variants of its user experience. For this, I recommend LaunchDarkly, which I’ve been using for several years now (they also offer a free Developer tier).

Combining feature flags and funnel analysis helps find the optimal experience that allows users to navigate through desired flows effectively.

💡 Actionable Tip

Before shipping any major feature, apart from defining events and funnels, define one key metric that will measure its success. For example:

• If you are launching a new onboarding flow, track the Time to First Action to see if users learn the product more easily and engage more quickly.

• If you are rolling out an AI-powered search, measure the Search Completion Rate to see if results are relevant.

2. Retention & Engagement: Do Users Keep Coming Back?

🎯 Why This Matters

A one-time user is a failed experiment. Many products attract an initial wave of users through marketing, getting viral, or curiosity about a new feature. But it means nothing if they don’t stick around.

Retention is the ultimate validation of Product-Market Fit (PMF) because it signals that users see sustained value over time, not just a one-time novelty.

📈 How to Measure It

Retention is often visualized as a curve over time tracking the percentage of users who return after their first visit.

A product with strong PMF has a retention curve that flattens at a healthy level (in the example above it is 30%), while a product without PMF shows a steep drop-off.

For example, for applications that aspire to be daily tools (e.g., task tracking) or habits (e.g., Instagram) for users, there are three important retention indicators:

• Day 1 retention: Did users come back after their first visit?

  • If not: Users don’t see value fast enough during the onboarding.

• Day 7 retention: Are users integrating the product into their daily flow?

  • If not: The product might be too complicated to use.

• Day 30 retention: Are users still engaged after a month?

  • If not: Maybe the product doesn’t deliver ongoing value.

To measure retention, find an event or events that indicate the real usage of your applications and track them using the same methods I described in the previous section (Amplitude, GA, Tableau, etc.).

If we take a task-tracking app as an example, it can be “Tack created”, or “Task status changed”. Tracking events as sign-in or loading the home page isn’t enough because they do not indicate in any way that the product brings value to users.

💡 Actionable Tip

Find the main action that drives retention. Example for a task-tracking app: “Users who successfully create and finish a task within 3 days of sign-up are 5x more likely to retain”.

If you identify this pattern, optimize the product to drive more users toward that action. Make that user flow easier to find and execute.

For example, include a non-obtrusive in-product guide (e.g., via Pendo.io) to point users to this action.

3. Performance & Usability: Are Technical Issues Blocking Adoption?

🎯 Why This Matters

Even a product with perfect PMF will fail if performance issues, bugs, or bad UX frustrate and drive users away.

As engineers, we have direct control over removing technical blockers.

📈 How to Measure It

If high-value user actions have high error rates, PMF is at risk. For example if 5% of users experience an error when trying to checkout, fixing that bug could boost revenue instantly.

DataDog is de-facto an industry SaaS standard for observability. Monitoring error rates through custom metrics or crashes through error tracking is easy to set up either by hand in their app, or via Terraform using their provider.

As en example, formula for measuring success rate can be done as:

The second area to focus on is performance and latency. Even as simple as 0.85 second reduction in page load time can bump conversion by 7% [source].

Percentile metrics are a great way how to assess user experience and measure time users spent waiting on the system to respond:

💡 Actionable Tip

Identify 2-5 main high-value user flows and set up alerts for their reliability and performance spikes.

At the same time, set up also monitoring for percentage of unfinished flows indicating drop-off rate, which can be caused not only by the product offering, but also by technical errors:

Summary

Most engineers focus on shipping features and optimizing code, but it’s not enough. The best engineers ensure what they build actually matters.

If people aren’t adopting a feature, it’s either unnecessary, confusing, or has friction.

To measure PMF effectively here’s a 3-step approach you can use:

• 1️⃣ Before launching a feature: Define what success looks like, e.g., 30% of new users should be still active users after 7 days.

• 2️⃣ After launching: Use event tracking, funnel analysis, and A/B testing to measure impact.

• 3️⃣ Iterate: If results are low identify why or create a hypothesis, then iterate and experiment.

📖 Read Next

Discover more from the Product Engineering track:

If you’re looking for a space where you can learn more about software engineering, leadership, and the creator economy, with Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve created the Engineering & Leadership discord community:

Join

📣 Recommended Reading

• Managing people you don't like by and in

• 15 lessons from 15 years in tech by and in

• Mentoring doesn't solve all problems. The 4 people every software engineer needs to grow fast without burnout by in

There’s a worrying piece of code that isn’t entirely up to par, but you tell yourself, “I’ll clean it up later.”, even though you know it’s not going to happen.

You ship the feature and score positive feedback from users. However, in the back of your mind, you know there’s a price tag on that corner you just cut. But have you ever asked yourself:

What is the price tag of that technical debt?

Traditionally, we think of technical debt as a ticking time bomb, a messy leftover. But what if we began viewing it more like a strategic investment?

When managed well, temporary shortcuts can deliver quick wins—faster market entry, early validation, or resource savings—that might outweigh the cost of refactoring later.

This article explores how to approach technical debt as a calculated risk rather than a looming liability and how to be a product-minded strategist.

Every Debt Has an Interest Rate

Technical debt refers to the design or implementation choices that speed up delivery in the short term but potentially incur higher maintenance costs later.

Ward Cunningham, who created the term, compared these compromises to financial debt:

When you borrow “quick fixes” now, you must eventually repay the “principal” (e.g., refactoring effort) with “interest” (increased complexity, defects, and slower feature delivery).

Unlike in finance, the time required to pay interest is not fixed in software development. Some codes can have large tech debt for years without incurring large interest payments. On the other hand, for some code, the interest rate can grow exponentially.

That means we can make deliberate choices to meet pressing deadlines or market opportunities while controlling how and where tech debt is created.

Subscribe now

To start, let’s divide technical debt into Strategic (Positive) and Accidental (Negative) debt:

• Strategic Technical Debt:

  • Debt that is created intentionally.

  • Its goal is to maintain speed or quickly validate a product concept.

  • It’s strategic because you are accepting a known cost to capture a bigger opportunity in the near term.

  • Example: You take shortcuts when prototyping a new feature to test market interest. If the prototype flops, you’ve saved lots of time and resources. If it succeeds, you can refactor or completely redesign it later.

• Accidental Technical Debt

  • Unplanned debt.

  • Created by poorly managed processes—rushed sprints, inadequate testing or code review, lack of documentation—or poor architecture choices.

  • Leads to code that’s fragile and hard to maintain.

  • Often accumulates unnoticed, accumulating interest until it causes major slowdowns.

Recognizing Debt That Hurts

Not all debt impacts your codebase in the same way. Before deciding whether to refactor it, first identify whether the debt is actually hurting your ability to deliver quickly and reliably.

As stated above, Accidental tech debt is an unplanned or poorly managed debt that emerges from existing code. Therefore, deciding whether to pay it down must focus on its present impact:

On the other hand, for tech debt to be Strategic, it has to be a decision to skip certain best practices based on balancing its future impact and benefits:

The questions above should help you identify trouble spots in your existing or planned code. At the same time, from my experience, every dev team has an extensive “To Do” list of tech debt tasks they’d like to address. This list usually hangs at the very bottom of their backlog.

Since time is limited, which debt should be prioritized and addressed first?

Share

When to Pay It Down

The key to deciding which tech debt to pay first is to identify the debt that is causing the most damage to your product and your team.

Three major factors often guide this decision: the Frequency of change, the Criticality of the area, and the Opportunity cost.

Frequency of Change

The more often you touch or modify code with already existing debt, the more debt you accumulate. Frequent changes increase the complexity and risk of introducing regressions.

❓Questions to Consider

• How often does the team update or extend the functionality in this part of the codebase?

• Have minor modifications caused production incidents, bugs, or required patching?

• Does the current design adhere to the Open-Closed principle, or would extending the current functionality require a rewrite?

💰 When to Pay It Down

• If the area sees constant churn and every update is painful or risky.

• If upcoming roadmap items depend heavily on this code.

• If the debt is likely to block or slow development.

🛵 When to Let It Ride

• If the code is stable or rarely touched.

• If you plan to deprecate or replace this section soon, making a full refactor unnecessary.

Critical Area

Certain parts of your product or system are inherently more critical. This could be the core architecture on which every feature is built, a must-have product experience your users rely on, or key non-functional requirements such as security or performance.

❓Questions to Consider

• Is this code integral to the product’s core value proposition?

• Would a failure here cause significant business or reputational damage?

• Is the current design flexible enough to accommodate future planned features and growth?

💰 When to Pay It Down

• If the debt exists in high-risk areas—e.g., security modules, payment processing, or core user flows—it’s often safer to resolve issues sooner rather than later.

• If a breach or major outage in this area would significantly harm customer trust or violate regulations.

🛵 When to Let It Ride

• If short-term pressure to validate a market assumption outweighs the cost of addressing a less critical part of the code.

• If the area isn’t mission-critical and the frequency of change is low.

Opportunity Cost

Every hour you spend refactoring is an hour not spent on feature development, experimentation, or other business drivers.

However, the reverse is also true. Ignoring crippling debt can block new initiatives or slow down the entire product roadmap.

❓Questions to Consider

• What features or improvements are you postponing because you’re stuck managing debt elsewhere?

• How does the debt impact your team’s overall velocity and creativity?

• Are there pressing business opportunities, like an upcoming launch, requiring a stable, robust product?

💰 When to Pay It Down

• If the technical debt is causing incidents over and over again, e.g., constant memory leaks crashing servers

• If the debt slows down development, leading to missed market opportunities.

• If cleaning up the debt now unlocks a more scalable architecture, making future feature-work simpler and faster.

🛵 When to Let It Ride

• If you’re in a critical go-to-market push and the debt is unlikely to impact the short-term milestone.

• If the product is still in an experimentation phase and the cost of continuous refactoring doesn’t justify the uncertain ROI.

Summary

As a product engineer, you must take a pragmatic approach to tech debt. It’s not just about how great or lousy a piece of code is but about the value it brings to both customers and the business.

Being strategic about where and how you create tech debt can result in debt that is well isolated, outside of the critical path, and not slowing down future expansion.

And that’s the yin-yang of the development, the balance in the force.

📖 Read Next

Discover more from the Product Engineering track:

If you’re looking for a space where you can learn more about software engineering, leadership, and the creator economy, with Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve created the Engineering & Leadership discord community:

Join

📣 Recommended Reading

• 7 proven mental models for engineering managers in by and

• How to develop EQ as an engineer or a manager in by and

• Data Lakes For Complete Noobs: What They Are and Why The Hell You Need Them in by

There is a moment of silence. Then, the VP of Product Management shifts his gaze from you to your manager. “I thought we wanted to improve load time for a smoother user experience. Why are we burning time with this work?”

At that precise moment, it doesn’t matter that you’ve actually improved the load time by all those optimizations. What matters is that you failed to bridge your engineering efforts and product outcomes.

Engineering is often framed as a purely technical discipline—master the code, build the product, and ship the features. But that’s an incomplete story. Communication is not a “soft skill.” It’s a core product driver.

No matter your technical brilliance, when communication is ineffective, it remains hidden or misunderstood.

However, when communication is successful, teams align quickly, product decisions are more informed, and stakeholders have confidence in the roadmap.

In this article, you’ll explore communication not as a box to check on a “soft skills” checklist but as a fundamental component of your engineering toolkit.

Know Your Audience Before You Speak

Effective communication starts with understanding who you’re talking to and what they care about.

Whether it’s an executive, a customer, or a colleague from another department, you’ll need to choose the level of detail, vocabulary, and emphasis that resonates most with them.

Think of it as applying product thinking to your communications: you’re building a “message product,” and your stakeholders are your users.

The first step is to understand who the “customer”, aka. the listener, is and how much technical depth they can handle. You might ask simple questions like:

• “Have you dealt with similar systems before?”

• “Are you familiar with microservices, or should I briefly explain?”

Their answers—or puzzled expressions—will help you decide how detailed your explanation should be.

The table below captures several core stakeholder groups, their primary focus, and an ideal communication style for effectively engaging them.

Subscribe now

Make Your Message Stick

Once you know whom you’re speaking to, the next challenge is communicating technical information in a way that resonates.

The goal is to create common ground and understanding as soon as possible. If you delay this step or overwhelm your listeners with technical details before it, you will either lose their interest or make them irritated that you are wasting their time.

In my experience, two simple yet effective approaches, depending on the context, are Analogies and Tiers.

I use analogies to create common ground in shorter interactions or ad hoc discussions.

Analogy:

• a comparison of two otherwise unlike things based on resemblance of a particular aspect. [Merriam-Webster]

Analogies or metaphors are great tools to use when you want to translate a complex concept into a more familiar realm. For example, I often use analogies with the car industry:

• “In the initial phases of the project, there will be fewer visual features to demo for stakeholders, as we need to bootstrap the overall architecture for it. Think of it as building a strong engine for a car so that later on, we can add features such as autonomous driving, self-parking, or heated seating.”

This way, listeners can create a mental image of something they are familiar with (a car) and project that image onto the product’s development process.

Will it be 100% accurate? Of course not. But if you start by explaining to your Go-to-Market manager how you are going to write Terraform scripts to run your Go service on demand in Docker containers on AWS Lambdas while having a cross-regional NoSQL Mongo database cluster, they will probably find someone else to talk to.

Leading with the “Why”

This is an essential tactic for almost any communication. As posts on social media try to engage user attention with “hooks” within their first few sentences, you can do the same with your communication.

Explain how your technical work addresses a real problem—like reducing page load time by 30%, improving reliability during high-traffic events, or streamlining checkout processes to boost conversion.

Remember, writing code is never the end goal. Framing the conversation around outcomes keeps non-technical listeners engaged. Your goal is to help them see the direct line between an engineering work and its impact on users or the business.

Tiered Levels of Explanation

You often need to present your work or ideas to a diverse audience. An example is a sprint review or a planning meeting, where people from engineering, product, customer support, or marketing are present.

Using Tiers in this context means starting from high-level benefits or impact and moving to deeper and deeper details tier-by-tier.

Let’s look at an example of how to lead with “why” and offer a multi-tier level of technical details:

Why: “Based on our forecasts, we expect a significant increase in user traffic next quarter. If we don’t prepare, our users could face slow load times or errors, which risks revenue and customer satisfaction because our current system is already at its limits.”

Tier 1 (High-Level): “We’re shifting from a single, monolithic database to a distributed architecture to handle this growth seamlessly. We’ll no longer rely on just one database instance for all reads and writes. Instead, we’ll adopt a cluster of smaller, independent databases that can operate in parallel.”

Tier 2 (Mid-Level): “We’re moving from our traditional relational database to a NoSQL solution that scales horizontally. This move fits our use case and will help us handle large volumes of concurrent requests more efficiently.”

Tier 3 (Detailed): “Under the hood, we’ll implement Apache Cassandra, known for distributing data across multiple nodes. This setup minimizes bottlenecks and improves fault tolerance, so the system keeps running if a node goes down.”

Share

At this point, everyone in the meeting should have enough information to be on the same page and broadly understand what will happen and why. A follow-up discussion will be much more effective.

Quick-Reference Checklist

With the previous sections in mind, the next step is to embed these ideas into your day-to-day workflow. Keep this checklist on hand for a quick reminder before jumping into a meeting or creating a presentation.

1️⃣ Clarify Your Purpose

• What are my goals? Why am I communicating this information?

• What do I need from my audience? Is it a decision, an approval, or is my purpose to share knowledge?

• What is my “success metric” for this communication?

2️⃣ Know Your Audience

• What is their familiarity with the topic?

• What do they care about?

• Do they need to know technical details, business impact, user experience, ROI, or something else?

  • Use the table from the section “Know Your Audience Before You Speak”

3️⃣ Create a Tiered Message

• Start with high-level benefits or impact first.

• Dive deeper into technical or business details based on your audience.

4️⃣ Anticipate Roadblocks

• What concerns might be raised (budget, timeline, priorities)?

• How will I address them?

• Can I address them proactively?

5️⃣ Include The Audience

• Include sections to pause for clarifications or go deeper if there’s interest.

• Reiterate the core points to make sure everyone understands.

• Summarize key points after the discussion, especially if decisions were made, and share them.

Summary

Effective communication is as integral as writing clean code. It bridges the gap between technical details and strategic objectives.

Use the checklist above to gauge if your communication aligns with that goal.

It’s your responsibility to help stakeholders understand how your technical work solves real problems.

📖 Read Next

Discover more from the Product Engineering track:

If you’re looking for a space where you can learn more about software engineering, leadership, and the creator economy, with Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve created the Engineering & Leadership discord community:

Join

📣 Recommended Reading

• 6 secrets for never being blocked again by and in

• 5 learnings when building and scaling a service-based tech company by and in

• The deadline doom loop by in

But when the quarter ends, the mood shifts. Adoption of those new features is flat, and customer churn remains the same and the leadership is asking a simple yet frightful question:

“What’s the return on all this engineering effort?”

Many engineers must yet learn to accept that writing more code or completing more sprints will not improve revenue or save their product.

For years, engineering success was measured by outputs: the number of features shipped, tasks completed, or lines of code written. While these metrics quantify effort, they fail to capture what truly matters—the impact of that effort.

This disconnect between traditional engineering metrics and business outcomes is why many engineers struggle to demonstrate their value in their companies.

This article explores why traditional metrics fall short, what outcome-oriented metrics look like, and how they can transform how you communicate your impact. Ready to rethink how you measure success?

Subscribe now

From Outputs to Outcomes

Story points completed, number of features released—these metrics are easy to track and provide a sense of progress. But they also create a dangerous illusion.

Just because you’ve delivered something doesn’t mean you’ve delivered something valuable.

The problem with output-based metrics is that they focus on activity (output) rather than impact (outcome). A feature might ship on time and within scope, but the effort has little value if users don’t adopt it.

Outputs: Deliverables that quantify effort.

Output metrics often fail to answer the critical questions:

• Did it solve the right problem?

• Did it make users happier?

• Did it help the business grow?

• Did this work make the product better?

• Did it help users achieve their goals?

This is why outcome-oriented metrics are so powerful. Instead of measuring what you did, they measure what you achieved.

Outcomes: Results that demonstrate impact.

Examples:

• increased customer retention

• increased feature adoption

• reduced churn

• increased revenue

• faster time-to-market

Outcomes reflect the value created by engineering work. These are the metrics that matter in a product-driven organization. It shifts the narrative from “What did we deliver?” and starts answering “What did we achieve?”.

Why the Shift is Critical

You might wonder why this change is more critical now than ever. Modern engineering operates in a context where business needs, user needs, tech advancements, such as LLMs, and the economy itself change rapidly.

To stay relevant, teams must demonstrate how their work directly contributes to these needs and goals. If it doesn’t, there might soon come a time when the company’s investment might be worth more somewhere else.

Outcome-oriented metrics provide that clarity. They ensure alignment between engineering efforts and the broader objectives of the organization.

Share

Metrics That Count

Let’s say your team currently tracks several output-oriented metrics. What outcome-oriented metrics can they transform them into to better align with the business?

Mapping for your team might look different, but the list below should give you an idea of how to improve the visibility of your team’s contributions:

❌ Output: Features Released

✅ Outcome: Customer Retention Rate

Has the recently released feature improved the product? If not, does it still need to exist? A feature that does not contribute to a product's growth or stability is only a liability.

Retention metrics measure how well the product keeps customers engaged over time by providing solutions for their problems.

❌ Output: Lines of code

✅ Outcome: Feature Adoption Rate

High adoption rates indicate that features address user needs. They also show how effectively the written code impacts user engagement. The adoption rate is calculated as a percentage of active users using a given feature.

Delivering a feature with a high adoption rate is a great indicator, as it often goes hand in hand with retention rates. Building such a “sticky” customer base is one of the main goals of subscription-based services, as it provides a stable prediction for recurring revenue.

❌ Output: Sprint Velocity

✅ Outcome: Time-to-Market

The goal of calculating sprint velocity is to have predictability for team deliverables. Unfortunately, there is often a disconnect between “story points” and the work needed for a feature to be delivered to customers.

If this disconnect happens, it’s possible for the team to improve the velocity metric but degrade on time-to-market. People optimize what they measure, and if the measurement is story points “done”, then that’s what will be optimized.

Instead, track how quickly your team delivers new features to users. You’ll quickly realize that the main question to answer is what scope you will tackle in MVP and what will be part of further iteration.

Faster delivery enables faster testing of hypotheses and more effective adaption to customer needs.

❌ Output: Resolved Bugs Count

✅ Outcome: System Reliability

Has fixing bugs led to a more stable, reliable product for your users? Image a backlog of 10 bugs. You know that nine are relatively easy to solve in the code you are familiar with and are maybe even interesting, but their impact is low. The last one will be a pain in the keyboard and take a long time to solve, but it will improve the user experience and reliability the most. Which one will you choose?

Metrics like uptime, error rates, and response times reflect the stability of technical infrastructure and are directly related to customer satisfaction and retention. It might be more tempting to flex on stand-up with “I solved nine bugs this sprint.”, but if it hasn’t had any user impact, it’s just wasted effort.

❌ Output: Infrastructure Spending

✅ Outcome: Cost Per Active User

Suppose your company currently spends $2M/month on cloud services. Is it too much or not? Hard to say. What if the cost grew to $2.5M/month in the follow-up quarter?

Infrastructure spending is an important indicator to track, but it cannot be used alone to make decisions. In the example above, if spending grew by 25%, but the number of monthly active (and paying) users also grew from 1M to 1.5M at the same time, the cost per active user decreased from $2/month to $1.67/month. This might indicate that the platform scales well with the growing customer base.

Metrics that balance cost, performance, and scalability are highly valuable for engineering teams. Cost-efficient solutions save money and enable reinvestment in other areas, such as retaining engineering talent.

❌ Output: Code Coverage Percentage

✅ Outcome: Error Rate

A high code coverage requirement is a great example of a vanity metric. If developers must cover with tests, e.g., logic-less getters and setters in classes, just to hit the coverage bar, it’s just a waste of time.

On the other hand, measuring the overall error rate of a feature or a component is a highly valuable metric to have in place. If an error rate is high, the owning team might introduce or increase a required coverage percentage and observe if this effort impacts the error rate and, therefore, user experience, or they should try some other hypothesis.

❌ Output: Pull Requests Merged

✅ Outcome: Cycle Time

“I created a pull request for this feature yesterday. Please review.” is a statement that shouldn’t be overheard on any team’s stand-up. There are better methods to let your team know that a PR has been created (automatic chat, email, or push notifications) than waiting 12-24 hours for a meeting.

The cycle time metric tracks the time from opening a PR to merging it with the main branch or deploying it to a production environment. The goal is to ensure development efficiency. If a review takes multiple days or weeks, with daily ping-pong of “Here is a comment.” and “I pushed an update.” over and over again, it might mean the team is not prioritizing reviews, which can cause a lot of delays and context-switching.

Summary: Reflect on Your Current Metrics

Take a moment to evaluate the metrics you or your team currently use. Are they focused on outputs—like the number of features shipped—or do they reflect outcomes, such as user satisfaction or revenue growth?

Focus on whether these metrics measure the value your work brings to the organization.

Start by listing your team’s top three metrics and asking these questions:

1. Do these metrics align with the product’s goals?

2. Can they demonstrate impact on users or the business?

3. Are they actionable, helping guide future decisions and improvements?

For example, if one of your key metrics is sprint velocity, think about how it connects to company objectives. Does it only track effort? If yes, how could you adjust it to focus on outcomes, such as reduced time-to-market?

📖 Read Next

Discover more from the Product Engineering track:

If you’re looking for a space where you can learn more about software engineering, leadership, and the creator economy, with Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve created the Engineering & Leadership discord community:

Join

📣 Recommended Reading

• 5 skills to develop to grow from Senior to Staff Engineer by and in

• The victim trap of engineering managers by and in

• Balancing Engineering Excellence with Business Value by and in

We are often focused on the immediate task and miss the bigger picture until circumstances remind us again of a valuable lesson: in software development, everything is connected, and ignoring that interconnectedness can lead to a disaster.

Enter systems thinking—a way of looking at the whole rather than just the parts. To build robust, reliable software, we must understand how pieces fit together, anticipate the unexpected, and continuously learn from our outcomes.

Discover how a product engineering mindset can future-proof your career and bring deeper satisfaction to your work. Subscribe to Enginuity for practical guidance delivered straight to your inbox.

Subscribe now

The Butterfly Effect in Systems

We often focus intensely on the immediate problem at hand. We're eager to add that new feature, fix that bug, or push that release. But in this haste, we sometimes overlook how a small change can cascade through the system, leading to unintended consequences that can be costly or even catastrophic.

Anticipating and being on the lookout for these consequences helps avoid the costly cycle of fixing problems after they've been deployed. The bottom line is that users expect software to work reliably. Frequent bugs or downtime can damage a product’s reputation and lead users to seek alternatives.

Unintended consequences are often the result of not seeing the forest for the trees. We need to step back and consider the wider impact of our work.

Let’s look into two common system-thinking mistakes and then discuss a simple strategy to promote preemptive anticipation of such ripple effects.

Mistake 1: Ignoring the Bigger Picture

Individual software components rarely function in isolation. They are part of a larger ecosystem where each element influences and is influenced by others. This interconnectedness means we must think beyond the immediate scope and consider the broader impact of our work.

The impact of interconnectedness within the system is often revealed in later stages of development or only after the system is released and integrated within the production environment:

• Hidden Dependencies: Modifying an API without realizing other modules rely on it.

• Integration Problems: New code might work perfectly in isolation but cause issues when integrated with other components.

• User Experience Impacts: A backend change that slows down response times can frustrate users, even if the change was meant to improve functionality.

• Resource Constraints: Ignoring how a new feature might consume memory, CPU, or network bandwidth can degrade overall performance.

Mistake 2: Assuming the Happy Path

We often write code assuming everything will go as planned—the user provides valid input, the network is reliable, and resources are plentiful. But real-world conditions are messy.

Engineers often refer to conditions that are not on the assumed “happy path” as edge cases. The name itself holds the assumption that the probability of the occurrence of such an event is low, which is generally an incorrect assumption. You can only make such a statement after you confirm it via production telemetry over a long period of time. Failing to handle “edge cases” can lead to system failures, race conditions, and data corruption.

A popular scenario of unhandled edge cases is assumed inputs—not validating user or API inputs, leading to security vulnerabilities or system crashes when unexpected data is processed.

Scenario Planning with "What If?" questions

One of the most powerful tools in anticipating unintended consequences is scenario planning. Start by mapping out how your software is intended to work and what different scenarios a user can encounter. Then, deliberately consider how it might fail. Sometimes, this practice is called pre-mortem (as opposed to post-mortem). For example:

• User Behavior: What if users input unexpected data? What if they click buttons in an unforeseen order?

• System Failures: What if the database goes down? What if the network is slow or unreliable?

• External Dependencies: What if a third-party API changes or becomes unavailable?

• Security Threats: What if someone tries to hack into the system?

“What if” questions help you see the bigger picture, analyze the potential impact, and develop mitigation strategies for the highest-priority scenarios.

The Case Against Oversimplification

In the rush to deliver, it's tempting to boil down complex problems into simple solutions. We tell ourselves that by simplifying and overlooking nuances, we'll move faster and more efficiently.

But oversimplification often comes at a steep price. When we cut corners to make a problem seem easier, we lose critical information. As the saying goes: “The devil is in the details.” Ignoring them can lead to solutions undermining all sides of the Value - Quality - Time triangle:

• Value: Oversimplification can lead to a flawed product design. If we don't understand user needs, we risk building something that doesn't solve the right problem.

• Quality: Avoiding the inherent system complexity results in architectural tech debt. As the system grows, the ignored debt resurfaces, making the codebase harder to maintain and evolve. This debt is often paid with high interest. Future changes become more costly and time-consuming, slowing down progress and innovation.

• Time: Underestimating the complexity of a task leads to unrealistic timelines and resource allocation. Projects overrun their deadlines, budgets grow, and teams burn out. As a result, such projects often move much slower than well-estimated ones and lose the trust of stakeholders.

How To Use Complexity Effectively

1. Ask the Right Questions: Probing into the difficult aspects of a problem can save you a lot of future efforts for overhauling your overall solution. Try asking:

   1. What dependencies exist?

   2. What are the edge cases?

   3. How might the system evolve?

   4. How can risks of possible future issues be minimized?

2. Iterate Thoughtfully. The goal is to break down the complexity into smaller, manageable pieces. This is different from oversimplifying, which avoids addressing many of those pieces from the very beginning. Tackle each piece methodically, ensuring it fits into the larger picture.

3. Invest in Good Design: Take architectural decisions that favor flexibility and scalability. I cannot stress this enough! If your system is extendable and maintainable, you are on a great path. This investment pays off by reducing future work and technical debt.

4. Be Honest in Estimates: When planning, factor in the complexities. It's easy to be a “yes-man” and underestimate the work ahead, but in the end, stakeholders prefer realistic timelines over unreachable ones.

Applying Feedback Loops

Feedback loops are the mechanisms for learning and adaptation. Because of them, we know whether what we're doing is working or not, and without them, we're just guessing.

Positive feedback loops reinforce a particular behavior and lead to its amplification, for example:

• Feature Adoption: A new feature that gives users a lot of value leads to increased usage, encouraging further investment in that area.

• Code Quality Improvements: Refactoring code makes it easier to add new features, which leads to more refactoring also in other areas because the benefits are clear.

Negative feedback loops counteract changes and promote stability and balance. Examples:

• Automated Testing: Failing tests signal issues that need fixing before code is merged, preventing bugs from reaching production.

• Continuous Integration (CI): Build and integration failures alert the team to problems and incompatibilities early, avoiding bigger production problems later.

Mistake 1: Delayed Feedback

Delaying feedback usually means ignoring negative feedback loops.

One example is the use of long-running feature branches. They often accumulate and batch many changes before testing or deploying, which delays feedback and makes it harder to pinpoint issues. Another variant is infrequent batched releases, where teams merge their changes to the main branch, but the production release itself happens only a few times a year.

Another common mistake is ignoring test failures. As we move up the testing pyramid, the probability of tests being flaky increases because more systems are included in the tests. Proceeding despite failing tests undermines the purpose of testing. Of course, the worst scenario is not writing or running tests at all.

Prevention:

• Automated CI/CD: Automating testing and deployment ensures that test suites are run consistently and deployments to all environments are repeatable and reliable.

• Fail Fast: Early detection mechanisms, such as unit tests, type checks, linting, and code coverage, prevent issues from snowballing into bigger problems.

  • At the same time, knowing that tests have your back makes it safer to improve the codebase and refactor with confidence.

• Code Reviews: Reviewing the code before it’s committed to the main releasable branch is one of the most effective ways to prevent architectural tech debt and avoid changes that decrease the system's maintainability.

Mistake 2: Low-Value Feedback

First, let’s address the "no feedback” scenario. Without monitoring or observability in your production system, you won’t know when and why things break. Long outages damage product reputations and cause user dissatisfaction and churn.

Low-value feedback is feedback collected without a proper strategy, prioritization, and categorization of its sources. For example, tracking too many metrics can obscure what's important. It can desensitize the team to the data arriving from the system, causing important signals to be missed.

One of the worst-case scenarios is when these low-value or limited feedback loops drive drastic product course corrections. Such knee-jerk reactions result in long-term instability and loss of product’s identity.

Prevention

Prevention strategies, in this case, are focused on identifying high-value positive feedback loops:

• Feature Prioritization: Instead of having dozens of different metrics and statistics, focus on having a few high-value composite metrics coming from validated sources. You can create valuable insights only when you trust the data coming from the system.

• User Behavior Tracking: Understand how users interact with your software to make informed decisions. Just because two things happen together doesn't mean one caused the other. Be cautious when drawing conclusions. Check out ’s article Correlation Does Not Imply Causation.

Summary

Let’s sum up the three principles:

1. Consider the interconnectedness of components to avoid the butterfly effect, in which small changes ripple through the system and cause damage.

   1. Employ scenario planning with "What If?" questions to discover the real radius and impact of the change.

2. Avoid oversimplification, which leads to loss of critical information, and instead use complexity effectively through exploration, good architectural design, and iteration.

3. Leverage positive feedback loops to achieve amplification of valuable outcomes and negative feedback loops to prevent harmful changes from destabilizing your system.

📖 Read Next

Discover more from the Product Engineering track:

If you’re looking for a space where you can learn more about software engineering, leadership, and the creator economy, with Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve created the Engineering & Leadership discord community:

Join

📣 Top Picks

• The one framework every engineer should know by and in

• Safe vs ALL-IN Engineering Management by in

• Code Quality in the Age of AI by and in

However, as the years pass and tangible impact becomes increasingly difficult to see and measure, this drive for excellence and endless career opportunities starts to diminish.

You might change your project, your team, or perhaps your job, but at the end of the day, no external change really changes how you feel.

You’ve hit a career plateau.

Career plateaus don’t happen due to a lack of talent or effort. They are a natural phase that arises from repetitive tasks, diminished challenges, or a misalignment between personal values and professional roles.

If this resonates with you, know that you're not alone. And more importantly, there are strategies to reignite your growth and motivation.

Subscribe now

Active Boredom

It's possible to be knee-deep in tasks, meetings, and deadlines and still feel bored. I call this state "active boredom”—your hands are busy, and your calendar is full, but your brain is on autopilot, and you don’t feel fulfilled.

Active boredom often creeps in unnoticed during a prolonged period of time when your work doesn’t resonate with your values and goals. You are walking a path that leads, at best, nowhere and, at worst, in the opposite direction to where you’d like to be heading. You find yourself:

• Going through the motions: Completing tasks out of obligation rather than interest.

• Lacking passion: Feeling indifferent about successes or milestones that once excited you. Sometimes, you might even feel indifferent about the failures of projects your team or the company itself works on.

• Avoiding reflection: Sidestepping questions about your career satisfaction or possibilities for improvement.

Ignoring these signs can lead to prolonged dissatisfaction. While a job change might seem like the solution, without understanding the root cause, you risk repeating the same pattern elsewhere.

Actionable Takeaway

You’d be surprised how many people know what they don’t want in their careers but at the same time don’t know what they want. I was one of them for a long time. And no, “more money” is not enough if you are looking for a fulfilling career.

Set aside a few hours this week (I find weekends to be better for long-term planning) and clearly outline what you’d want to achieve in your career. I wrote an article Career Progression For Software Engineers: How To Own Your Success, which contains steps and questions to help you with this assignment.

There is one more benefit of having a defined career plan: By focusing on the steps you should take to achieve your goals, you can make your 1:1s with your manager much more effective and targeted.

The Power of Diverse Perspectives

Engineering is, by nature, rich in specializations. Whether we’re talking about frontend, backend, data, infrastructure, or ML, these specializations are required because those fields are huge and growing even more every year.

However, staying within the confines of your technical specialty can become comfortable and stale. You write code, debug, repeat. But engineering isn't just about lines of code. It's about creating products that make a difference.

Therefore, to reignite your passion and see the true impact of your work, consider stepping beyond your usual role.

Cross-functional collaboration breaks down the walls that keep you in your comfort zone. When you start interacting with other teams, design, marketing, or customer support, you see how your work fits into the larger puzzle.

By seeing multiple perspectives, you begin to understand the real-world impact of what you build.

Asking "Why does this feature matter?" means you're considering user needs, business goals, and market context. This transforms you from a narrow specialist into someone with deep expertise in one area and a broad understanding across multiple domains.

This transformation doesn't dilute your skills. It amplifies them. You bring valuable insights from different perspectives, making you more adaptable and effective.

You're not just coding. You're contributing to the product's success.

Actionable Takeaways

1. This week, schedule a time to engage with someone outside your team. Ask them about their work, challenges, and how it intersects with yours. You can join a demo session of another team, an architecture review, or sit in on meetings with product or customer service. Understand their challenges and how they relate to your work.

2. Learn about the underlying problem your product solves. How does your work influence revenue, user retention, or growth? What are the areas that cause the most customer pain? Are there some 80/20 tasks (aka. low-hanging fruits) that would greatly benefit users?

The Hidden Assets

In engineering, technical knowledge (hard skills) is often prioritized over soft skills. However, skills like emotional intelligence, storytelling, negotiation, persuasion, and leadership are very much needed for career advancement and work satisfaction.

Improving my soft skills in the areas mentioned above has been a highly motivating activity for me over the last year and has brought me a considerable amount of fulfillment.

The opportunities for learning in the soft skills space are infinite. I want to discuss one concept I’ve been focusing on for some time now. This area is often overlooked, but it’s one of the best ways to influence your work and align it with your values and goals.

Communicating and "Selling" Ideas

You might have the most innovative solution to a problem, but if you can't communicate it effectively, it will be as good as nonexistent.

Effective communication, persuasion, and idea-selling isn't about manipulating others. It's about clearly articulating your vision so that others understand its value.

Why does this matter? Because every project lives or dies by the buy-in it receives from stakeholders. Whether you pitch to management for resources, collaborate with other teams, or convince clients, your ability to convey the importance and benefits of your ideas determines their fate.

There are three main aspects you should pay attention to when presenting and pitching your ideas:

1. Understand The Problem: We are returning to the idea of perspectives. Problems are rarely single-faceted, and you can bet that if you consider only one perspective (which in software engineering is usually a technical complexity), you’ll find it very hard to create a compelling argument.

2. Understand Your Audience: Tailor your message to goals and level of technical understanding of your target audience. Not adjusting the message to the audience is the biggest mistake I have seen SWEs on all levels of seniority make.
   Ask yourself:

   1. What problems do they face?

   2. What do they hope to achieve?

   3. Is my idea aligned with their goals or going against it? Or even worse, are they completely indifferent to it?

   4. Should you communicate more details or strategy?

3. Craft a Clear Message: Based on the two points above, create the message you want to present. It should be clear what the problem is, why it has occurred, how you propose to solve it, and what the outcomes and benefits will be.

Of course, your idea can still be declined. However, improving your ability to articulate your vision clearly and make others listen with attention is something you will benefit from in the long run. It’s a skill that will help you when you search for your next job, apply for promotion, or decide to start a company and build your own product.

Summary

You might have noticed that most of the advice in this article can be summed up to:

Understand what you truly want and then take action in that direction.

In my case, and I strongly believe it applies to many of you, both the understanding and the action are rooted in looking at work and career more holistically. Career plateaus are not so much about the external elements as they are about discovering who we are, where we want to go, and what the best ways to get there are.

So what are you waiting for?

📖 Read Next

Discover more from the Product Engineering track:

If you’re looking for a space where you can learn more about software engineering, leadership, and the creator economy, with Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve created the Engineering & Leadership discord community:

Join

📣 Top Picks

• You May Need To Start Creating - One Year and 50,000 Words Later by in

• Engineering management in the next unicorn app by in

• 5 Lessons I learned the hard way from 10+ years as a software engineer by and in

But how to guide your team through both creativity and precision?

From a product engineering perspective, we must ensure that we build something meaningful rather than just build something fast.

We need an approach that helps us explore a wide range of possibilities and then zero in on the right solution with laser focus.

Divergent and Convergent Thinking

Exploration is the first step in a problem-solving process (or at least it should be if you care about the result). It is a broad, unconstrained search for any potential solutions.

This is where divergent thinking comes into play. It’s your team’s chance to be creative, to think outside of the box, and to generate as many ideas and perspectives as possible.

Divergent thinking prevents you from jumping straight into a solution before fully understanding the problem. This is the foundation of product engineering.

Once you've explored all the possibilities, it’s time for convergent thinking. This is where you start narrowing down options.

1. Review the ideas generated in the divergent phase

2. Eliminate the ones that don't align with the core problem

3. Focus on refining the most viable solutions

This process ensures the solution you will build is rooted in real user needs, feasible from an engineering point of view, and aligned with business goals.

Subscribe now

The Double Diamond Design Process

The Double Diamond (developed by The Design Council) is a framework for solving complex problems. It’s built around two key phases, each containing divergent and convergent thinking stages.

The goal is to first thoroughly explore the problem space and then focus on delivering the right solution with feedback loops in the process.

Discover (Explore)

This stage is a divergent process. You're not limiting your scope or making assumptions. Your goal is to:

Explore the full problem space and ensure the team is focused on solving the right problem, not just what seems like the obvious one.

How to Implement

The discovery phase is your chance to challenge assumptions, investigate real user pain points, and bring valuable insights to the team.

• User Interviews: Talking directly to users or customers is one of the best methods to understand their pain points and expectations.

• Competitive Analysis: Look at direct competitors and products that offer similar feature sets to see how they solve this problem.

• Journey Mapping: Visualize a user's experience with your product or service to identify gaps and pain points that need to be addressed.

• Data Analysis: There is a high chance you already have a ton of existing product data (telemetry, session recordings, analytics, data lakes) you can dig into. Whether you discover success stories or failures, you can use the 5 Whys Framework to analyze the root causes for those outcomes.

At this stage, product and tech leaders must ensure that all relevant voices—designers, engineers, marketers, and users—are heard.

Define (Reframe)

Next is a convergent thinking stage, which narrows down the wide range of insights gathered to a clear, specific problem statement. The goal is to:

Extract key insights from the discovered data to reframe the problem so you can focus on the most pressing issue that needs to be addressed.

This is where the user-centric mindset of product engineers shines. By the end of this stage, every team member should know what problem they’re solving and why it matters.

How to Implement

• Affinity Mapping: Use affinity diagrams to group and categorize your research findings. You might have used Affinity mapping without realizing it before—it’s often used to group similar topics during Scrum’s Retrospective sessions. The goal is to identify common patterns and themes.

• Problem Statement Template: To frame the problem clearly, use a well-defined template. You might have already encountered a template for user stories—“As a [persona], I [want to], [so that].”. A problem statement template may look like this:

  • “[User] needs [need] because [insight].”

  • For example: "New users need a simple onboarding process because they feel overwhelmed by too many features at once."

• How Might We Questions: These questions help reframe the problem so that it encourages solution-focused thinking. For example:

  • "How might we simplify the onboarding process to prevent users from feeling overwhelmed?"

• Prioritization Frameworks: Use tools like the Eisenhower Matrix or RICE to prioritize problems based on their impact and effort. Tech leaders should guide the team in making tough decisions about prioritizing issues based on business goals and user needs.

Develop (Create)

The second diamond (phase) starts again with divergent thinking. This stage is all about experimentation:

Thinking creatively to generate a range of possible solutions to the defined problem and explore them in detail.

How to Implement

• Brainstorming: It doesn’t have to be specifically structured. Even free-from brainstorming works in many situations. If you want to add more structure to it, you can use methods like SCAMPER to promote creative thinking and generate new ideas. Experiment with approaches that might initially seem out-of-the-box.

• Wireframing: Wireframing is an easy and effective way to sketch out multiple solutions. Keep things low-fidelity and focus on functionality rather than aesthetics. The goal is to explore multiple possibilities before narrowing down.

• Rapid Prototyping: These prototypes should be fast to build and easy to iterate on. Creating the first version of prototypes using tools like Figma, low-code approaches, or code gen is a quick way how to get something out of the door that can be validated with users.

Deliver (Catalyse)

The last stage is again a convergent thinking stage. You test identified solutions with real users, gather feedback, and implement the most effective one:

Deliver a solution that works, solves the core problem, and meets user needs and expectations.

This is the part that is most familiar to us, software engineers. It’s about incrementally delivering value in the highest possible quality based on the results from the previous stage. Let’s look at some techniques to make the delivery more effective:

How to Implement

• A/B Testing: Testing multiple versions of your solution in real-time is the best way to get immediate and measurable feedback from users. With tools such as LaunchDarkly, you can leverage feature flags to switch between multiple versions, and analytics tools like Google Analytics or Amplitude provide means to measure the usage, conversion, and effectiveness of your solutions.

• Feedback Loops: Whether it is alpha/beta testing with selected users or in-app surveys, direct feedback from customers can uncover “unknown unknowns” you would otherwise miss just by analyzing your product telemetry.

• Continuous Integration & Deployment: Now standard across the industry, CI/CD pipelines are a great tool for rapidly and frequently releasing improvements. It not only helps with quickly reacting to users’ feedback, but another fortunate benefit is that you can quickly fix or revert any mistakes you make along the way.

Summary

Even though the double diamond design process is over 20 years old (check Design for Planet for an updated advanced version), it’s still very relevant in the effective development of new products or features.

The relevancy is in combining divergent and convergent thinking while analyzing and solving challenges. The next time you are presented with a complex challenge, try using this method to explore and think outside the box before converging on the actual solution you will deliver.

📖 Read Next

Discover more from the Product Engineering track:

Are you looking for a space where you can learn more about software engineering, leadership, and the creator economy and network with other like-minded professionals?

With Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve just opened the gates to a new Discord community: Engineering & Leadership

Be among the first members and join us for virtual meetups, Q&A sessions, and events:

Join

📣 Top Picks

• 5 Keys to the Hiring Manager Interview from a Meta Senior Manager by and in

• Regression Towards the Mean - Mental Model by in

• The importance of data when making decisions in the engineering industry by and in

You can find all three tracks in the main menu of the Enginuity Newsletter.

Subscribe now

Customer satisfaction isn’t just a nice-to-have. It’s a necessity and a survival tactic.

Build something that doesn’t address your users’ needs and expectations, and they’ll move on to the next option faster than you can roll out an update. But there’s more:

Great companies don’t just meet expectations—they exceed them, and they exceed them consistently.

No one does this better in the long term than Apple. They know how to nail the essentials while pushing boundaries.

Let’s explore why delivering great products isn’t about luck but about adopting the product engineering mindset and how Apple does it with every product launch.

The Kano Model

The Kano Model is a straightforward but powerful framework for prioritizing features that matter. Features that not only satisfy customers but also delight them.

The model breaks down customer satisfaction into five categories:

1. Basic Needs (Must-Haves)

   • These are the features that users expect as a baseline. For example, in smartphones nowadays, users expect basic functionalities like making calls, sending texts, and decent battery life. And they better be flawless.

   • If your product lacks these, it’s game over. It won’t matter what else you offer.

   • But there’s a catch: Meeting these needs won’t satisfy your users. Your product must have them to be even considered as a viable alternative.

2. Performance Needs (Satisfiers)

   • These features directly impact satisfaction. The better you deliver on these, the more satisfied your customers become.

   • Returning to our smartphone example: faster processing speeds, better camera quality, and longer battery life. These are the things people will notice and appreciate. If you focus on delivering and improving performance features, you’ll consistently improve your user experience.

3. Delighters

   • Delighters are features that can really separate you from the competition. These are the innovations no one saw coming, but now they can’t live without.

   • They go beyond what users ask for—they surprise and delight, creating a lasting emotional connection. They make customers think: “This company really understands and cares.”

4. Indifferent

   • Not everything you add to a product will have an impact. Indifferent features don’t affect user satisfaction one way or the other.

   • Examples may be flashy UI elements that users don’t care about, improving features with extremely low usage, or adding new features that do not complement your product’s purpose.

5. Reverse

   • More isn’t always better. Some users might actually dislike certain features, like too many configuration options, too complex setup, or information overload on main screens.

   • Understanding what not to prioritize is just as crucial as knowing what will delight your users.

Apple's Application of the Kano Model

The Kano Model is baked into Apple’s DNA. Apple’s product engineering consistently strikes a balance between meeting expectations and delivering features that wow their customers.

1. Must-Haves

   • There are three main basic needs Apple is not willing to compromise on: reliable hardware, security, and seamless integration.

   • Whether we are talking about design and materials, Secure Enclave, FaceID, Handoff, or AirDrop, these features are present in almost every new product, and at this point, removing any of these features would highly dissatisfy their customers.

2. Satisfiers

   • These are the features that make people feel they’re getting real value for their money (and let’s be honest, Apple products don’t come at low price tags).

   • Improved camera quality, higher processing speed, or longer battery life—all these performance improvements show an understanding of customer needs, and they raise the bar where it matters most.

3. Delighters

   1. What sets Apple apart is the ability to introduce features that no one asked for, but now users can’t imagine living without.

   2. Introduction of M1 chips (Apple silicon) is the perfect example of this strategy. Moving away from Intel processors gave users unprecedented power in tasks like video editing or 3D rendering and, at the same time, changed what users thought was possible in a laptop or tablet.

The Arrow of Time

Today’s innovations eventually become tomorrow’s basic expectations.

This concept describes how Delighters can, over time, transform into Basic Needs that customers can’t live without.

When Face ID was first introduced with the iPhone X in 2017, it was a bold leap. Apple replaced the Touch ID fingerprint sensor with a facial recognition system that worked simply by looking at your phone.

At the time, Face ID was a Delighter—a feature no one had specifically asked for. Fast-forward a few years, and what was once an innovation is now an expectation.

This is the arrow of time in action. Users now take Face ID for granted. It’s a feature every iPhone must have to meet their baseline expectations.

Application to Carrer Progression

While writing this section, I remembered reading the article by and : The 3 Big Mistakes That Almost Cost Me My Promotion (And How You Can Avoid Them).

In the article, they describe the concept of the “Promotion Donut”— Forgetting to fulfill your current responsibilities while aiming for promotion to the next level.

Forgetting about the arrow of time is exactly the same principle. If you focus only on Delighters (responsibilities of the next level) and forget to pay attention to Must-Haves (responsibilities of the current level), you are setting yourself up for failure in the long run.

Lessons for Product Engineers

Applying the Kano Model is about transforming your understanding of customers’ needs into actionable strategies.

💡 The "Why" Behind Development

Invest in features that contribute to and follow your vision for your product.

One of the biggest lessons Apple offers is its commitment to product development driven by purpose. Every feature has a strong “why” behind its existence.

Let’s take their M chips as an example—the motivation wasn’t only about performance. Apple wanted better control over the user experience, which is the centerpiece of their vision, by creating a much more seamless hardware-software integration.

Therefore, identify what users expect, what will improve their experience, and what will positively surprise them:

1. Start with the essentials—are there features that are missing or features that frustrate users? In many cases, you already have that data, but it’s hidden in customer support tickets, product telemetry, or past surveys.

2. Ask yourself and your team: “What could make our product’s experience faster, smoother, or more intuitive?” Look at the features that are heavily used. Even small improvements in those areas will increase customer satisfaction.

3. Dedicate (and fight for) time for experimentation and innovation. Creativity cannot be forced, and it definitely cannot expected in stressful environments and teams with low psychological safety.

🎯 User-Centric Approach

• Solving Real Problems: Many companies nowadays create features for the sake of innovation. They don’t solve real issues, just create artificial hype. Just look at all the AI-powered nonsense we are bombarded with daily. There is a place for AI and its power, but it doesn’t have to be in every product.

• User Research and Feedback Loops: Run usability testing and surveys and keep a constant feedback loop with your target audience. Don't wait for the product to be finished before collecting input. Apple is known for keeping their innovations secret, but they still listen to feedback. For example, after the criticism of the butterfly keyboard, they redesigned and re-introduced the scissor-switch keyboard in their newer models. Ideally, in your case, it won’t take as long as it took Apple.

🧭 Building Leadership Qualities

Shipping code is not everything. Great product engineers help shape the culture around them. I’ve already written about empathy in 3 Lessons for Engineers from Well-Written Product Requirement Documents, but it’s something worth re-iterating:

Leading with empathy means understanding your users and your team. Creating an environment where team members feel empowered to share their insights is key to building products that actually solve problems.

Vision-Driven Innovation

Strong leaders can articulate a vision and purpose that go beyond the immediate sprint or release cycle. They focus on the long-term impact of what they’re building.

Don’t forget that true leadership is not a formal role granted by your work title. It’s a mindset and commitment that you have towards your profession.

It’s up to each of us to create an environment in which failure is perceived as a learning opportunity that pushes toward innovation.

We must value bold ideas and customer satisfaction over bureaucracy or rigid hierarchies. We must encourage experimentation and be willing to take risks when they’re grounded in real user needs.

Are you looking for a space where you can learn more about software engineering, leadership, and the creator economy and network with other like-minded professionals?

With Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve just opened the gates to a new Discord community: Engineering & Leadership

Be among the first members and join us for virtual meetups, Q&A sessions, and events:

Join

📣 Top Picks

• Bottlenecks - Mental Model by in

• Growing from engineer to Staff engineer and thriving by and in

• Managing Growth and Expectations in Engineering Teams by in

You can find all three tracks in the main menu of the Enginuity Newsletter.

Subscribe now

“30% of developers say knowledge silos impact their productivity ten or more times per week.” — StackOverflow

If your team suffers from a lack of cross-organization knowledge sharing, you are not alone. Knowledge silos are a productivity killer, frustrating developers and slowing down innovation.

But why do they happen? And more importantly, how can you prevent them from forming?

Let’s examine their root causes and practical, no-nonsense advice on preventing them and eliminating them if they’re already causing chaos.

Why Knowledge Silos Matter?

Knowledge silos happen when information, skills, or expertise are confined to specific individuals or teams rather than being shared across the organization.

They are often created unintentionally by incorrect processes, lack of communication, or even a sense of ownership over information.

When it happens, it becomes difficult for others to access it, leading to slower decision-making, slower collaboration, and duplication of effort.

Knowledge silos are a significant productivity blocker.

The 2024 StackOverflow Developer Survey shows the real impact of knowledge silos:

• 45% of developers report that knowledge silos negatively impact their productivity three or more times a week.

• 45% of developers say that these silos prevent them from getting their ideas across the organization.

Think about it—almost half of your team is frustrated several times a week because they can’t find the information they need or their ideas are stuck in the same feedback loop.

Knowledge silos don't just waste time. They kill momentum and slow down creativity.

Root Causes of Knowledge Silos

Organizational Structure

1. Isolated Teams: When teams are too isolated or too specialized, they usually develop their own processes and their way of working.
   This specialization can lead to “us vs. them” thinking, where teams focus more on their tasks than the organization's goals. This leads to a fragmented organization where knowledge doesn’t cross team boundaries.

2. Information Flows Only Vertically: Many traditional organizations are hierarchical. Information flows up and down the chain of command but rarely horizontally.
   This means knowledge is locked within departments. Employees may feel they have no voice and valuable knowledge never reaches decision-makers or other teams who could benefit from it.

Cultural Factors

1. A Culture of Control: In some organizations, information is treated as a commodity — something that gives power to those who hold it. This is called the “information is power” mindset.
   Teams or individuals might withhold information intentionally, believing it makes them irreplaceable and protects their status.

2. Lack of Trust Between Teams: Teams become protective when they fear sharing knowledge will expose their weaknesses or lead to negative feedback.
   One cause can be a highly competitive environment where departments compete against each other — for headcount, bonuses, praise… you name it. Without trust, knowledge remains confined to its original holders.

Lack of Tools and Processes

1. Absence of Effective Knowledge-Sharing: Without dedicated platforms for sharing knowledge, information gets lost in emails, Slack discussions, or ad-hoc documents. Teams spend valuable time reinventing the wheel because they can’t access existing resources or learnings.

2. Inadequate Onboarding: Onboarding is a time for sharing knowledge. And many companies struggle at this critical phase. When new employees don’t know where to find the information or who to ask, knowledge silos are created from day one.

How to Prevent Knowledge Silos

Knowledge sharing must be a fundamental part of your company culture. This is the only proactive approach that works.

Teams should be encouraged to hold regular cross-functional meetings to discuss ongoing projects, share learnings, and openly ask questions. These can be regular meetings organized by a “virtual” horizontal group:

• Guilds: A"guild" is a group formed around shared interests like frontend guild, backend guild, machine learning guild, or product design guild. These guilds meet regularly to share best practices, discuss challenges, and collaboratively solve problems.

• Architecture Review Sessions: These sessions open up technical decision-making to the whole organization. Everyone can participate as a passive listener to gain knowledge or as an active contributor to influence decisions. This ensures everyone has visibility and can contribute their insights.

• Meetups: Whether in a longer form or a lightning talk format, these sessions are open to anyone to share knowledge internally and, at the same time, offer an opportunity to gain experience in presenting to a broader audience.

Effective Knowledge Management

The single must-have, non-negotiable functionality your knowledge management tool (Confluence, Notion, etc.) must support is effective searching. Luckily, with LLMs being integrated into these tools, searching is becoming more and more effective.

Even if your knowledge base is spread across multiple platforms (wikis, docs, chats, etc.), having a centralized search (tools like Glean) across all of them will tie all these sources together in one knowledge-sharing platform.

But still, two problems need to be addressed:

(1) This sounds great on paper, but most of the documentation becomes obsolete quickly, and no one reads it anyway.

Minimum Viable Documentation (MVD)

The purpose of writing documentation is not to produce a lot of text. It’s to capture the most vital knowledge about the reasoning behind crucial decisions and fundamental principles (architecture, protocols, etc.).

Minimum Viable Documentation focuses on capturing only the essential knowledge people need to perform their jobs effectively. This keeps it relevant and actionable for the whole time of its existence.

(2) Writing documentation is a tedious process. Where and how should I even start?

Standardized Documentation

When everyone uses the same templates and tools for documenting their work, much of the work can be automated.

Engineers do not need to think about “Where should I create the documentation?” or “I’m not good at writing. How and what to write?”. They generate a predefined skeleton for the document, fill in the blanks, and it becomes part of the searchable shared knowledge.

Architecture Decision Records (ADRs) are a great example of embedding crucial knowledge directly into the codebase with a shared format and goal.

ADR is a short document that captures:

• Problem

• Important context

• Alternative decisions

• Chosen decision

• Impact and consequences

By keeping these records close to the source code, developers always have the information they need to understand why specific decisions were made, who are the contact persons, and what their impact is.

How to Remove Existing Knowledge Silos

In case knowledge silos have already formed, a strategic approach is needed to create an environment where information flows freely.

All the techniques discussed above can be used, but apart from them, it’s important to understand what has caused these silos to form and how to tackle them as soon as possible:

Organizational Network Analysis

ONA can help visualize communication paths and identify isolated areas. By mapping out these connections, you can find the bottlenecks and areas where knowledge is trapped.

See this document from Delloite that explains Organizational Network Analysis in detail.

Rotating Team Members

One of the most effective ways to break down silos is to rotate team members across different teams. Ideally, the teams or individuals that you have identified via ONA as holding the most knowledge or the ones that are the most isolated.

Benefits:

• knowledge exchange

• building relationships between teams

• sharing best practices in team processes

• gaining a different point of view on the product

Team members should ideally be rotated at the beginning of a new product or a feature development so rotated members get a whole end-to-end experience with the hosting team.

Summary: Take Action

Knowledge silos are a common and costly problem, but they are not impossible to overcome.

Don’t wait for someone else to solve this problem. Start by sharing what you know and documenting your work—improve your team’s knowledge base, host a cross-team knowledge-sharing session, or propose ADRs. When others see you taking the action, they’ll follow.

And never underestimate the power of curiosity. The more questions you ask, the more you learn and the more connections you build across your organization.

Are you looking for a space where you can learn more about software engineering, leadership, and the creator economy and network with other like-minded professionals?

With Dariusz Sadowski, Michał Poczwardowski, and Yordan Ivanov 📈, we’ve just opened the gates to a new Discord community: Engineering & Leadership

Be among the first members and join us for virtual meetups, Q&A sessions, and events:

Join

📖 More From Enginuity

If you don't own your work from start to finish, you’re leaving value on the table. You’re missing out on the chance to truly make an impact. Learn how to incorporate end-to-end ownership into your engineering work:

📣 Top Picks

• A 3-step framework to never get down-leveled in your behavioral interviews again by and in

• When your PM drives you crazy by in

• Become the engineer everyone wants to work with by and in

You can find all three tracks in the main menu of the Enginuity Newsletter.

Subscribe now

End-to-end (E2E) ownership isn’t just another buzzword — it’s a mindset that separates those who deliver real value from those who merely show up.

When you commit to E2E ownership, you take full responsibility for every aspect of your work, from inception to delivery and beyond.

It’s about seeing the big picture, understanding how your piece fits into the puzzle, and ensuring it all comes together seamlessly.

Here’s the hard truth:

If you don't own your work from start to finish, you’re leaving value on the table. You’re missing out on the chance to truly make an impact.

Outcomes: The Only Metric That Matters

Having ownership isn't about moving tasks from “ToDo” to “Done”. It's about ensuring that the work between those points is meaningful and measurable.

Taking ownership makes you feel inherently accountable for the outcomes. It leads you to ensure that what you do moves the needle.

It’s easy to check off a list of tasks, but it’s far more challenging and rewarding to demonstrate the tangible impact of your work.

Measuring outcomes provides you with hard proof of your contribution. It’s no longer about vague statements like:

❌ "I worked on this feature."

It’s about being able to say:

✅ "This feature increased user engagement by 10%."

Your success is tied directly to the results, and that kind of accountability drives real progress not just for you but for the entire team.

Replicate Success; Learn From Failure

Owning outcomes doesn’t mean you’ll succeed 100% of the time. In fact, that’s not the goal.

The essence of E2E ownership lies in the iterative knowledge you gain through both your successes and your failures. You’re not aiming for perfection. You’re aiming for progress.

Success is easy to celebrate. However, the actual value comes from understanding why something worked and how to replicate it in the future. That builds a playbook of strategies that you can apply again and again.

In the context of E2E ownership, failure isn’t a setback—it’s a step forward, an opportunity to learn. Don’t just move on from it. Dissect it. It’s a data point that helps you refine your approach and grow your understanding of what truly drives success.

“5 Whys” Framework

One of the most effective tools for examining root causes of outcomes is the 5 Whys framework.

It’s a straightforward but powerful method in which you ask “Why?” five times to gain deeper insights into the explored topic.

Let’s look at two examples—one for success and one for failure— to see how to use the framework in practice:

Example 1: A New Feature Increases User Engagement

You’ve just delivered a software feature, and it’s been a great success. The feature is driving increased user engagement, stakeholders are thrilled, and you want to understand why it was successful and how to replicate that success in future projects:

1. Why did user engagement increase after launching the new feature?

   1. Because the feature addressed a specific user pain point.

2. Why did the feature address this pain point effectively?

   1. Because we did thorough user research before development.

3. Why was the user research so thorough?

   1. Because we allocated extra time to ensure we understood users' needs.

4. Why did we allocate more time and resources to user research?

   1. Because the previous project suffered because we didn’t validate our assumptions with users.

5. Why did we change our approach from assumption-based to research-driven?

   1. Because we had a retrospective that highlighted the importance of validating assumptions through user feedback.

Learning: This feature's success was rooted in prioritizing thorough user research based on lessons learned from past failures. Allocating sufficient time and resources to user research should now become a standard practice in the development process.

Share

Example 2: A Feature Fails to Drive Expected Results

Imagine you’ve delivered a feature that didn’t meet expectations. The feature failed to drive the anticipated results, and stakeholders are disappointed. Now, you want to dissect the failure and extract valuable lessons:

1. Why did the feature fail to drive the expected results?

   1. Because users didn’t adopt the feature as anticipated.

2. Why didn’t users adopt the feature?

   1. Because the feature didn’t integrate well with their existing workflows.

3. Why didn’t the feature integrate well with their workflows?

   1. Because we didn’t fully understand the user workflows during the design phase.

4. Why didn’t we understand the user workflows?

   1. Because we skipped the user testing phase to meet an aggressive deadline.

5. Why did we prioritize the deadline over user testing?

   1. Because there was pressure from stakeholders to release quickly.

Learning: The failure came from skipping user testing due to deadline pressure. To avoid this situation in the future, the team needs to advocate for the time to do thorough testing, even if it means pushing back on aggressive deadlines.

To extract the best learning, you need to be able to understand the whole end-to-end process to a reasonable degree. To do that, you need to balance your level of expertise across multiple disciplines.

T-Shaped: The Shape of Success

Being T-shaped is about having:

• The depth to solve complex problems in your field

• The breadth to collaborate effectively with other teams and understand different aspects of the business.

To have ownership and, therefore, be T-shaped, you need to get comfortable with the uncomfortable. It includes engaging with elements that are not your primary focus: frontend, backend, infrastructure, observability, product metrics, user research, market research, sales, and more.

You shouldn't aim to become an expert in all these areas but to understand how these components interact, influence each other, and work together to deliver value to users. This builds real expertise.

Being T-shaped in this way means you are a Product Engineer—you balance technical proficiency with a deep understanding of user experience, business strategy, and the operational realities of testing, deploying, and maintaining software.

If you want to learn more about the benefits of the Product engineering mindset, see:

Relationships That Pay Off

Owning the end-to-end process means you naturally collaborate with a diverse group of people across various departments to make that delivery happen. This builds relationships that can last for a long time, even throughout your career.

Relationships are the currency of influence.

The more people you know and the more people who know you, the more impact you can have. When you work closely with teams across your organization, from UX designers to product managers, from customer support to operations, you’re building a network of professionals who:

• know your work

• trust your judgment

• see your commitment to delivering results

When you’ve proven yourself in multiple projects across different departments, people will naturally think of you when new opportunities arise.

Whether it’s a critical initiative, a promotion, or a high-impact team, your network becomes a powerful asset. People trust you to deliver because you’ve shown that you are dependable and competent—time and time again.

Are you looking for a space where you can learn more about software engineering, leadership, and the creator economy and network with other like-minded professionals?

With , , and , we’ve just opened the gates to a new Discord community: Engineering & Leadership

Be among the first members and join us for Q&A sessions, virtual meetups, and events:

Join

📖 More From Enginuity

In the previous article, I shared a method that I use to increase my productivity and impact without increasing stress and time spent on all the tasks that need to be done:

📣 Top Picks

• The 3 Big Mistakes That Almost Cost Me My Promotion (And How You Can Avoid Them) by and in

• What Can I Do to Make You Stay? by in

• Engineer’s guide to convincing your Product Manager to prioritize technical debt by and in

Below, you’ll also find a FigJam template I’ve created as an accompanying resource for this post.

Subscribe now

True productivity is about working smarter, not harder—achieving meaningful results without sacrificing well-being.

It is not about doing everything but about doing the right things in the right way.

In this article, I want to look at productivity from a sustainable perspective. Instead of focusing on how much you can do, we’ll explore how to maximize your impact.

I call this method PEAR (an acronym for its steps), and I’ve been using it for some time now. It has proven incredibly helpful, so I want to share it with you.

Step 1: Prioritize Effectively

Not all tasks are created equal.

80% of your results come from just 20% of your efforts

— Pareto Principle

A small fraction of your tasks will have a disproportionately large impact on your success.

This means that unless you effectively prioritize what you focus on, how well you execute it will not matter. Your work's impact will be sub-optimal. You’ll do more work, but you’ll not do better work.

Multiple methods exist for effective prioritization. Below, we’ll explore two that I find the most useful.

Eisenhower Matrix

The matrix helps you categorize tasks based on their urgency and importance.

This aligns well with the Pareto Principle by helping you focus on high-impact activities.

First, make a to-do list of all the tasks you think you should do. Try to include everything you can think of. After that, split these items into four different categories:

• Urgent and Important — Do First

  • Tasks that are both urgent and critical to your goals.

  • Either they have upcoming deadlines or not making them would have a negative effect on you.

  • These are often your 20% tasks that will yield 80% of your results.

• Not Urgent but Important — Schedule

  • High-impact tasks that aren’t urgent but are crucial for long-term success.

  • These are usually connected to your mid and long-term plans

• Urgent but Not Important — Delegate

  • Tasks that need immediate attention but don't significantly contribute to your goals.

  • Even though they don’t contribute to your overall goals, they still need to be done.

• Not Urgent and Not Important — Eliminate

  • Tasks that don’t contribute to your productivity.

  • Eliminate or minimize these.

ABCDE Method

In this method, you categorize tasks into 5 different categories by ranking them by priority first.

Think about it as a sorted to-do list, allowing you to focus on the most impactful tasks first:

This is the simplest form of addressing your randomly organized to-do list.

The magic is not in the method but in the time you spend thinking about why you need to do what you originally planned.

Resource: FigJam Template

As an accompanying resource to this article, I’ve prepared a FigJam template with even more prioritization methods.

It’s totally up to you which one you’ll choose. Pick one that works best for you and to which you can consistently stick:

Go to FigJam Template

Step 2: Execute with Focus

The Law of Diminishing Returns is an economic principle that states:

After some optimal level of capacity is reached, adding an additional factor of production will actually result in smaller increases in output

In the context of productivity, this means that:

The longer you work on a task without taking breaks, the less efficient and effective you become.

Your focus decreases, your energy goes down, and the quality of your work suffers.

To combat this, the Pomodoro Technique offers a structured approach to maintaining high levels of productivity:

1. Working in short, focused sessions (Pomodoros) — typically 25 minutes

2. After each session, take a 5-minute break.

3. After completing 4-5 sessions, take a longer break of 15-30 minutes.

Regular breaks prevent mental fatigue, allowing you to return to your task with renewed energy and focus.

I use this technique both at work, while programming, and also when writing this newsletter.

My adjustments

I use Pomodoro breaks not just to rest but also as opportunities to step back and reflect on the task at hand.

I ask myself questions like:

• "Is this the best way to approach this?"

• "Am I still aligned with main objectives?"

• "Does it make sense as a whole?"

This thinking helps me stay on track with my larger goals. Give it a try, mainly when you feel you are missing something or something is off while working on a task.

Share

Managing Context Switching

Each time you switch tasks, your brain needs time to adjust to the new context, which leads to increased cognitive load and mental fatigue.

The more frequently you switch, the more time you lose to this adjustment process, making it harder to maintain focus and momentum. It can take your brain 10 minutes to adapt to the new context.

To minimize the negative effects of context switching, it’s important to plan your day so that you don’t frequently need to switch between different tasks.

Batch Similar Tasks

Group similar tasks together and tackle them in one block of time.

For example, if you have several emails to respond to, you can use a specific time block just for email management rather than spreading it throughout the day.

Thematic Days or Half Days

This strategy, also used by people like Steve Jobs or Elon Musk, involves dedicating entire days or large portions of your day to specific types of work.

For instance, mornings could be reserved for deep work, and afternoons might be set aside for creative tasks or planning. If you wear multiple hats at work, this is a great way to juggle them.

By organizing your work into themes, you stay in one mode of thinking for extended periods, reducing the cognitive load associated with switching.

Step 3: Allocate Buffer Time

Work will expand to fill the time allotted for its completion.

— Parkinson's Law

If you give yourself too much time to complete a task, it will likely take longer than necessary. This phenomenon is common in professional and personal life — tasks that could be completed quickly drag on because there's no urgency to finish them.

Therefore, the recommendation is to set clear and realistic deadlines for each task to prevent them from taking more time than they should. Having a target to work towards creates a sense of urgency and helps you stay focused.

But there is a problem:

Most people are too optimistic about what they can achieve within a given time.

This is one of the effects of a phenomenon known as Optimism bias:

• We overestimate our likelihood of experiencing positive events

• We underestimate our likelihood of experiencing negative events

For many of us, it can go like this:

1. Prioritize tasks that need to be done.

2. Schedule time blocks in the calendar or estimate the time needed to complete them.

3. We don't hit our targets because of unexpected delays, lack of energy, or interruptions.

4. We need to move unfinished work to the next day, piling more work on an already busy schedule.

The result is frustration, overworking, and, if it goes on for a long time — burnout.

Account for Unexpected Delays

This law highlights the reality that tasks often take more time than anticipated, even when you try to plan for delays.

• Build Buffer Time

  • Deliberately add extra time to tasks or meetings to account for potential overruns.

  • For instance, if you estimate that a task will take two hours, allocate an additional 30 minutes to an hour as a buffer.

  • This way, if the task takes longer than expected, you won’t be forced to cut into the time reserved for other tasks.

• Leave Extra Time Between Major Tasks

  • Avoid scheduling tasks or meetings back-to-back.

  • Leave some space between major activities to give yourself breathing room to handle anything unexpected without increasing your stress levels.

• Learn to Say No

  • Not every task or request deserves your time. If it doesn’t, practice saying no or delegate the task to someone else.

  • Productivity isn’t about doing as much as possible. It’s about doing what’s most important and doing it well.

  • and recently wrote a great article on this topic with many practical tips: How to say "No" and win back your time as a software engineer

Step 4: Reduce Mental Clutter

People remember incomplete or interrupted tasks better than completed ones.

— Zeigarnik Effect

This psychological phenomenon says that unfinished tasks linger in your mind, causing distractions and making it difficult to focus on other work.

The weight of these incomplete tasks can contribute to a sense of overwhelm and mental clutter, reducing your productivity and increasing stress.

Luckily, there are ways to counteract the Zeigarnik Effect:

Break Down Large Tasks into Smaller Ones

One of the most effective ways to manage the stress of incomplete tasks is to break them down into smaller, more manageable parts.

Instead of having one overwhelming entity, divide it into smaller, actionable steps that can be completed more efficiently.

Completing smaller steps provides a sense of progress and reduces the mental burden of an unfinished task.

Use Task Management Tools

There are many tools for tracking your tasks and progress, such as Notion, Trello, Todoist, and Any.do. Even a simple paper notebook will work great.

These tools allow you to:

• 📈 visualize your progress

• ✅ check off completed tasks

• 📌 keep an overview of what still needs to be done

The goal is to externalize your tasks into some system so you can eliminate the need to remember every detail, which helps reduce mental clutter.

Prioritize Task Completion

To minimize the cognitive load of unfinished tasks, prioritize completing tasks whenever possible.

Instead of juggling multiple incomplete tasks, focus on finishing one before moving on to the next.

Summary

Let’s recap the PEAR method:

1. Prioritize Effectively

   1. Use the Eisenhower matrix or ABCDE method to find 20% of tasks delivering 80% of results (Pareto Principle).

2. Execute with Focus

   1. Use the Pomodoro technique (Law of Diminishing Returns) together with thematic time blocks to keep maximal focus and avoid context switching.

3. Allocate Buffer Time

   1. Schedule realistic deadlines (Parkinson's Law) that account for unexpected delays (Hofstadter's Law) to avoid reducing your productivity and increasing stress.

4. Reduce Mental Clutter

   1. Externalize your planning with a task management tool and prioritize task completion before starting yet another workstream to eliminate mental overwhelm (Zeigarnik Effect).

Remember, it’s not about how great a process you have but how well you execute it. Therefore, the lighter the process, the easier it is to follow and benefit from.

📣 Top Picks

• Build Your Credibility As You Grow by in

• Empathy is a superpower in the engineering industry by in

• How I went from Tech Lead → Engineering Management by in

📖 More From Enginuity

Check my previous article, in which I discuss 3 key lessons engineers can learn from well-written Product Requirement Documents:

• 🎯 Clear Objectives

• 🤝 User-Centric Focus

• 🚢 Prioritization and Trade-offs

You can find all three tracks in the main menu of the Enginuity Newsletter.

As software engineers, we encounter Product Requirement Documents (PRDs) most frequently in the late stages of creation or already when they are ready to be seen by development teams.

We read through them with the intent of understanding what needs to be done and how it can be done from a tech perspective.

In this post, we’ll look at PRDs from a different point of view:

1. What key principles are used in well-written PRDs that make them successful?

2. What lesson can we learn from these principles as software engineers?

3. How can we leverage these lessons in our day-to-day work?

Subscribe now

Clear Objectives

A well-written PRD starts with defining clear and measurable objectives.

These objectives provide a roadmap for the entire project. The aim is that every team member understands:

• the end goal

• the criteria for success

• the overall product vision

🎯 Alignment

Clearly defined objectives ensure all team members can align their efforts towards a common goal.

It removes ambiguity and reduces the risk of diverging paths. Two simple questions can validate every engineering or UX decision taken down the road:

1. Is this decision in alignment with the PRD’s objective?

2. If we do this work, will it bring us closer to achieving the objective?

If at least one of the answers is “no,” a different decision should be made, one that ensures the cohesiveness of the development process.

📊 Measurability

Clear objectives are measurable.

These metrics can be used to gauge whether the team is on track and, if not, make necessary adjustments as soon as possible.

You can use them at any point to assess the current state, progress, and, eventually, success.

At the same time, this approach minimizes the risk of misinterpretation and future errors.

Lesson 1: Communicate Clear Objectives

• Communication with Management:

  • When communicating with management, articulate your priorities, goals, and progress clearly.

  • It should be apparent how your goals and the value you bring as a professional go hand-in-hand with their objectives.

  • wrote a great article about how to make an impact when communicating with management in How to Impress in High-Stakes Presentations, which nicely ties to this topic.

• Communication with Other Teams:

  • Clearly defined objectives make collaboration with other teams or departments, such as marketing or design, more effective.

  • Other teams are more willing to contribute to the project when they don’t need to exert an extra mental effort to understand the problem and how they can contribute to solving it.

• Performance Reviews:

  • Setting clear, measurable objectives for personal and team goals helps in providing concrete evidence of achievements and areas you’ve improved.

  • The result is that performance evaluations are based on objective criteria rather than a subjective judgment of your superiors.

User-Centric Focus

A crucial aspect of well-written PRDs is their focus on understanding and addressing user needs.

Incorporating User personas and Use cases shifts perspective from purely technical considerations to a more user-centric approach, where the end product meets actual needs and provides an excellent user experience.

🤝 Empathy

Developing empathy for users helps create more intuitive and user-friendly products.

Understanding the context in which users operate enables PMs, UX designers, and engineers to anticipate and effectively address their needs.

Use cases—Specific scenarios in which a user interacts with a product to achieve a particular goal.

Analyzing and describing Use cases and testing the product with these use cases helps develop empathy for users.

Putting yourself in users’ shoes allows you to foresee how they will interact with the product and identify potential issues or improvements.

and wrote a great article about why and how to practice “dogfooding” in Using your own product is a superpower.

✅ Relevance

Products designed with a user-centric focus are more likely to meet market demands and succeed.

User satisfaction translates into higher adoption rates and better user retention.

User personas—Fictional characters that represent user types that might use your product in a similar way.

Identifying User personas allows us to use real data to understand what needs the product is going to satisfy and what potential market size it’s going to address.

These personas help visualize the end users and their:

• needs

• preferences

• pain points

This understanding is the core of designing features that genuinely solve user problems.

Lesson 2: Advocate For User Needs

• Professional Growth:

  • Thinking about problems from the user's perspective improves your problem-solving process. The solutions for these problems are usually more innovative and effective, translating into your higher value within the team and the organization.

  • Engineers prioritizing user needs more easily identify opportunities for new product improvements or features. This proactive approach can position them as key contributors to the company's strategic direction.

  • If you haven’t yet, check my previous post on this topic:

• Influence:

  • Engineers who advocate for users can become leaders within their teams or product areas. They can help make decisions that prioritize user satisfaction and drive product success.

  • When proposing new features or changes to management, frame the discussion around user needs and benefits. It makes it more persuasive. Demonstrating user-centricity can help secure buy-in from stakeholders.

  • Implementing a user-centric approach in QA processes ensures that testing scenarios reflect real-world use cases, which leads to products that better meet user expectations.

Prioritization and Trade-offs

As we all know, not all features and requirements can be implemented simultaneously. Therefore, prioritizing features is a critical aspect of creating effective PRDs.

By prioritizing, we can focus on delivering the most valuable aspects first, ensuring that the product meets essential user needs and business objectives within the given constraints.

⏳ Value Delivery

Prioritization ensures that the most important and impactful functionalities are developed first, providing immediate value to users and stakeholders.

Without proper prioritization, the allocated time and budget are not used effectively because people are scattered across a multitude of tasks.

That’s why all available resources should be focused on high-priority features from the beginning.

Delivering these critical features early reduces the likelihood of major setbacks later in the project because potential risks can be identified and resolved as soon as possible.

⚖️ Informed Decision-Making

Effective PRDs include discussions about trade-offs.

It involves making informed choices that align with overall project goals while considering:

• technical limitations (quality, NFRs, …)

• resource constraints (time, money, …)

• user needs (UX, feasibility, …)

The problem is that even though some informed choices are made at the beginning of the project, as time passes, many limitations and constraints also change.

Therefore, both the PRD and the development process must adapt to these changing circumstances and requirements.

They need strategic thinking—weighing the pros and cons of different options to determine the best course of action for the project’s success.

Lesson 3: Think Strategically

• Time Management:

  • Identify and focus on tasks that have the highest impact on your project or personal development. For example, tackling critical bug fixes or key features first can lead to significant improvements.

  • The same principle applies to professional development. When deciding which skills to develop, consider trade-offs—time investment vs. potential benefits. Focus on skills that offer the highest value and align with your long-term goals.

  • wrote an extensive article, which I highly recommend, about different frameworks that can be used for prioritization based on specific prioritization attributes: How to set priorities?

• Negotiation:

  • In PRDs, critical functionalities are distinguished from nice-to-haves. Do the same before entering a negotiation—identify and prioritize your needs and wants. Understand which aspects are non-negotiable and where you have flexibility.

  • At the same time, considering the priorities of the other party aims for a win-win scenario. Understanding the needs and constraints of the other party allows you to propose solutions that satisfy both sides.

Summary

The three lessons from the well-written Product requirement documents:

1. Communicate clear objectives to create alignment, measurable outcomes, and targeted focus.

2. Advocate for user needs to identify new opportunities and create impactful changes both in products and your career.

3. Think strategically by addressing the most impactful and relevant work from the beginning while keeping in mind trade-offs and constraints.

📖 More From Enginuity

In the last installment of the Product Engineer track, I explored the foundation of the product engineering mindset and how it creates opportunities and builds a well-rounded approach to software engineering:

📣 Top Picks

• What a Home Designer turned Software Architect can teach you about Software Design by and in

  • A superb collaboration exploring ways to balance uncertainty and upfront design and how to stay flexible in inevitable times when things change.

• How to propose an impactful improvement to the codebase and own the implementation by in

  • Great tips on finding and proposing changes and improvements that not only have an impact on the codebase and product but also on one’s career development.

You can find all three tracks in the main menu of the Enginuity Newsletter.

Subscribe now

Your career progression requires not only hard work but also the strategic seizing of opportunities that align with your long-term goals.

In this post, you will learn how to effectively identify and capitalize on these opportunities.

I will share 8 tips that have significantly helped me in my career.

Following these tips, we’ll look at two crucial concepts every software engineer should internalize: having agency in one’s career and effective risk management.

This post is the second in the series focusing on career progression. If you haven’t yet read the first one, start here:

🧲 Taking on Challenging Projects

Challenging projects allow you to showcase your skills and your ability to take on responsibilities.

They increase your visibility and demonstrate your value within your company.

The more positive momentum you build, the easier it will be for you to participate in high-visibility projects in the future.

1️⃣ Express your interest

It all starts with expressing your interest in taking on more challenging projects.

Leverage your regular 1-on-1 meetings. One of the main topics in these meetings should be your career growth (not project status updates, which is often the case).

Discuss your current skills and past experiences and how they can be leveraged in future intra or cross-team projects.

2️⃣ Propose solutions

Identify gaps or problems in current projects and propose solutions. Having such initiatives can open opportunities to lead or participate in these projects. For example:

Are customers always complaining about the low performance of your product?

Then, not only solve the performance problem but identify a way to proactively monitor performance—ideally in a way that will benefit multiple teams or the whole platform.

Participating in Cross-Team Projects

Cross-team initiatives offer a unique opportunity to learn from different departments and expand your skill set.

They also help you build a network across the organization, increasing your influence and visibility.

3️⃣ Be a champion for your team

Look for projects, guilds, champion programs, or committees that involve multiple teams, and volunteer to represent your team.

This shows you are willing to collaborate and contribute beyond your immediate role. Examples:

• ✅ a security champion representing your team

• ✅ an active member of a frontend guild

• ✅ a member of a committee for the software engineering hiring process

Use the tip n.1 to discuss it with your manager. I’ve never met a manager who would be unhappy because their subordinate is showing initiative and drive.

4️⃣ Be active in internal chats

Leverage any internal platforms, such as Slack or Microsoft Teams, or meetings where cross-team initiatives are discussed to express your interest and availability.

Sharing your knowledge and participating in discussions can build your internal brand. It’s one of the best ways to get noticed outside of your current team or project, and the best thing is that it does not take much time.

5️⃣ Showcase your contributions

Document your contributions to projects. It helps you showcase your impact and value to the organization.

One of the simplest ways to start is to keep a running document with your accomplishments. recently wrote the article Managing up: 3 things I wish I realized sooner, which describes his process of removing uncertainty by documenting weekly plans and achievements.

At the same time, someone other than your direct manager might be one to decide whether you will participate in the new projects. It might be agreed at higher levels, and having a portfolio of your work, including goals, contributions, and outcomes, can come in handy.

Building Relationships

Building relationships with colleagues from other departments can often lead to being invited to join cross-functional projects.

As with anything in this world, your skills also need to be “marketed”.

And the best way to do it?

6️⃣ Help others

There are many opportunities to help other colleagues every day. Go beyond what is required.

• When solving a customer issue, offer help to customer support folks or customer success managers

• When collaborating with product managers, offer ways how to collect or connect customer data for making better data-driven decisions

• When someone asks for knowledge-sharing, think about who else can benefit from it and schedule a knowledge-sharing meeting or at least record and share it

7️⃣ Connect

Seek out coffee chats or lunch meetings with colleagues you don’t usually interact with.

Unlike other tips, you don’t need to be strategic about it. Do it for the sake of getting to know your colleagues better.

Use these opportunities to learn about their work and the problems they face in their current projects and discuss common interests.

8️⃣ Find a Mentor

Look for mentors within or outside your organization who have followed a career path similar to the one you want.

recently wrote a post that gave great advice about how to pick a relevant mentor: Know Who to Take Advice From

Approach them with specific questions or areas where you need advice.

A mentor can provide valuable advice, share their experiences, and offer insights to help you avoid common pitfalls.

🏹 Having Agency in One’s Career

Having agency means:

✅ taking control of your career

✅ being proactive

✅ making intentional decisions

✅ recognizing that you have the power to influence the outcomes

It’s about recognizing that you are the primary driver of your success and that your actions, choices, and efforts directly influence it.

• Self-Belief:

  • You have the ability to influence your career trajectory. Recognize your strengths, talents, and the unique value you bring to your organization.

  • Reflect on past achievements and how your actions contributed to those successes.

• Ownership:

  • Taking ownership of your career development means setting your goals, seeking opportunities for growth, and being accountable for your progress.

Having agency means accepting responsibility for your choices and actions. It means making decisions and being accountable for the outcomes.

⚖️ Risk Management

Taking on new opportunities is stepping into the unknown, and therefore, it naturally involves a certain level of risk.

The goal is to keep the risk level in a manageable range.

The first step to do so is assessing what can go wrong in three main areas:

• Project-Specific: Risks related to the specific project or opportunity, such as technical challenges, resource limitations, people already involved, or tight deadlines.

• Personal: Even though some projects might look good on paper, getting involved in them might impact your work-life balance because of extra workload or your long-term goals because the project would require you to put them on hold for the time being.

• Organizational: There are opportunities that might affect your standing within the company or your relationships with your colleagues. Even though a project promises, for example, to revolutionize the tech stack within the company, it can backfire if there is not the right time and circumstances to do so.

The main point is not to go to a new project mindlessly. Think about possible red flags and their probability.

When it comes to identifying opportunities for career growth, you don’t want to grab the first shiny thing that comes your way; you want to be strategic about it.

Summary

I hope you’ve found some inspiration in this post.

Fully taking responsibility for your career decisions and outcomes can feel scary in the beginning, but the alternative of aimless floating and not fulfilling your potential should be even scarier.

The first step in building a great plan is understanding that you have agency in your career. The rest then starts falling into place.

Before you go, I’d appreciate if you could leave feedback so I can create content that is valuable for you:

📣 Top Picks

• 10 common ways engineering managers get stuck by in

• How to build good relationships inside and outside your engineering team by in

• Are You Getting the Most Out of People by in

📖 More From Enginuity

In the last post in the Product Engineer track, I looked into how going beyond a traditional software engineering mindset and expanding it with a product mindset can elevate your career:

You can find all three tracks in the main menu of the Enginuity Newsletter.

Subscribe now

There are two types of software engineers—those who plan their professional growth and those who are surprised every year why they haven’t been promoted yet, why they are overlooked, and why no one recognizes their contributions.

This post is for both types.

For those who are already proactive about their career, I want to provide practical tips for fine-tuning your plans and goals.

For those who want to start planning their growth, I want to provide clear steps on where to start.

Aim for career security, not job security. — Gergely Orosz

Understand What You Want to Achieve in Your Career

Navigating your career progression begins with understanding your personal and professional goals.

Start by asking yourself:

• Which aspects of my current role do I enjoy the most?

• Which aspects of my current role do I dislike?

• What skills and experiences do I want to develop further?

• Does my current job allow me to grow in a way I need?

• How do my personal values influence my career decisions?

• Where do I see myself in the next 2 to 5 years?

It’s ok not to have answers to these questions immediately. The important thing is that you start thinking about them.

This process helps you gain clarity about what drives you and what you aspire to achieve.

It’s essential to look beyond the day-to-day tasks and consider the broader impact you want to make. The direction you’ll set for yourself should be both fulfilling and strategically sound.

🪴 Align Your Career Goals with Personal Values and Interests

When your professional path reflects who you are and what you stand for, you’re more likely to stay engaged and resilient, even during challenging times.

The goal is to cultivate a sense of purpose.

Examples:

• Values:

  • If you value innovation, engage with projects that require creative problem-solving and pioneering new technologies.

  • If collaboration is important, find opportunities to work in team-centric environments.

  • If you value independence, build a side project that can, with a bit of luck, start your entrepreneurship career.

• Interests:

  • If you have a passion for data analysis, extend your expertise in data science.

  • If mentoring others excites you, look for roles and projects where you can coach and guide more junior colleagues.

  • If you are interested in UX, engage with a UX design group in your organization.

A Japanese concept called Ikigai (translated as “a reason for being”) refers exactly to this. It describes a set of circumstances that give a person a sense of purpose:

🎯 Define Success for Yourself

Success is not a one-size-fits-all concept. Your definition of success might differ from that of your peers.

And that’s perfectly okay.

Not everyone needs to be a manager, a C-level executive, or an “ivory tower” architect.

Here are some success markers for ICs that you can use to construct your definition of success. It’s not an exhaustive list, but it should give you enough inspiration:

• Technical Mastery: Becoming a go-to expert in a specific technology or domain, continuously expanding your knowledge and skills. This might be needed more than ever in the age of AI, as LLMs are great for surface-level tasks, but connecting dots to unlock deeper-level insight is not their strong suit.

• Business Impact: Leading significant projects that drive meaningful change within your organization or industry. It shows your ability to solve complex challenges and provides satisfaction from helping users solve their pain points.

• Recognition: Gaining acknowledgment from colleagues or fellow professionals validating your contributions and expertise. It can be through publications, speaking, courses, etc.

• Innovation: Creating patents, developing new tools or technologies, contributing to open source—making a dent in your field, and demonstrating your forward-thinking capabilities can be worthy goals.

• Mentorship: It can be deeply satisfying to play a role in mentoring and developing others. Contributing to the growth and success of your peers can be the whole purpose of your professional and personal life.

Self-Assessment As a Guide

Self-assessment is critical. It involves a deep and honest evaluation of your skills, experiences, and aspirations.

By taking the time to reflect on these, you will gain clarity on your career direction and be able to make informed decisions about the future.

The goal is to understand:

• what you have achieved

• what you need to work on

• gaps in your skills or knowledge

• opportunities for growth

Identifying Strengths, Weaknesses, and Opportunities

A thorough self-assessment involves identifying your strengths, weaknesses, and opportunities for growth:

• 💪 Strengths: List your core competencies and areas where you excel. It will allow you to leverage them in your career planning and seek roles that capitalize on what you do best. These could include:

  • technical skills

  • problem-solving abilities

  • leadership qualities

  • education in other fields that can be relevant (marketing, sales, …)

  • unique experiences that set you apart

• ⚓ Weaknesses: Acknowledge areas where you need improvement. It’s not about self-criticism but rather about recognizing areas for development and planning how to address them. This might involve:

  • technical skills you're less confident in

  • soft skills like communication or teamwork

  • lack of self-confidence

  • time management

  • experiences you lack

• 🚀 Opportunities: Brainstorm about opportunities inside or outside your organization that align with your strengths and career goals. You need to proactively seek them out and position yourself for growth. These could be:

  • new projects in your organization

  • creating a side project (open source, micro-startup, …)

  • learning emerging technologies

  • professional development programs

  • networking opportunities

  • courses to address areas you want to improve

Below is a shortened example of such analysis for an IC data science engineer with aspirations to learn more about artificial intelligence and become a tech lead:

Setting SMART Career Goals

By using the SMART criteria (Specific, Measurable, Achievable, Relevant, Time-bound), your goals will be well-defined and achievable.

I’m not going to dive into the concept of SMART—I recommend you read this Atlassian blog post—but let’s look at some practical tips for goal setting.

🏠 Tips for Defining Long-Term Goals

Long-term goals should define your career trajectory and overall vision. They help you stay focused on what matters, guide your short-term efforts, and allow you to say “no” to unimportant things.

1. Define success for yourself: As explained in the first part of the post, this is the most important part of your goal setting. Without it, your goals will be random and have little to no impact on your career.

2. Don’t plan for 10+ years ahead: I recommend your long-term goals be within a 1-5 year time range. Of course, your career vision might be to be CEO at Apple, but that’s not a goal. You need to break down your vision into smaller, achievable milestones—long-term goals.

The best time to plant a tree was 20 years ago. The second best time is now. —Proverb

3. Be open to change: The industry or your personal circumstances may evolve. Look at recent events—the end of ZIRP, AI hype, etc. In such cases, reassess and adjust your long-term goals to keep them relevant.

🧱 Tips for Defining Short-Term Goals

Short-term goals are the building bricks. They provide immediate targets to work towards, keeping you motivated and focused.

Each short-term goal should bring you closer to achieving your long-term goals.

1. Avoid vagueness: Being as clear as possible will help you focus your efforts. For instance:

   1. Vague goal: Improve coding skills

   2. Clear goal: Complete an advanced Python programming course

2. Don’t be too ambitious: Setting overly ambitious goals can lead to frustration and, if you push yourself too hard, even burnout. For example, if you have a full-time job, set a goal to complete one course over a quarter rather than attempting to finish multiple courses and, in the end, not finishing any of them.

3. Prioritize: Not all goals you set have the same urgency or importance. Focus on the goals that will bring the most benefit or are prerequisites for other goals. Decide based on their impact on your long-term career objectives and current job performance.

4. Evaluate and adjust: The benefit of short-term goals is that you can evaluate their effect in a relatively short time (weeks, months). Consider:

   1. Have this goal moved me in the correct direction? If not, what mistake did I make when defining it?

   2. Have I learned something I didn’t know about before based on which to adjust my future goals?

   3. Is my effort sustainable over the long run?

Summary and Part 2

Setting meaningful goals starts with understanding where you are and where you want to go. Self-assessment helps you identify the former, and reflecting upon your values, interests, and long-term vision will help you with the latter.

Set aside time to think about where you are and where you want to go. It’s worth it.

In Part 2, you will learn how to seek opportunities for advancement and how to showcase your achievements for a successful career progression. See you there!

📣 Top Picks

• 5 Writing Tips to Overcome the Blank Page by in

• Probabilistic Thinking - Mental Model by in

• Managing up: 3 things I wish I realized sooner by in

• Optimizing Your "20% Time for Technical Debt" by in

📖 More From Enginuity

Last week’s post was focused on the System design track, and we looked into how Walmart’s data platform leverages Google Cloud Spanner to combine the power of SQL with the scalability of NoSQL:

You can find all three tracks in the main menu of the Enginuity Newsletter.

Subscribe now

As a company expands its operations worldwide, robust, scalable, and highly available databases become a must-have.

For Walmart, a retail giant with 11,000 stores in 20 countries and 265 million weekly active users, managing data efficiently is crucial.

The company has faced significant challenges during peak periods, such as Black Friday and Cyber Monday, when transaction volumes can increase by 100x.

In this post, we look at how Walmart solved its scalability challenges using Google Spanner, a globally distributed and horizontally scalable relational database that combines the benefits of traditional SQL databases with the scalability of NoSQL databases.

Walmart's Need for a Scalable Database System

Walmart processes vast amounts of data related to sales transactions, inventory levels, supply chain logistics, customer interactions, and much more.

Traditional database systems posed several challenges for Walmart. The original data platform used SQL engines such as MySQL and Oracle, as well as NoSQL systems such as Cassandra.

The challenges include limited scalability, high maintenance costs, and difficulties in ensuring high availability and fault tolerance.

Walmart required a database system that can:

• 📈 Scale horizontally

• 📚 Handle massive data loads

• 📉 Ensure high availability with minimal downtime

• 🔗 Provide strong consistency across globally distributed operations

That’s why Walmart decided to adopt Google Spanner.

The Power of SQL With The Scale of NoSQL

Google Spanner is designed to offer the best of both traditional SQL databases and modern NoSQL databases.

Horizontal Scalability

Traditional SQL databases, such as MySQL, face several challenges with horizontal scaling:

• Manual data sharding and cross-shard queries

• Distributed transactions with two-phase commits and increased latency

• Synchronized schema management and data migrations between shards

On the other hand, Spanner natively scales horizontally by adding more nodes. This allows it to handle massive data volumes and high transaction rates.

At the same time, Spanner’s horizontal scalability allows scaling across multiple regions.

For Walmart, even a single minute of downtime can result in a huge revenue loss. Their platform cannot afford to wait multiple minutes for the system to failover to another available data center or region.

Global scalability is a core design factor that enabled Spanner to have >= 99.999% availability SLA and remain operational even in case of hardware or network failures.

Global Consistency

Spanner automatically replicates data across multiple regions. This results in lower latency because users access data replicas nearest to them.

The key strength is its ability to maintain strong consistency across globally distributed data centers. This is achieved through the use of synchronized clocks and advanced algorithms (more about this later), ensuring that all replicas of the data are up-to-date and accurate.

The result is that Spanner supports ACID transactions, ensuring data accuracy and reliability across all replicas. Therefore, it offers full SQL support (also with PostgreSQL interface) with complex queries and analytics.

The Architecture of Google Spanner

The architecture is composed of several main components:

• Spanservers:

  • They store data and handle read and write operations.

  • Each server manages a subset of the data.

  • Data is automatically partitioned and distributed across multiple servers.

• Zones:

  • Data are deployed across multiple geographic regions called zones.

  • Each zone has one Zonemaster, which assigns data to Spanservers

  • Each zone contains multiple Spanservers, providing redundancy and fault tolerance.

• Directories:

  • Directories are the units of data distribution.

  • Each directory is a collection of data that is managed and replicated as a single unit.

To achieve global consistency, Spanner implements a highly accurate time synchronization mechanism using GPS and atomic clocks. This setup is known as the TrueTime API.

TrueTime API

The core function of TrueTime is to provide accurate time information, which is critical for coordinating distributed transactions and maintaining data consistency.

• Clock synchronization

  • Each data center is equipped with GPS clocks that provide precise time information directly from satellites.

  • In addition to GPS, atomic clocks are used as a fallback to ensure time accuracy even if GPS signals are unavailable.

  • Each server periodically synchronizes its clock with the TrueTime service within its data center.

• Time Representation

  • TrueTime represents time as an interval [earliest, latest].

  • earliest and latest are the lower and upper bounds of the current time.

  • This accounts for any potential clock skew, ensuring that all operations are timestamped within a known uncertainty range.

    

• Achieving External Consistency

  • 1️⃣ When a transaction starts, Spanner assigns it a timestamp represented by a TrueTime interval. This timestamp is guaranteed to be consistent across all nodes.

  • 2️⃣ After a transaction is assigned a timestamp, Spanner waits until the TrueTime interval has safely passed beyond this timestamp before committing. This ensures that no two transactions are assigned the same timestamp.

  • 3️⃣ Any subsequent transactions will have a later timestamp, preserving the order of transactions globally.

Key Insights from Walmart’s Digital Transformation

1. 🎯 Aligning Technology with Business Objectives

   1. Walmart ensured that its technology infrastructure could support its strategic objectives and drive growth. They didn’t just go for technology “upgrades” but for transformation that would unlock more business value.

   2. At the same time, Walmart decided to adopt such a robust technology only when its business needs required it. Their original stack (MySQL, Oracle, Cassandra, etc.) handled the original requirements well enough until true global scale and adaptability were needed.

2. 🤝 Building a Partnership with Technology Providers

   1. A strong partnership with Google was crucial for Walmart’s successful adoption of Spanner.

   2. Such partnership allowed them to leverage Google’s expertise through close communication, joint planning, and tailoring the solution to Walmart’s specific needs.

3. 🌱 Focusing on Customer Experience

   1. Walmart’s choice was between building its own data platform on top of its already existing stack or adopting an already existing solution and focusing on its market differentiator.

   2. They chose the latter and decided to focus their resources on delivering new capabilities within their product and to shorten the time-to-market for their new features.

Summary

Walmart’s scale with 11,000 stores and 265 million weekly active users required transitioning from homegrown solutions based on a combination of SQL and NoSQL engines to:

• A unified data platform that combines SQL’s ACID properties with the horizontal scalability of NoSQL.

• A solution that will allow them to shorten time-to-market so they can focus on their business differentiators.

• A technology with high availability and dynamic scalability to sustain 10-100x increases in traffic.

And their choice of Google Spanner has satisfied these requirements.

Disclaimer: This case study showcases industry best practices based on publicly available information. The specific underlying structure of Walmart might be different.

📖 More From Enginuity

Last week’s post was focused on the Product Engineer track and focused on the role, benefits, and key responsibilities of Product engineers and how to adopt the Product engineering mindset:

📣 Top Picks

• How I plan my week as a Senior Engineer in Big Tech by in

• Margin of Safety - Mental Model by in

• The Apprentice, The New Boss, The Successor and The Pioneer by in

You can find all three tracks in the main menu of the Enginuity Newsletter.

In recent years, the role of software engineers has undergone a significant transformation.

It's becoming increasingly important to transition from focusing solely on "how" to create software to becoming a well-rounded Software Product Engineer who understands the "what" and "why" behind products.

This post discusses how this mindset offers significant career growth opportunities and job satisfaction. We’ll explore:

• The role of the Product Engineer

• Benefits of the Product Engineer mindset

• Key responsibilities of Product Engineers

• How to adopt the Product Engineering mindset

Subscribe now

The Evolving Role of Engineers

As engineers, we are expected to write and maintain code, fix bugs, and ensure that systems operate correctly.

The main focus has been on the "how" — how to design, how to develop, how to deploy, how to maintain software.

But the landscape is changing. Factors such as:

• 🏎️ Popularity and proven efficiency of cross-functional agile teams

• 🤖 Rise of GenAI providing ever more powerful assistance in software development

• 🧑‍💻 Multiple rounds of layoffs in many companies indicate product development can be done with acceptable outcomes with fewer people

Engineers are now expected to participate in product strategy, understand market needs, and align their work with business goals.

Understanding the "what" and "why" behind a product is just as important as knowing how to build the product.

Engineers who understand customer needs and business contexts can contribute more effectively to product innovation and success. And organizations have started to recognize them.

This brings us to the role and mindset of a Product Engineer.

Understanding the Product Engineer Role

A product engineer combines deep technical expertise with a strong understanding of product strategy, customer needs, and business goals.

Product engineers are involved in a product's entire lifecycle, from ideation and design to development, launch, and iteration.

Key Responsibilities

🎯 Product Strategy: Collaborating with product managers to define product vision, goals, and roadmaps.

🤝 Customer Empathy: Understanding and advocating for the needs and pain points of end-users.

📈 Business Alignment: Ensuring that engineering aligns with business objectives and contributes to overall company success.

🪢 Cross-functional Collaboration: Working closely with designers, marketers, sales teams, and other stakeholders to ensure effective product development and delivery.

♻️ Iterative Development: Continuously improve the product based on user feedback and market changes​.

Benefits of Becoming a Product Engineer

Career growth

A skill set that includes technical hard skills, product management, business understanding, and customer centricity makes you a more versatile and valuable employee.

Product engineers are well-positioned for leadership roles and even C-level positions in tech-driven companies.

Their ability to bridge the gap between engineering and business strategy allows them to influence what products are developed and why, further increasing their visibility.

Market Demand

The ability to understand and influence both the technical and business aspects of product development makes product engineers highly sought after in the job market.​

The U.S. Bureau of Labor Statistics projects that employment in software development will grow by 25% from 2022 to 2032, much faster than the average for all occupations.

This growth is driven by the need for innovative software products.

Job Satisfaction

Being involved in the entire lifecycle of a product — from the initial idea and design to launch and iteration, and seeing the impact of your work on end users can lead to a deeper sense of purpose.

This alignment with the overall mission and goals of the company can be very motivating.

At the same time, working closely with other teams and participating in company strategy provides new challenges and learning experiences.

Adopting the Product Engineering Mindset

1️⃣ Shifting from a purely technical mindset to a holistic product-focused mindset can be challenging and requires a shift in how you approach development:

• Product Discussions: Actively participate in product meetings and discussions to understand different viewpoints and customers’ personas.

• Broader Perspectives: Start thinking about the product from the user's and business's perspectives, not just the technical implementation. Here are some questions you can ask yourself:

  • Who are the primary users of this product?

  • What are the main problems or pain points these users face?

  • Are there ways to align my work more closely with these goals?

  • Who are the key stakeholders involved in this product's development?

  • How does our product differentiate itself in the market?

  • What metrics and data can I use to measure the product’s success?

2️⃣ Continue improving both your technical and product skills:

• Stay Up-to-Date: Read industry blogs, attend webinars, and take new courses to stay updated with the latest trends in product management and technology. Here is a short list of superb product-focused newsletters for follow:

  • Product Growth

  • Lenny's Newsletter

  • Creator Economy by Peter Yang

  • Product for Engineers by

• Experiment: Continuously improve by experimenting with new tools and methodologies and iterate based on feedback and results.

3️⃣ Balancing technical and product responsibilities requires careful prioritization and time management:

• Prioritize Tasks: Use prioritization frameworks like the Eisenhower Matrix to manage your tasks and focus on high-impact activities.

  • You can check ‘s post about delegation using the Eisenhower Matrix.

• Time Management: Allocate specific time blocks in your calendar for technical work and product-related activities to ensure adequate attention in both areas.

Share Enginuity

Now is the time to start developing the skills needed to become a product engineer.

Invest in your education and training, gain practical experience through projects and mentorship, and build a strong network inside and outside of your company.

📖 More From Enginuity

Last week’s post focused on the System design track. We looked into data engineering in DataDog and how they combine Apache Spark, Kubernetes, and YuniKorn to process a vast amount of data daily:

📣 Top Picks

• How to get promoted: Lessons from an ex-Amazon VP by in

• Is it Still Worth it for Big Tech? by in

• How to keep up with all the digital content by in

But how has DataDog scaled its data engineering platform to handle trillions of data points per day without compromising speed or reliability?

In this article, we look into the technical details that allow DataDog to maintain its scale, providing a detailed look at:

• What is Apache Spark, and how it works

• Powerful combination of Apache Spark and Kubernetes

• Optimizations for processing hundreds of terabytes of data daily

⚡ The Power of Apache Spark in DataDog’s Infrastructure

Apache Spark is a powerful, open-source distributed data processing engine known for its speed and ease of use.

It supports various workloads, including batch processing, streaming, machine learning, and interactive queries, making it a perfect fit for DataDog’s large-scale data processing needs.

Subscribe now

Key Features of Apache Spark:

• In-Memory Processing: Spark's ability to process data in memory significantly reduces the time required for complex data operations. It retains intermediate results in memory, which is much faster than the disk-based intermediate storage of Hadoop MapReduce.

• Rich API: Spark provides high-level APIs in Java, Scala, Python, and R, making it accessible for developers with different expertise.

• Fault Tolerance: Through lineage information, Spark tracks the path of data end-to-end, from source to destination. This can help it to recover from faults, ensuring data integrity and reliability.

• Scalability: Spark’s architecture can efficiently handle growing volumes of data, which is crucial for DataDog’s scale.

DataDog relies heavily on Apache Spark for its batch processing capabilities, and using Spark, it processes hundreds of terabytes of data daily.

🏗️ Spark’s Architecture

Spark’s distributed architecture consists of:

• Driver Program coordinating tasks

• Cluster Manager allocating resources

• Multiple Executors running the tasks

Data in Spark is represented via a Resilient Distributed Dataset (RDD), a fundamental data structure that models a distributed collection of objects. RDDs are partitioned and distributed across the cluster with three main goals:

• Optimizing parallelism

• Providing fault tolerance

• Minimizing network bandwidth usage

✨ Running Spark on Kubernetes Clusters

DataDog uses Kubernetes to deploy and manage Apache Spark. Kubernetes is an open-source container orchestration platform that allows DataDog to manage resources efficiently across multiple cloud environments.

Before Kubernetes, DataDog integrated directly with:

• Azure HDInsights

• GCP Dataproc

• AWS EMR

By leveraging Kubernetes, DataDog decreased maintenance costs across different cloud providers and gained more control over specific resources allocated to tasks. Kubernetes now acts as a Cluster Manager in DataDog’s Spark architecture:

To further optimize resource management, DataDog uses Apache YuniKorn, a resource scheduler for Kubernetes.

YuniKorn improves resource utilization by efficiently scheduling Spark jobs, preventing resource deadlocks and starvation, and enabling gang scheduling.

YuniKorn’s gang scheduling ensures that all necessary resources for a job are available before execution, minimizing idle time and resource wastage.

⏱️ Optimization Techniques for High-Volume Data Processing

Scaling to hundreds of terabytes per day needs more than powerful tools. It requires strategic optimization to ensure both performance and cost efficiency.

We discussed one such optimization in the previous section — strategic resource allocation in Kubernetes using Apache YuniKorn. Now, let’s look at how DataDog optimizes data transformations in Spark.

Spark Shuffle Operations

A Spark shuffle operation is a mechanism that redistributes data across the cluster. During a shuffle, Spark transfers data from multiple partitions to reorganize and group records based on keys.

While powerful, shuffle operations are resource-intensive and can be a performance bottleneck because the shuffle time grows exponentially with the size of the data being shuffled. At the same time, large datasets can bring the following challenges:

• Out of Memory (OOM) Errors: Large shuffle operations can exhaust the memory available to executors, leading to OOM errors.

• Disk Swapping: When the data size exceeds the available memory, Spark may need to swap (spill) data to disk, which significantly slows down the process due to increased disk I/O.

• Network Overhead: Transferring large amounts of data across the network can introduce significant latency and congestion, further amplifying performance issues.

Choosing Optimal Shuffle Partition Size

To mitigate the challenges of large shuffle operations, DataDog focuses on optimizing the size of shuffle partitions. They found that aiming to keep shuffle partitions around 100s MiB works best for them. This size is a sweet spot that balances memory usage and performance:

• 💾 Memory Fit: Partitions of this size can typically fit into memory, reducing the need for disk swapping and avoiding OOM errors.

• 🗃️ Improved Compression: Well-sized partitions can be compressed more efficiently, reducing storage and transfer costs. Too small partitions do not compress so efficiently.

• 🛜 Reduced Network Requests: By not having unnecessarily small partitions, DataDog minimizes the number of network requests required during shuffle operations, reducing overall latency.

Share Enginuity

Tungsten format and DSL

Tungsten is an internal row-based format in Spark that minimizes memory overhead by storing data in a compact, binary format, avoiding creating bulky Java objects.

Tungsten supports off-heap memory storage, further enhancing performance by reducing garbage collection overhead.

There are two main ways how to perform operations on top of Tungsten format:

• DataFrame DSL (Domain Specific Language) transformations

• Lambda transformations

DataDog prioritizes DSL transformations, which operate directly on Tungsten-encoded data and are more efficient than Lambda transformations that require converting data into Java objects.

✏️ Summary

DataDog successfully manages to ingest and process terabytes of data daily by leveraging advanced data processing techniques and maintaining a robust data engineering platform. The core architecture pieces that allow for this scale are:

• Apache Spark

• Kubernetes cluster management

• Apache YuniKorn resource scheduler

• Optimizing data partition size and shuffle operations

• Off-heap data in Tungsten format and transformations via DataFrame DSL

Disclaimer: This case study showcases industry best practices based on publicly available information. The specific underlying structure of DataDog might be different.

📖 More From Enginuity

In the last issue, we looked into how Netflix scaled its Content Delivery Network to serve hundreds of millions of customers:

📣 Top Picks

• The Ultimate Guide to Becoming a Rockstar Product Engineer by and in

• Senior Engineer to Lead: Who to promote and how to train them by in

• How Instagram Scaled to 2.5 Billion Users by in

• 5 Domains to Address While Leading Teams by in

Netflix has been a pioneer in the online video streaming industry for years. It has not just exemplified but redefined the principles of scalability and performance through its innovative Content Delivery Network (CDN).

This case study explores how Netflix's CDN supports terabytes of data per second and the technologies that enable such impressive scalability.

What is a CDN

A Content Delivery Network (CDN) is a distributed server network designed to deliver content, such as static web pages, images, and videos, to users efficiently and reliably.

Subscribe now

Breakdown of how CDNs work:

• Distributed Server Network

  • CDNs consist of multiple servers, also known as edge servers, distributed across different locations worldwide. Edge servers store copies of content from the origin servers where the content is hosted.

• Content Caching

  • When a user requests content from a service, the CDN directs the request to the nearest edge server. This server delivers the cached content to the user. This way, the distance between the user and the data is as short as possible, which improves load times and reduces latency.

• Load Balancing

  • CDNs use load balancing to distribute traffic across multiple servers to prevent any single server from becoming overloaded. This technique is very useful for handling load during peak traffic times.

• Geographic Proximity

  • By placing servers in various geographic locations, CDNs reduce the physical distance between the user and the server. This minimizes latency and is particularly important for services with users spread across different regions.

• Redundancy

  • CDNs store multiple copies of the same content across different servers. If one server goes down, another can take over, ensuring uninterrupted service.

Netflix’s Need for a Custom CDN

Initially, Netflix outsourced its content delivery to major CDN providers like Akamai. However, as its number of customers grew, and therefore, demand for streaming grew, these providers struggled to keep pace.

Netflix had to focus on how to improve:

• cost 💵

• scalability 🚀

• user experience 🤩

That’s how a custom CDN, Open Connect, was born with the goal of creating a robust and scalable network.

Architecture Designed for Scalability

The Open Connect CDN is specifically tailored to Netflix's needs of transferring terabytes of data at high speeds all across the globe. Its success is achieved mainly through the following design concepts:

Edge Locations

Netflix picked locations for its Edge servers at strategic points around the globe. These locations are typically within the data centers of Internet Service Providers (ISPs) or at Internet Exchange Points (IXPs). IXPs are nodes on the boundaries of two (or more) ISPs when the traffic flows from one ISP provider to another.

Server Infrastructure

Netflix chose a combination of different storage solutions to handle its large content library and high data transfer demands:

• Storage Servers: These servers are capable of holding up to 360 TB of data and generating approximately 96 gigabits per second (Gbps) of throughput. They store large volumes of content, ensuring availability even during peak times.

• Flash Servers: Equipped with solid-state drives (SSDs), these servers can hold up to 24 TB of data but can deliver up to 190 Gbps of throughput. They are used for delivering high-demand content quickly.

• Global Servers: For smaller ISPs and emerging markets, Netflix uses a downscaled version of storage servers offering up to 120 TB of storage and 18 Gbps of throughput.

Data Flow Optimization

Data flow optimization involves managing how data is transferred from central to edge servers. This process ensures that data is delivered efficiently, minimizing latency and avoiding bottlenecks.

Distribution of new content from the origin servers to the edge servers can be done outside of the peak hours, for example, during nighttime when the ISP traffic is lower, so transferring a large amount of content does not cause latency or reliability issues within the network.

Share

The scalability of Open Connect is achieved via:

• Horizontal Scaling 📈: Adding more servers across various locations to handle increased load.

• Efficient Caching 🗂️: Storing frequently accessed content at edge servers to reduce redundant data transfer.

• Proactive Monitoring ✅: Constantly monitoring network performance to identify and mitigate bottlenecks in real-time.

Summary

CDN is an essential infrastructure component for any online service aiming to deliver content quickly, reliably, and securely to a global audience. By leveraging:

• distributed servers

• content caching

• and load balancing

CDNs ensure optimal performance and smooth user experience.

Disclaimer: This case study showcases industry best practices based on publicly available information. The specific underlying structure of Netflix might be different.

📖 More From Enginuity

In last week’s issue, we looked into how Slack was able to build an AI solution with customer’s data privacy as a first-class citizen. Give it a read:

📣 Top picks

• Pt.2: How to get more done when things are crazy by in

  • Don’t forget to check also Part 1

• Deadlines are not so bad, I’ll tell you why by in

• GraphQL Intro 101 (part 2) by in

  • Don’t forget to check also Part 1