How do we measure DX?

Measuring the DX offered by our tools enables us to gauge where we are and take the necessary steps to apply the DX principles discussed earlier. There are different frameworks and methods available to measure DX. The following is a summary of these methods:

Measuring UX characteristics

Since we initially used UX characteristics to define DX, metrics for measuring DX can judge the availability of the following UX characteristics in our products.

Usability

Usability tells how easy the product is to install, set up, and use. The product should be intuitive, consistent, and performant to ensure usability. Most methods generally recommended for measuring usability involve getting feedback from real users by showing them the product or prototype. These include:

• Guerrilla tests - ask a random developers to use the product and note down the issues found and sentiments observed.
• Lab tests are used to gather in-depth information by calling a group of developers to a usability lab where a moderator asks them to complete specific tasks. Metrics such as task times and completion rates may be used to quantify the results.
• Remote usability tests are unmoderated tests where developers may complete the tasks remotely from their usual work environment.
• Interviews involve interviewing a group of people based on a script. The interviewer collects qualitative data (e.g., how did it feel) and quantitative data (e.g., time to complete the task). This method may be simplified through a survey form to answer specific questions relevant to the product.

When conducting usability testing, be sure to follow the guidelines on recruiting users, so that you recruit users who best represent your user base.

This can help to avoid effects of possible bias due to factors such as technical competency or location (remote vs office) of developers.

Accessibility

Accessibility checks how accessible the product is for different roles which can use it. Developer journey maps and the challenges encountered in executing the steps in the map may be used to measure the accessibility of the product.

Here are three ways to identify friction and understand if the defined journeys are working optimally:

1. Ask the community about challenges and improvement opportunities.
2. Conduct friction audits to log the friction points at different touchpoints in the journey.
3. Analyze the data collected and assign metrics to each data point for evaluation.

Accessibility testing is especially relevant to critical developer journeys.

Findability

Findability defines how fast a user can find what they need in the product. It could mean finding the product itself, finding a specific feature, or the resolution or workaround for an issue.

The primary way to validate this is by measuring the time used by the user to find something in the product. We can use laboratory tests with screen recording for this measurement.

Credibility

Credibility is about delivering stable products with the correct functionality. Measuring credibility is usually about measuring bugs in the software. Traditional manual or automated functional testing methods may be used to test credibility.

In addition to measuring issues within the software, we should also consider process bugs that arise due to developers' lack of understanding of how something works.

Usefulness

Usefulness addresses the specific needs of developers. The measurement of this characteristic ensures that we have addressed use cases executed most often by developers.

Ensuring that critical developer journeys are fulfilled without accessibility or credibility issues can help determine the product's usefulness and value.

Value

This characteristic indicates the value our products provide to developers. Data that is relevant to the value provided is usually easy to gather. Some metrics that could prove helpful are.

• The number of users signing up or paying for a particular product
• The number of users using it more than once
• The number of users recommending it to others.

Teams can also define the product's value proposition and ask users how the product meets this goal.

Other Methods

The HEART framework

Google's HEART framework is recommended for measuring UX for web applications on a large scale using automated means. This framework can be adapted to measure DX directly or if you are already using it to measure UX, you might be able to determine if implementation of a DX strategy triggered a change in UX.

Developer Productivity

The immediate goal of having a DX framework is to improve the productivity of developers and teams who use our tools. Thus, in addition to improvements in UX, developer productivity also makes a logical metric for measuring DX. We can measure the improvement in productivity that teams using our products experience over time.

Note the use of teams in this context. Traditionally used metrics like lines of code or hours worked though convenient do not tell the true story of developer productivity. They discount the "negative work" that is sometimes done by developers - work that does not add value to the project.

Similarly, individual developers' contributions may or may not be relevant to the actual completion of the project. Team velocity or the number of features shipped by the team gives a more accurate picture of teams' productivity. Thus, improvement in team velocity after shipping changes to developer tools or APIs can be used to measure DX.