6 Ways Quality Drives Product Development at AppDirect

Updated from original November 21, 2022 version

At AppDirect, our commitment to providing the best software possible is one of AppDirect's defining characteristics and we believe that ensuring the highest quality for our products is everyone’s responsibility—not just the job of our quality engineers.

We use the Net Promoter Score (NPS) as our baseline metric to measure the quality of service we deliver to our customers and how quality affects their level of satisfaction. Used widely as a key metric, the NPS score is based on a survey that asks customers to rate how likely they are to recommend a product.

Quality is intricately woven into every product

With quality and customer satisfaction as key drivers in our product development approach, the AppDirect Quality Center of Excellence (QCoE) team is committed to advancing AppDirect's quality culture and fostering continuous quality improvements. The team establishes quality standards and trains all our engineers to ensure a consistent understanding and application of the standards across the company. We use a combination of industry standard testing practices and cutting-edge test tooling and processes for all teams to follow. This also allows us to scale consistent quality across the entire organization.

In this blog

In this blog, we’ll share the 6 primary mechanisms our Quality Center of Excellence (QCoE) team uses to ensure our SaaS products meet the highest quality standards:

1. Automation-first mindset

2. Three-environment deployment approach

3. AppDirect testing pyramid

4. Quality-driven software development lifecycle process

5. CI/CD (Continuous Integration/Continuous Deployment)

6. Feature release

1. Automation-first mindset

AppDirect utilizes an automation-first approach.

• For product developers, this means ensuring unit and component testing is in place before code is merged.

• For quality engineers, it means that integration and end-to-end testing has to be planned and implemented, and the definition of done needs to be determined at the outset.

• For each scrum team, it means the team has automation in place within the three-tier deployment pipeline to verify every change that gets merged.

Having a robust automation suite ensures that manual testing is focused on corner cases, sanity testing, and the user experience.

2. Three-environment deployment approach

AppDirect uses a three-environment approach in its release pipeline—integration, staging, and production. Each environment has inbound and outbound testing requirements, ensuring that all code changes flowing through pass our test automation coverage and product acceptance criteria.

As the code builds, unit, component, and narrow integration tests are run as part of continuous integration. Code linting and security scans are also run to ensure no new vulnerabilities are introduced, and poor coding practices are avoided.

1. Once these tests are passed, the code is deployed to the integration environment, where end-to-end and broad integration tests are run.

2. The code is then promoted to the staging environment where end-to-end, broad integration tests and system integration tests are run.

3. The code is finally promoted to production behind the release flag where final sanity checks are performed.

3. AppDirect testing pyramid

The AppDirect testing pyramid follows industry standards closely. It outlines exactly which tests should be automated and where they should be run.

The testing pyramid aligns teams and processes across the entire AppDirect organization by defining supported testing types, as well as when, where, and how to execute them.

The specific definitions are as follows:

Automated tests

The primary set of tests we rely on for day-to-day operations are all automated and run as part of every code merge. These tests are run for every team, for every merge, and in some cases, are run across multiple environments:

• Unit and component tests—The most numerous tests because they run the fastest, these tests verify that each method produces the expected results and that the collection of methods work together to provide the needed product functionality.
• Narrow integration tests—Conducted in CI using test doubles, these tests validate the interactions and integrations with its dependencies.
• Contract tests—Contract tests are specialized cross-team component tests that verify if the provider interface matches the consumer expectations.
• Broad integration tests—Run in a test environment using real services to validate the system, these tests provide the desired functionality with all dependencies and integrations working together.
• End-to-end UI tests—Often referred to as E2E tests, these cover critical workflows through the AppDirect platform. The E2E tests ensure a collection of services work together to accomplish a workflow goal.
• Security tests—AppDirect adheres to the Security Development Lifecycle (SDL) process to ensure security is integrated into each stage of Software Development Lifecycle (SDLC).
  • The SDL process helps developers build more secure software and address security compliance requirements while reducing development cost.
  • The information security team (InfoSec) security review consists of three major stages:
    • Security design review (SDR)
    • Static application security testing (SAST)
    • Dynamic application security testing (DAST)
• System integration testing—Often referred to as SIT tests, these cover critical user journeys through the AppDirect platform. SIT ensures the entire stew of services, databases, UIs, third party services, and background jobs function in a production-like environment.

On-demand tests

In addition to the testing we conduct using the standard test pyramid, we also run the following additional on-demand tests on AppDirect products, as required:

• Performance tests—Performance tests measure how well a service operates under production-like data volume and web traffic. They run on a dedicated load environment where services are measured by their p95 and p99 response times.
• Accessibility testing—Often called A11Y tests, these define A11Y best practices that are followed during SDLC. This testing ensures the AppDirect platform conforms to WCAG AA compliance standards. For this, we heavily rely on the Google Lighthouse tooling.
• Cross-browser and visual regression testing—AppDirect platform leverages both cross browser and visual regression testing (VRT). Cross-browser testing ensures web pages are functional across different browsers and free of HTML, CSS, and JavaScript issues. VRT validates that changes made to the platform do not negatively affect the visual appearance of the UI.
• Pen testing—The AppDirect InfoSec team utilizes standard as well as in-house tooling to ensure our platform is and remains secure from internal and external threats. Penetration Tests are run ad-hoc and any issues identified are given the top priority for development to address.

4. Quality-driven software development lifecycle process

At AppDirect, we know that quality is one of the top features customer are looking for. That's why it's an integral and essential consideration at every step of the software development lifecycle (SDLC). The following diagram showcases how we put this into practice—from planning to feature development, through to feature release.

Let's look at each process in depth.

Feature planning

New features are prioritized by the product organization and the domain product owner before being presented to the scrum team for discussion. The planning team discusses features to give the team a better idea of the functionality required. Together, developers, quality engineers, and the product owner define the acceptance criteria and discuss and document design considerations, integrations, functional and non-functional testing (performance, security), and potential risks associated with the feature.

Next, the team documents and estimates user stories for feature development, test automation, and non-functional testing (if necessary). The feature ticket is not closed until all stories are complete. This is where the larger product vision is broken down into small, quickly deliverable parts that can be independently tested and shipped into production with automation coverage.

Sprint planning

At AppDirect we follow SAFe agile practices, with sprint planning being an essential component. During the sprint planning meeting, the features that are ready to be worked on are added to the sprint plan, and the developers and quality engineers take stories to work on.

Development

Developers are primarily responsible for writing unit, component and narrow integration tests as part of feature development. These tests are run as part of the continuous integration workflow whenever code is committed to the branch, and primarily serve the purpose of assisting the developer rapidly iterate on the codebase with confidence they are not breaking existing flows. Having a high level of coverage here also ensures future development does not break their features unknowingly. Developers are encouraged to maintain 80 percent or higher unit test coverage using static code analysis tooling.

.
Quality engineers are responsible for writing broad integration and E2E UI tests as part of feature delivery, and also build a testing strategy and execution plan around performance testing the feature as a whole.. A separate suite of smoke tests is also created and run as part of a continuous deployment workflow which runs after every new deployment. Smoke tests verify that a service is working and ensure there are no show stoppers in the build being deployed. In essence, it’s a small and rapid regression test suite of the major functionalities is run before live traffic is directed to the updated service.

The security team (InfoSec) is responsible for the SDL process.

• The scrum team submits an initial risk assessment, after which the Infosec team starts the security design review (SDR), which is an in-depth analysis of application design.

• The team then conducts dynamic application security testing (DAST) to identify security flaws and vulnerabilities through penetration testing.

• Static application security testing (SAST) happens as part of a continuous integration workflow, with automated deployment blocking in the event that security vulnerabilities with a certain severity threshold are detected.

In order to reinforce and improve the security and quality of the software we deliver, all new services must successfully pass the SDL security review process before being deployed in production.

5. CI/CD (continuous integration/continuous deployment)

At AppDirect, we use continuous integration and continuous deployment (CI/CD) to ensure we can deliver the highest-quality products quickly, securely, and efficiently.

Continuous integration

A continuous integration workflow allows us to implement both quality and security checks on every code check-in. A developer opens a pull request once they wish to merge their code changes. Quality checks for unit, component, and narrow integration tests are run to provide rapid feedback. For security checks, static code analysis is done on the pull request and blocks the merge if security vulnerabilities are detected.

Continuous deployment

AppDirect follows continuous deployment practices to deliver small incremental changes to production as quickly as possible. Deployment happens sequentially through integration and staging environments before changes are deployed to production. Smoke, broad integration, and E2E UI tests provide the necessary quality checks for each environment before code is promoted to the next environment. AppDirect also utilizes release flagging to ensure we can continuously deploy new code into production without impacting users unexpectedly.

6. Feature release

Once the feature is built, the next steps are to deploy the feature to the marketplace.

After the code for a feature is deployed, we use release flags to quickly and easily enable or disable it with no need for additional code changes. With release flags, AppDirect delivers every new feature to production incrementally by releasing it in three phases: preview, early adopter (EA) or general audience (GA). New features are enabled in test environments first for quality sign-off before they’re enabled in production.

Release sign-off

Once a week, the product team, in collaboration with engineering, schedules the next set of release flags to be enabled. Even though most code changes are deployed into production automatically once they pass all automated quality checks, release flags allow us to control when the changes are made available to customers. Prior to enabling a release flag for everyone, quality engineers and product managers run tests in production by enabling the features first in production sandbox environments.

Monitoring

The quality teams’ job is not finished after the features are delivered and enabled on production. We have to ensure everything is working, and continues to work over time. AppDirect makes heavy use of DataDog to continuously monitor all of our environments. While many may consider software testing and production monitoring two different problems, at AppDirect we treat them as the same problem at different stages of development. After all, what is a monitor if not a test case that is being asserted every few seconds?

Engineering teams also leverage real user monitoring (RUM) and application performance monitoring (APM) to view and resolve factors affecting platform performance and response codes. These tools enable us to maintain system availability, enhance the user experience, and improve service performance and response times. For each service, monitors are set up to continuously check metrics and scrum teams can be alerted when critical shifts occur, allowing them to take action quickly.

Conclusion

The approaches we’ve described provide AppDirect with a strong quality foundation. Our QCoE approach and processes play a crucial role in defining the quality vision and maintaining a quality culture. QCoE ensures scrum teams adhere to quality processes, it produces in-house training to upskill quality engineers, and it defines quality key performance indicators (KPIs) to gauge whether quality is trending in the right direction.

This article was written through a collaboration between AppDirect’s Brian Albright, Darpan Shah, Madhav Bhogaraju, and Rahul Jain