Platform Insights: Balancing Agility and Efficiency with Confluent's Kafka Data Streaming

If you’ve ever wondered what's the foundation of a world-class subscription commerce platform, it all comes down to—you guessed it—platforms. Or as Tom Vogel, AppDirect’s Director of DevOps & Cloud Platform Engineering, puts it, "it’s platforms, built on platforms, built on platforms."

In his role, Tom oversees the internal developer platform the AppDirect engineering team uses to build the AppDirect marketplace platform. In this blog, we’ll highlight some of the responsibilities of the DevOps platform engineering team, and touch on its impact on the AppDirect platform our customers use today, making it the leading commerce platform for selling, buying, and managing recurring technology services. We’ll also highlight a recent interview conversation with Tom and Confluent. 

What is Confluent ?

Confluent is a cloud-based data streaming platform that helps users access, store, and manage data in real time. The Confluent platform is built on top of Apache Kafka® (an open-source platform that stores, processes, and analyzes streaming data in real time).

What is DevOps? What is platform engineering?

DevOps is a combination of development (Dev) and operations (Ops) that aims to improve the speed and quality of software development. Platform engineering is a subset of DevOps that focuses on designing, building, and managing the infrastructure that supports efficient software development.

The DevOps platform engineering team is involved in creating an environment that streamlines software development processes to ensure the efficient and reliable delivery of software products. He stresses that a cornerstone of DevOps culture is that operational excellence is the responsibility of all engineering teams. 

Reducing the 'cognitive load' for all teams

However, not all teams are equipped to be experts at server operations, or security compliance. As such, by leveraging an internal developer platform, Tom has reduced the “cognitive load” for all teams, allowing them to focus on the core outcomes and KPIs of their product.

Tom was recently interviewed about his experiences and expertise in using Confluent’s data streaming application in building AppDirect’s developer platform. He shared his insights in creating this platform and ultimately its impacts on the larger AppDirect marketplace development platform.

In the interview with Confluent, Tom credits several strategic pillars that have contributed to the success of the AppDirect developer platform and improved customer outcomes.

• Early adopter of Kubernetes: Since 2016, AppDirect has adopted Kubernetes, an open-source platform for automating the deployment, scaling, and management of containerized applications, efficiently managing large-scale applications.

• Microservices architecture: AppDirect uses microservices, smaller independent services that work together, making the system more flexible and easier to manage.

• DevOps: By adopting DevOps practices, tools and culture, AppDirect combines software development and IT operations into autonomous end-to-end delivery teams, in order to ensure faster and more reliable software delivery.

• Self-serve internal developer platform engineering: AppDirect has created internal software development platforms, which build, test, and deploy applications independently, speeding up development, providing governance, ensuring reliability, and fostering innovation.

• Software-as-a-Service: AppDirect is a B2B subscription commerce marketplace: The power of SaaS liberates AppDirect, and its customers, from unnecessary complexity. This allows organizations to focus on maximizing their business outcomes and customer value, rather than solving solved problems.

These tenets also led the AppDirect engineering team to invest in Confluent’s data streaming platform. Confluent simplifies real-time data processing, enabling businesses to build scalable, responsive applications with Apache Kafka® at its core.

You can find Tom’s full interview here: “The Power of Data Streaming in Digital-Native Organizations: A Look Inside AppDirect.”

Improving the internal dev experience + efficiency gains 

The AppDirect team’s decision to adopt Confluent’s platform was driven by two primary objectives: First, to improve the internal developer experience and second to enable teams to spend less time managing the Apache Kafka® infrastructure, which fuels their existing streaming projects. Tom describes the challenges of managing Kafka systems.

“Kafka systems are very large and complicated to manage. Confluent provides us with a fully managed Kafka solution that has better uptime and resiliency than the open source solution. This allows our developers to focus on building core customer deliverables rather than overseeing complex Kafka operations.”

Previously, AppDirect’s microservices communicated using basic REST.

“With Confluent, we’ve seen significant improvements: reduced complexity, higher throughput, better schema enforcement, and more secure access control lists (ACLs),” Vogel explains.

Kafka Connect also enables integration with multiple powerful source and sync connectors that interface with AppDirect’s proprietary mix of industry-standard data sources, including search engines, SQL, NoSQL, and more. This integration allows AppDirect to streamline data synchronization and power real-time analytics.

Streamlining billing and commission tracking

Data streaming is more than a technical upgrade for the AppDirect platform—it's essential for customers’ business operations. Real-time billing and commission tracking are crucial for the AppDirect marketplace, ensuring agents receive immediate updates on their sales and payouts.

“The whole concept of a real-time, business-to-business, omnichannel, subscription-commerce platform would not be possible without data streaming.” Tom Vogel, AppDirect’s Director of DevOps & Cloud Platform Engineering

By leveraging data streaming with Confluent, AppDirect powers critical business functions like billing, commissions, and analytics with real-time data flows.

"Without data streaming, we wouldn’t be able to scale our platform as easily or provide the level of customer service that sets us apart," Vogel explains.

To dive deeper into how AppDirect leverages Confluent, read the full Confluent blog.

Other use cases: Internal developer platform & Kafka data streaming

In recent years, AppDirect has been leveraging its internal developer platform and Kafka data streaming architecture, in order to unlock new developments in AI. Curious to learn more? Check out the AppDirect AI Marketplace and Creation Studio. AppDirect AI is a secure, user-friendly platform that organizations and individuals can use to create AI apps—no coding required. AppDirect AI offers top-tier data protection, rapid deployment, and easy customization of AI apps, from data sources to model customization. Getting started is simple—embed your AI apps with just a few lines of code using this easy-to-follow guide. Not signed up yet? Join AppDirect.ai today.

Also check out our AppDirect AI webinar for more details!