How DATEV Used Confluent to Unlock Its Mainframe Data to Build Cloud-Native Applications

More than two and a half million German businesses benefit from DATEV software to perform financial accounting, resulting in 11 million payslips processed monthly and over 2 million business assessments.

What started as a cooperative business for the tax advisory profession in 1966 has developed into a successful package of data center operations, software development, and consulting services for around 40,000 members from tax consultancy, audit, and legal consultancy.

DATEV eG sees itself as a “forerunner of digitalization” for the tax advisory profession, and wants to help bring digital transformation fully into tax chambers. To make this possible for customers, DATEV has been making a long-term switch of its breadth of product and consulting services to a cloud-native environment.

This requires a powerful technical component within the overarching architecture of the business to allow them to deal with the multitude of microservice communications in a robust and scalable way.

This is where Confluent comes into the picture as a strategic partner. Using Confluent as the backbone of their new event-driven architecture, DATEV is accelerating its digital transformation journey, combining existing mainframe applications with newly developed applications across public and private clouds.

As Martin Dengler, lead devops engineer, explains: “At DATEV, we’re independent, autonomous, and we’re never afraid to try something new. While we may be known for more conventional technologies, we’re highly modern at heart and use a range of cutting-edge technologies, now including Confluent.”

The Challenge

Nothing is as constant as change—especially in IT. And the tax advisory profession is no exception to the game-changing nature of digitization. Customers expect DATEV to provide strong, cooperative support to help them along their digital transformation journey.

DATEV’s strategic decision to develop its new generation of software products as “cloud-native” represented a huge challenge, because becoming “cloud-native” meant adapting:

• New architectures and technologies in software development

• New technologies in data center and platform operations

• New forms of cooperation between developers and operations

The necessary strategic, technological, and cultural change began in 2018. Alongside this, DATEV had been developing and operating very successful “non-cloud-native” software for years, both in mainframe environments and via on-premises installations.

This naturally presented the additional challenge of creating a “good” integration of these two separate technological worlds: the distinct technology stacks must be connected to one another in the most effective way.

After weighing the options, DATEV decided on a data streaming architecture—based on Apache Kafka®—as an integrator and “bridge” between the two worlds.

“We decided on Kafka because, for us, there was no other choice. There are alternative technologies for very specific use cases—like ActiveMQ, RabbitMQ, and Apache Pulsar—but we needed a solution that was much broader,” Dengler says.

After introducing the technology, the Kafka team at DATEV recognized the resource-intensive nature of managing Kafka themselves. Given limited personnel resources, the decision was made to implement Confluent Platform as the basis for a new, event-driven architecture.

Confluent is a central component of the new, cloud-native tech stack at DATEV, and an important integrator that enables a frictionless transfer of information between two very different technological worlds.

The Solution

The road to cloud-native development

The first step to cloud-native development involved breaking down existing monolithic applications into smaller microservices, and ensuring these services communicated seamlessly with one another. Then, software products were redesigned from the ground up, with new architecture and code moulded to fit the cloud-native demands.

With this strategy, DATEV can deliver its portfolio of applications directly to customers. Multi-datacenter replication is implemented as a connector between cloud-native and non-cloud-native worlds.

Managing the shift to event-driven architecture with Confluent

Open source Kafka had been in place at DATEV eG since 2017, but developer adoption was low. The reasons for this were:

• Lack of know-how around Kafka operations

• Lack of Kafka developer know-how

• A lack of self-services for developers

• No defined support

Expanding Kafka within the business saw DATEV face a number of hurdles. “First of all, we had to get to grips with Kafka. Addressing all the topics ourselves wasn’t enough. The biggest issues came from very specific questions or demands—or how Kafka works with our internal processes compared with the technologies we’ve used in-house up until now,” says Dengler.

Confluent was a significant help here.

“There was also a huge need for knowledge transfer to help us make the leap, and Confluent has been essential in helping us build the necessary knowledge and know-how to up our Kafka-DevOps. We’ve used the consulting offerings of the Professional Services and Training team at Confluent along the way, and still do. Having this permanent support for Kafka from a resident Confluent consultant has been critical to success from day one,“ explains Dengler.

A substantial growth in users and use cases

The resident solution architect from Confluent supported and advised DATEV in various areas regarding the data streaming platform and self-service function. A technical account manager was also on hand to help with questions around specific ops themes which arose during daily use.

“With Confluent’s help, our teams have begun to see the potential of the platform. We’ve seen the demand for new use cases consistently grow as we’ve been able to demonstrate ROI and avoid errors both in the design of the platform and in specific use cases. Working with a partner like Confluent means you get all the latest info, can go to events, and understand how others are doing on their data streaming journey, which helps us understand issues and problems at their core,” explains Dengler.

Now, there are over 100 microservice-based use cases across DATEV eG, which has helped deliver greater value to customers.

A new, event-driven architecture has also enabled new products, features, and applications for customers.

With the future cloud-based application for tax office management for example, tax advisors can digitalize and automate the business processes in their tax chambers, including invoicing. The architecture of the solution was built using event-driven design, with Kafka serving as an asynchronous data hub for multiple application domains. The benefit here lies in the scalability and separation of application domains.

With the AI-supported Automatisierungservice Rechnungen, tax advisors can generate account record suggestions from scanned-in receipts and invoices automatically. And, thanks to Confluent, DATEV uses event-driven architecture to process these documents—expediting product releases and passing on improvements to customers much faster.

The Results

Mainframe integration and cloud-native architecture powered by Confluent has enabled:

Support for numerous use cases

With Confluent, DATEV eG has unlocked its mainframe data and implemented a platform to provide new, exciting digital services for customers. Currently, there are over 100 microservicebased use cases powered by mainframe data that represent improvements in terms of service delivery, customer features, and product updates.

As Confluent offers complete scalability, the Kafka team at DATEV are confident they can build, develop, and release new services— tweaking, improving, and scaling when customer needs change and evolve.

Optimal support to make a complex platform useful for new use cases

DATEV took advantage of Confluent’s consultation and professional services teams to problem-solve when building out use cases. Confluent’s premium support channels also enabled DATEV to discuss the inner workings of the technology with specialists, which helped improve understanding, and avoid some of the common pitfalls and technical problems with operating a new IT infrastructure.

“Without the consultants from Confluent, we wouldn’t be where we are now. We’d be light years behind. Kafka is like a nuclear power plant: if you didn’t have someone there to tell you how it works, you’d find it hard to learn it in a specific period of time, get it all right, and have no issues. We wouldn’t have been able to do it without Confluent,” says Dengler.

A flexible platform for developers to experiment

With a fully scalable, self-service platform to work with, every developer at DATEV can now play a role in developing and innovating new, exciting products across its ecosystem. This has helped teams collaborate across different departments, share inspiration, work on improvements, and get innovative ideas to market much faster.

“Thanks to Confluent, cloud developers our business has the ability to develop, to try things out, and to decide: is this useful for our customers? It’s all part of our new culture: our development and production teams are evolving to be more DevOps oriented and Confluent has given us a space where we can think creatively, and make something good out of it,” says Dengler.

What’s Next

“With Confluent, there’s massive potential that hasn’t been fulfilled yet: we’re just at the beginning of the journey and there’s many more things we can do—real-time event processing, new use cases… The potential is huge,” says Dengler.

DATEV has achieved a lot already, but intends to go further.

“From a business perspective, we’re undergoing investigations into the use of machine learning in monitoring, so we can better recognise anomalies in Kafka logs and metrics. We are also continuously expanding the self-service offering and automation around Confluent’s platform.,” Dengler adds.

Ultimately, everything DATEV has achieved so far—and what it hopes to achieve in the future—is designed to make work easier for employees and customers.

“Agility and technology are simply a means to an end. What we really need at the heart of it all is people. We have to create the framework in which people want to change and create something great. And I think, with Confluent, we’re now doing that,” concludes Dengler.

Learn more about DATEV