Streaming Platform for Connected Vehicle Data

Streaming platform of a major German car manufacturer: receive telemetry data from customer vehicles and distribute it in near real time to over 50 internal and external target systems – operated on AWS in the EMEA, USA and China regions and designed for several million events per day.

The platform is still in production and is being continuously extended.

Period: 2018 – 2024, 2025 – 2026 · Client: major German car manufacturer · Industry: automotive

Challenge

Connected vehicles continuously send telemetry data, e.g. vehicle status, diagnostic and location data. The platform receives it and distributes it to the target systems that need it: the manufacturer's internal business systems as well as external partners. Each target system receives only the data intended for it. Started in the client's on-premises data center, the platform was rebuilt over the years into a microservice architecture on AWS. What matters to the target systems: complete data, low latency and constant availability.

The technical challenges behind it:

• Throughput in near real time: Several million events per day must be processed and forwarded without loss and with low latency.
• Many target systems: Over 50 internal and external target systems each receive their slice of the data. New target systems are connected without affecting the existing ones.
• Uniform data formats: The data formats are managed centrally as Avro or Protobuf schemas in the Glue Schema Registry so that senders and target systems remain compatible even when changes are made.
• Multiple regions: From the start, the platform has run in the EMEA, USA and China regions. The infrastructure is described as code with Terraform and rolled out automatically to all regions.
• Migration during operation: The web services moved from the client's data center (on-premises and OpenShift) to AWS. Meanwhile, the target systems had to continue to be supplied without interruption.
• Information security: Vehicle data is sensitive. The platform and processes must meet the requirements of TISAX, the information security standard of the automotive industry.
• Operations and monitoring: The team operates the services itself according to DevOps principles, up to incident management. Health, throughput and errors are monitored with Datadog, Dynatrace and Grafana so that bottlenecks are noticed in time.
• Maintainability: Over the years, more than 20 web services have emerged. They are analyzed and consolidated so that the platform remains maintainable.

Role and Implementation

• Backend Developer, Test Automation Engineer
  Phase 1: 2018 – 2020 · Stabilization and go-live
  • Load tests, performance analysis and optimization of the web services before going into production (Gatling, JProfiler)
  • Error analysis and stabilization up to go-live (Payara, Spring Boot, Kafka)
  • Test automation (methods, API, components, integration)
  • Implementation of new business features
• Software Architect, Backend Developer, Business Analyst, Test Automation Engineer
  Phase 2: 2020 – 2022 · Migration to AWS
  • Architecture of the event-driven target platform on AWS (Kafka on Amazon MSK, Kinesis)
  • Migration of the on-premises and OpenShift web services to AWS during operation (ECS, Fargate, Lambda)
  • Infrastructure as code for all regions with Terraform
  • Schema management with Avro, Protobuf and Glue Schema Registry
  • Requirements analysis: business features worked out together with the client and implemented
  • Team training on new technologies and processes
• Software Architect, Backend Developer, Business Analyst, Test Automation Engineer
  Phase 3: 2022 – 2024 · Expansion
  • Consulting the client on architecture and integration concepts
  • Connection of new and migration of existing target systems
  • Expansion of CI/CD (GitHub Actions) and automated integration tests
  • Measures and documentation for TISAX certification
  • Team training
• Software Architect, Backend Developer, Business Analyst, Test Automation Engineer
  Phase 4: 2025 – 2026 · Consolidation
  • Analysis of the grown Kafka-based service landscape and design of the target architecture (Kafka Streams, Apache Camel)
  • Consolidation of the web services
  • Functional extensions of the Spring Boot services
  • Migration to Spring Boot 4.x and Java 21

Technologies

• Java
• Python
• Spring Boot 2.x – 4.x
• Payara
• Apache Camel
• Apache Kafka
• Kafka Streams
• Apache Avro
• Protocol Buffers
• AWS Glue Schema Registry
• Amazon MSK
• Amazon Kinesis
• Amazon ECS
• AWS Fargate
• AWS Lambda
• Docker
• OpenShift
• Terraform
• GitHub Actions
• REST
• OpenAPI
• Gatling
• JProfiler
• Datadog
• Dynatrace
• Grafana
• GitHub Copilot