학술논문

Optimization Techniques for Data Consistency and Throughput Using Kafka Stateful Stream Processing
Document Type
Conference
Source
2023 6th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI) Research of Information Technology and Intelligent Systems (ISRITI), 2023 6th International Seminar on. :480-485 Dec, 2023
Subject
Communication, Networking and Broadcast Technologies
Computing and Processing
Power, Energy and Industry Applications
Robotics and Control Systems
Signal Processing and Analysis
Seminars
Software architecture
Databases
Ecosystems
Data integration
Organizations
Throughput
data integration
message bus
kafka
stateful stream processing
data consistency
event driven architecture
Language
ISSN
2832-1456
Abstract
Nowadays, some organizations may have various application systems, databases, and services. Sometimes, each of them is designed using different technologies. Data integration has an important role in modern complex software architectures. In this context, data integration enables data to be spread across an organization's entire technology ecosystem. It requires a guarantee that the data being transmitted is valid and consistent. Therefore, a strategy is needed to ensure the validity of data exchange between applications. This research takes an in-depth look at the analysis and implementation of data integration using Apache Kafka. It discusses optimization strategies and techniques to improve data consistency and throughput in a Kafka environment. This research provides important insights into how Kafka Stream and Kafka Connect can improve data integration compared to Kafka's conventional producer-consumer model.