Problem Analysis:
The company needs to migrate both an application and an on-premises Apache Kafka server to AWS.
Incremental updates from an on-premises Oracle database are processed by Kafka.
The solution must follow a replatform migration strategy, prioritizing minimal changes and low management overhead.
Key Considerations:
Replatform Strategy: This approach keeps the application and architecture as close to the original as possible, reducing the need for refactoring.
The solution must provide a managed Kafka service to minimize operational burden.
Low overhead solutions like serverless services are preferred.
Solution Analysis:
Option A: Kinesis Data Streams
Kinesis Data Streams is an AWS-native streaming service but is not a direct substitute for Kafka.
This option would require significant application refactoring, which does not align with the replatform strategy.
Option B: MSK Provisioned Cluster
Managed Kafka service with fully configurable clusters.
Provides the same Kafka APIs but requires cluster management (e.g., scaling, patching), increasing management overhead.
Option C: Amazon Kinesis Data Firehose
Kinesis Data Firehose is designed for data delivery rather than real-time streaming and processing.
Not suitable for Kafka-based applications.
Option D: MSK Serverless
MSK Serverless eliminates the need for cluster management while maintaining compatibility with Kafka APIs.
Automatically scales based on workload, reducing operational overhead.
Ideal for replatform migrations, as it requires minimal changes to the application.
Final Recommendation:
Amazon MSK Serverless is the best solution for migrating the Kafka server and application with minimal changes and the least management overhead.
Amazon MSK Serverless Overview
Comparison of Amazon MSK and Kinesis
