Recently I have got an opportunity to work on a Kafka client implementation for an interesting use case. Till then my assumption was writing a Kafka client was as easy as we see in many examples on the net :) Although it's true for many see cases, we cannot say the same simple client work for all. Depending upon the complexity of the use case that you may deal with, the client's implementation might change and complexity may increase. Please note that this is not an introductory article on Kafka, also expected you to have some basic understanding of Kafka's client. I found the following article very useful while learning the basics of Kafka client: https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/ . This post will refer to various topics from this article as we move forward. In this post, I am sharing three common scenarios that you may need to deal with while using Kafka client in an message heavy system and possible solutions or patterns followed
- Other Apps
Apache Spark is a data processing framework that can quickly perform processing tasks on very large data sets, and can also distribute data processing tasks across multiple computers, either on its own or in tandem with other distributed computing tools About this example In this post I am sharing a simple Apache Spark example project. The source code used for this example is available here: https://github.com/jobinesh/apache-spark-examples.git Here is the quick overview of the modules that you may find in this project spark-job-common : All common classes that you need for building a Spark job are parked here. This approach may help you to avoid boilerplate code in your Spark job implementation spark-job-impl : A classic word count Spark example is available here. This class may help you to understand the structuring of the source and usage of common classes from spark-job-common module spark-job-launcher : The SparkLauncher helps you to start Spark applications programmatical
- Other Apps
While building applications that comprises of multiple microservices, it is essential to have a mechanism in place to collect and analyze the details of API calls, timing data needed to troubleshoot latency problems and logging error generated from API calls. Jaeger is one such solution and used for monitoring and troubleshooting application built following microservice based architecture with following capabilities: Distributed context propagation Distributed transaction monitoring Root cause analysis Service dependency analysis Performance / latency optimization Jaeger is hosted by the Cloud Native Computing Foundation (CNCF) as the 7th top-level project (graduated in October 2019). Eclipse MicroProfile, OpenTracing and Jaeger As there are multiple tracing solutions out there similar to Jaeger(such as Zipkin), it is really good to avoid vendor lock-in by having some standardization around APIs that work with different providers.The OpenTracing addresses this part of the problem, i
The views expressed on this blog are my own and do not necessarily reflect the views of my employer.