Kafka

Kafka is an asynchronous messaging queue. Kafka consumer, consumes message from Kafka and does some processing like updating the database or making a network call.

https://kafka.apache.org/documentation/

What is Apache Kafka? Why is it so popular? Should you use it?

https://techbeacon.com/app-dev-testing/what-apache-kafka-why-it-so-popular-should-you-use-it

What, why and How Apache Kafka

https://www.startdataengineering.com/post/what-why-and-how-apache-kafka/

Kafka partitions

https://www.educba.com/kafka-partition/

Getting started with Apache Kafka in Python

https://towardsdatascience.com/getting-started-with-apache-kafka-in-python-604b3250aa05

How can Kafka consumers parallelise beyond the number of partitions

https://medium.com/@jhansireddy007/how-can-kafka-consumers-parallelise-beyond-the-number-of-partitions-a0a46ade8a6c

PySpark Streaming

https://stackoverflow.com/questions/62342080/how-to-programmatically-load-and-stream-kafka-topic-to-a-pyspark-dataframe#62342332

Connecting the Dots (Python, Spark, and Kafka)

https://www.rittmanmead.com/blog/2017/01/getting-started-with-spark-streaming-with-python-and-kafka/