Config files for a Haddop + Spark + Hive + Kafka + Postgresql raspberry cluster (ubuntu 20.04)
The files in this repository are the ones you need to create/update in order to set up your cluster.
I did it in a raspberry pi 4 4gb cluster (3 nodes), but any instalation with enough resournces should do - such as virtual machines.
I published the step-by-step in the Towards Data Science:
A Data Science/Big Data Laboratory — part 1 of 4: Raspberry Pi or VMs cluster — OS and communication https://towardsdatascience.com/assembling-a-personal-data-science-big-data-laboratory-in-a-raspberry-pi-4-or-vms-cluster-ff37759cb2ec?source=friends_link&sk=3a4b90e57dc0fc0ec44a39d1aee2145c
A Data Science/Big Data Laboratory — part 2 of 4: Hadoop 3.2.1 and Spark 3.0.0 over Ubuntu 20.04 in a 3-node cluster https://towardsdatascience.com/assembling-a-personal-data-science-big-data-laboratory-in-a-raspberry-pi-4-or-vms-cluster-e4c5a0473025?source=friends_link&sk=d9588dd1597ee9c0811e82666b002e43
A Data Science/Big Data Laboratory — part 3 of 4: Hive and Postgres over Ubuntu in a 3-node cluster https://towardsdatascience.com/assembling-a-personal-data-science-big-data-laboratory-in-a-raspberry-pi-4-or-vms-cluster-8a1da8d49b48?source=friends_link&sk=4a481ee4e3778d6c9d4e5a305a407bb6
A Data Science/Big Data Laboratory — part 4 of 4: Kafka and Zookeeper over Ubuntu in a 3-node cluster https://towardsdatascience.com/kafka-and-zookeeper-over-ubuntu-in-a-3-node-cluster-a-data-science-big-data-laboratory-part-4-of-4-47631730d240?source=friends_link&sk=955731d942d6f83e7f00d731e830ba30