SparkDriverless is a totally decentralized system inspired by Spark, but with a totally different approach for distributed computation. It removes the Driver node from Spark to achieve single-point-failure-free. Every worker in the cluster is exactly the same. Even the client has similar mechanism with workers. Nodes communicate with each other by broadcasting messages. This allows any node, even the client, fails at any time, without break the computation process. And also it has a much simpler model compared to original Spark.

Fork me at Github. Details at Report

comments powered byDisqus