Scala vs Python vs Java :: Big Data processing with Apache Spark



Scala vs Python vs Java :: Big Data processing with Apache Spark


I have tried to compare Python and Scala on various parameters like: performance, ease of using the language, integration using existing libraries, support for streaming use cases and of Apache Spark’s core capabilities.

I did not try to evaluate Java for the following reasons:

  • Java does not support the REPL command line feature (Read, Evaluate, Print, Loop) which is very extensively used to check if small code-snippets are working as expected. 
  • Java is too verbose – It uses more lines of code, and displays more messages than needed.
  • Scala on the JVM is way more powerful and cleaner than Java.
SCALA
PYTHON
De-Facto language for Spark
Scala is the 1st preferred language for Spark as Spark itself is written in Scala, so developers can dig deep into the Spark source code whenever required.

New features of Spark are first available in Scala and are later ported to Python or Java.
Data Scientists would traditionally have more of a background in Python will not like to learn a new language.
Performance
Scala is claimed to be 10 times faster than Python for complex data processing - A compiled language will have better performance at run-time if it's statically typed (vs. dynamically)

Scala - Statically typed,
Python - Dynamically typed
Scala is a Statically typed language, so the type of a variable (Int / String / Float etc) is known at the compile time.

Compiler is proactive to catch the errors at a very early stage. Scala makes the compiler do additional checks and so you have to do less.
Python is a dynamically typed language where a declared variable can take any value at runtime.

Dynamically typed programming languages do type checking at run-time as opposed to compile-time.
More error-prone?
If we happen to miss any spots or leave the code in an illogical state, the code will not compile, preventing any surprising run-time errors.
Every time a code-restructuring is done, there is always a risk of breaking the logic and leaving out bugs while using Python.

Since Scala is a compiled language, it gets the advantage over Python here.
Streaming support
Python Spark streaming support is not advanced and mature like Scala.

Multi-threading
Big Data systems needs the development activities be linked across multiple databases and services.

Scala is preferred here for the Play framework that offers asynchronous libraries which are easy to integrate.
Python does not support true multi-threading.
External libraries
Scala has excellent built-in concurrency support and libraries like Akka which makes   it easy for developers to build a truly Scalable application.

Spark MLib – the machine learning library has enough algorithms ideal for most Big Data related use cases.
Python has several libraries for Machine Learning and Natural Language Processing.



Hope you enjoyed reading this post. Please feel free to provide your feedback. 

Thanks!

Comments

Post a Comment

Popular posts from this blog

Automated bash script to export all Hive DDLs from an existing environment at one go!

Exporting all Hive DDLs from an existing environment The below script scans through all the databases on your Hive system, and routes all the Create Table statements for all the tables to a file. This will be helpful when you need to set up a new environment based on the existing one. It has been tested on databases with over 500 tables. Steps to be performed are as follows: 1. Create a script, say  hive_ddls_export.sh  in the Hadoop box with HIVE CLI installed on it with the following content: #!/bin/bash databases =`hive -e "show databases;"` all_db_names =${databases} datetoday =`date +%Y-%m-%d_%H:%M:%S` touch dev_hive_ext_tables_$datetoday.sql chmod 744 dev_hive_ext_tables_$datetoday.sql for listofdatabases in $ all_db_names do   tables=`hive -e "use $listofdatabases;show tables;"`   all_tab_names=`echo "${tables}"`   echo " /****  Start DDLs for Tables in ${listofdatabases} ****/ " >> dev_hive_ext
Read more

Apache Spark :: Error Resolution :: 'value join is not a member of org.apache.spark.rdd.RDD'

Apache Spark :: Error Resolution :: 'value join is not a member of org.apache.spark.rdd.RDD' ERROR DESCRIPTION Consider 2 Spark RDDs to be joined together.. Say, rdd1.first is in the form of (Int, Int, Float) = (1,957,299.98) while rdd2.first is something like (Int, Int) = (25876,1) where the join is supposed to take place on the 1st field from both the RDDs. scala> rdd1.join(rdd2)  --- results in an error <console>:**: error: value join is not a member of org.apache.spark.rdd.RDD[(Int, Int, Float)] REASON Both the RDDs should be in the form of a Key-Value pair. Here, rdd2 -- being in the form of (1,957,299.98) -- does not obey this rule.. While rdd1 -- which is in the form of (25876,1) -- does. RESOLUTION Convert the output of the 1st RDD from (1,957,299.98) to a Key-Value pair in the form of (1,(957,299.98)) before joining it with rdd2, as shown below: scala> val rdd1KV = rdd1.map(x=>(x.split(",")(1).toInt,(x.split(","
Read more

Filtering out Nulls and Headers in Scala/Spark

Filtering out Nulls and Headers in Scala/Spark Consider a contact_file in HDFS location  /User/VJ/testfile  which has a null record in a non-null field. Here, the last line has no value in the 'age' field so the requirement is to filter all such lines. id,fname,lname,age,designation 1, amarnath, jaiswal, 61, Businessman 2, prakash, yadav, 30, Developer 3, vishal, jaiswal, 32, Engineer 4, ravi, jaiswal,, Builder Solution: Usage of mapPartitionsWithIndex to drop the 1st iterator for 0th index will filter the header from your input file, while the Usage of != "" on the 4th field will filter out the 3rd line scala> sc.textFile("/User/VJ/testfile"). mapPartitionsWithIndex ((x,y) => if (x==0) y.drop(1) else y).filter(x=>x.split(",")(3) != "" ).take(5).foreach(println) Output: 1, amarnath, jaiswal, 61, Businessman 2, prakash, yadav, 30, Developer 3, vishal, jaiswal, 32, Engineer Thanks, Vishal.
Read more