Big Data

Hi there! Apache Spark has been written in Scala originally, although Python developers are loving it’s wrapper-known as PySpark. One can work with RDD’s and dataframes in Python too. We,data science team @Talentica, love PySpark and mostly rely on Spark Clusters for data processing and other relevant stuffs. Recently, we faced one challange which is very important to be addressed. Spark Context Whenever in need, one can initialize the Spark Context in their py file and reuse it.

Hi there! Apache Spark has been written in Scala originally, although Python developers are loving it’s wrapper-known as PySpark. One can work with RDD’s and dataframes in Python too. We, the data science team @Talentica love PySpark and always rely on Spark Clusters for data processing and other relevant stuffs. Recently, we faced one challange which is very important to be addressed. Spark Context def get_spark_context(): conf = SparkConf().setAppName('app-name').set('spark.executor.memory', '8g').