Spark Read Modes & Details on Spark Session

 Spark processes the files in three modes:

  1. Permissive: This is the default read mode in spark. Say we are creating the dataframe in spark and we encountered a datatype mismatch. So, In case of datatype mismatch it will convert the value into NULL without impacting the rest of the results.  
  2. Drop Malformed: In case of spark mode, Any malformed records will be eliminated and rest of the records in proper shape will be processed. 
  3. Fail Fast: Errors out on encountering any malformed records. 

So, It is very important to choose the respective modes based on the business requirement.  

     Creation of Spark Session:

          - Spark Session acts as an entry point to the Spark Cluster. To run the

            code on Spark Cluster, a Spark Session has to be created.

          - In order to work with Higher Level APIs like Dataframes and Spark SQL,

            Spark Session has to be created to run the code across the cluster.

         - To work at RDD level, Spark Context is required.

         - Spark Session acts as an umbrella that encapsulates and unifies the

            different contexts like Spark Context, Hive Context, SQL Context…


Need for Spark Session when we already have Spark Context?

  • Spark Session encapsulates the different contexts like Spark,Hive,SQL

and allows these contexts to be accessed from a single session.

  • When there is a need to have more than one Spark Session for a single

application with their respective isolated environments.


Same Spark Context will be shared across multiple spark sessions

created.


 

Comments

Post a Comment

Popular posts from this blog

Spark Debugging And performance Optimzation

Image
  Introduction to Spark Optimization Optimizing Spark can dramatically improve performance and reduce resource consumption. Typically, optimization in Spark can be approached from three distinct levels: cluster level, code level, and CPU/memory level. Each level addresses different aspects of Spark's operation and contributes uniquely to overall efficiency in distributed processing with Spark. Cluster Level Optimization At the cluster level, optimization involves configuring the Spark environment to efficiently manage and utilize the hardware resources available across the cluster. This includes  tuning resource allocation settings such as executor memory, core counts, and maximizing data locality to reduce network overhead . Effective cluster management ensures that Spark jobs are allocated the right amount of resources,  balancing between under-utilization and over-subscription . Code Level Optimization Code level optimization refers to the improvement of the actual Spa...
Read more

Spark File Format

Image
  File Formats In the realm of data storage and processing, file formats play a pivotal role in defining how information is organized, stored, and accessed. Here's a table summarizing the characteristics, advantages, limitations, and use cases of the mentioned file formats: Format Characteristics Advantages Limitations Use Cases Comma-Separated Values (CSV) Simple, all data as strings, line-based. Simplicity, human readability. Space inefficient, slow I/O for large datasets. Small datasets, simplicity required. XML/JSON Hierarchical, complex structures, human-readable. Suitable for complex data, human-readable. Non-splittable, can be verbose. Web services, APIs, configurations. Avro Row-based, schema in header, write optimized. Best for schema evolution, write efficiency, landing zone suitability (1), predicate pushdown (2). Slower reads for subset of columns. Data lakes, ETL operations, microservices architechture, schema evolving environments. Optimized Row Columnar (ORC) Columna...
Read more

Spark Technique

Number of stage is dependent on wide transformations the number of stages = number of wide transformations + 1 Number of tasks = number of partitions Repartitions Vs Coalesce: Repartition can both increase and decrease the number of partitions in the RDD while coalesce can only decrease the partions Full shuffle is involved in case of partitions while in case of coalesce, full shuffle is not performed. Rather it reduces the partitions in much efficient manner say two partitions in one node will be combined together to form one partitions. when you have to increase the number of partitions you should use repartition. This will increase the parallelism 
Read more