Spark Read Modes & Details on Spark Session

 Spark processes the files in three modes:

  1. Permissive: This is the default read mode in spark. Say we are creating the dataframe in spark and we encountered a datatype mismatch. So, In case of datatype mismatch it will convert the value into NULL without impacting the rest of the results.  
  2. Drop Malformed: In case of spark mode, Any malformed records will be eliminated and rest of the records in proper shape will be processed. 
  3. Fail Fast: Errors out on encountering any malformed records. 

So, It is very important to choose the respective modes based on the business requirement.  

     Creation of Spark Session:

          - Spark Session acts as an entry point to the Spark Cluster. To run the

            code on Spark Cluster, a Spark Session has to be created.

          - In order to work with Higher Level APIs like Dataframes and Spark SQL,

            Spark Session has to be created to run the code across the cluster.

         - To work at RDD level, Spark Context is required.

         - Spark Session acts as an umbrella that encapsulates and unifies the

            different contexts like Spark Context, Hive Context, SQL Context…


Need for Spark Session when we already have Spark Context?

  • Spark Session encapsulates the different contexts like Spark,Hive,SQL

and allows these contexts to be accessed from a single session.

  • When there is a need to have more than one Spark Session for a single

application with their respective isolated environments.


Same Spark Context will be shared across multiple spark sessions

created.


 

Comments

Post a Comment

Popular posts from this blog

SQL Questions - Ready for Interview

 1.   U se case for each of the functions Rank, Dense_Rank & Row_Number Row Number -  the  ROW_NUMBER()  function is used to assign a unique integer to every row that is returned by a query. Syntax: ROW_NUMBER() OVER( [PARTITION BY column_1, column_2, …] [ORDER BY column_3, column_4, …] ) Let’s analyze the above syntax: The set of rows on which the  ROW_NUMBER()  function operates is called a window. The  PARTITION BY  clause is used to divide the query set results.  The  ORDER BY  clause inside the  OVER  clause is used to set the order in which the query result will be displayed. Query Format: SELECT mammal_id, mammal_name, animal_id, ROW_NUMBER () OVER ( ORDER BY mammal_name ) FROM Mammals; Rank: The  RANK()  function assigns a rank to every row within a partition of a result set. For each partition, the rank of the first row is 1. The  RANK() ...
Read more

Spark Streaming

Spark is internally a microbatch engine. At every interval, as and when the data arrives, a small microbatch gets created and this gives an illusion of a real-time streaming. However, internally it is getting processed in the form of micro-batches. To answer the following questions :  - What is the Size of the Micro-batch? - When is the Micro-batch triggered? It is required to have an understanding of different types of Triggers. Types of Triggers - 1. Unspecified (Default) In this case, once the first micro-batch (say there are 2 files - File1 & File2 in the first micro-batch) processing is complete, the second micro-batch will be triggered provided there is some data that needs to be processed. The subsequent micro-batch gets triggered only when there are some files that need to be processed. Second micro-batch gets triggered when the File3 arrives. Suppose File4 has arrived and File3 is still in process, then the third micro-batch will be triggered soon after the 2nd micro-b...
Read more