Spark Technique

Number of stage is dependent on wide transformations

the number of stages = number of wide transformations + 1
Number of tasks = number of partitions


Repartitions Vs Coalesce:
  1. Repartition can both increase and decrease the number of partitions in the RDD while coalesce can only decrease the partions
  2. Full shuffle is involved in case of partitions while in case of coalesce, full shuffle is not performed. Rather it reduces the partitions in much efficient manner say two partitions in one node will be combined together to form one partitions.
  3. when you have to increase the number of partitions you should use repartition. This will increase the parallelism 

Comments

Post a Comment

Popular posts from this blog

SQL Questions - Ready for Interview

 1.   U se case for each of the functions Rank, Dense_Rank & Row_Number Row Number -  the  ROW_NUMBER()  function is used to assign a unique integer to every row that is returned by a query. Syntax: ROW_NUMBER() OVER( [PARTITION BY column_1, column_2, …] [ORDER BY column_3, column_4, …] ) Let’s analyze the above syntax: The set of rows on which the  ROW_NUMBER()  function operates is called a window. The  PARTITION BY  clause is used to divide the query set results.  The  ORDER BY  clause inside the  OVER  clause is used to set the order in which the query result will be displayed. Query Format: SELECT mammal_id, mammal_name, animal_id, ROW_NUMBER () OVER ( ORDER BY mammal_name ) FROM Mammals; Rank: The  RANK()  function assigns a rank to every row within a partition of a result set. For each partition, the rank of the first row is 1. The  RANK() ...
Read more

Spark Streaming

Spark is internally a microbatch engine. At every interval, as and when the data arrives, a small microbatch gets created and this gives an illusion of a real-time streaming. However, internally it is getting processed in the form of micro-batches. To answer the following questions :  - What is the Size of the Micro-batch? - When is the Micro-batch triggered? It is required to have an understanding of different types of Triggers. Types of Triggers - 1. Unspecified (Default) In this case, once the first micro-batch (say there are 2 files - File1 & File2 in the first micro-batch) processing is complete, the second micro-batch will be triggered provided there is some data that needs to be processed. The subsequent micro-batch gets triggered only when there are some files that need to be processed. Second micro-batch gets triggered when the File3 arrives. Suppose File4 has arrived and File3 is still in process, then the third micro-batch will be triggered soon after the 2nd micro-b...
Read more

Spark Read Modes & Details on Spark Session

 Spark processes the files in three modes: Permissive: This is the default read mode in spark. Say we are creating the dataframe in spark and we encountered a datatype mismatch. So, In case of datatype mismatch it will convert the value into NULL without impacting the rest of the results.   Drop Malformed: In case of spark mode, Any malformed records will be eliminated and rest of the records in proper shape will be processed.  Fail Fast: Errors out on encountering any malformed records.  So, It is very important to choose the respective modes based on the business requirement.        Creation of Spark Session:           - Spark Session acts as an entry point to the Spark Cluster. To run the             code on Spark Cluster, a Spark Session has to be created.           - In order to work with Higher Level APIs like Dataframes and Spark SQL,     ...
Read more