Distributed Computing with Spark SQL (Coursera)

By

Overview

This course was a great positive surprise regarding the way the practical exams worked. You can think of this as a great introduction to Databricks, along with Spark, and a fact I think few people know: you can use Databricks for free if you create a community account.

Spark is one of the most important tools (if not the most important) for large data processing (a.k.a. Big Data). The main advantage it provides over traditional data processing tools is the possibility of horizontal scaling, which makes scalability almost limitless. Spark is a basic tool for Data Engineers and one of the largest problems one can face when learning Data Engineering is the problems we are trying to solve are usually for big corporations with a huge pile of data, which often involves high costs to simulate such situations.

This course is part of the SQL Basics for Data Science specialization.

Here are the course details from Coursera:


Pros

As mentioned before, the best part of this course was the practical exercises done in Databricks. That was an excellent opportunity to get to know a little bit more about Spark, especially the tuning of partitions, shuffling, etc.

There are many notebooks available in the course material which are not used in the classes, and those are totally worth studying "by yourself". The fact that Databricks has a community account is great for integrating a more sophisticated solution into your personal projects and boosting your personal portfolio in Github.

The whole course has a "hands-on" approach which I personally like a lot. A brief theoretical explanation is presented but soon we are going through the example notebooks with real code.

Cons

The only point the course could improve a bit is to have more exercises on fine-tuning and solving performance issues with Spark. No one uses Spark just for fun, so usually there is a performance component involved. It would be frustrating if after moving to Spark, the engineer still faces performance issues and is unable to solve them quickly. 

Conclusion

This course is definitely recommended if you never had any experience with Spark and really feel this knowledge is missing on your resume. Bear in mind this is an introductory course, and it does not go deep in one of the most important parts of Spark: improving performance. If you feel something is missing in this review, please write a note in the comments!

Comments

Post a Comment

Popular posts from this blog

Dealing With Large Features in Git Repos

By
Image
There is not a "correct" way to organize your servers and git repos and each company has its own problem and solutions. Some companies use test servers to run some very basic verifications in their codebase before deploying new features to production. Some even have a staging server that is as similar to production as possible, where new features are tested for a last time before deploying to prod. Those servers usually run deployed code that is in the development, staging, and master/main branches in the git repo. This would be what the usual flow of development looks like: Another good practice many companies adopt is the versioning of the code to keep track of what is deployed on each server. There is a lot about versioning alone that should be discussed in other posts, but let us adopt the format major.minor.build for our example. Dealing with Bugs in Tests Now, let us imagine that a bug was discovered in one of your features when it was being tested in the Test Server. W
Show me more »

Foreign Visitors in Brazil - 2005 to 2015 - Part II

By
Image
So this is the continuation of our previous analysis of foreign visitors from 2005 to 2015. The first thing we see is how the variable of interest (number of arrivals) behaves through the years. It is observable that there has been a slight increase in the number of foreign visitors since 2013. It is also possible to see how the accumulated visits are distributed through the continents on the graph below. From this graph we can understand that this increase is specially because of South America. For the other two continents with most visitors, Europe and North America, there has been a slight decrease in arrivals. Below we have the graph of arrivals per continent in this 10 year window. South America and Europe have by far the larger number of visitors. Next we observe how the sum of arrivals is ditributed through the months of the year. As expected, there is a larger number of arrivals during summer time in Brazil. However, there is an unexplained increase in July. So far, one possib
Show me more »