Top 6 Google BigQuery Pitfalls Data Engineers Need to Know
No matter what organization you are part of, chances are there's an IT problem that needs to be solved. From full digital transformations, to moving to cloud data storage, or data processing and analytics; we're all facing our fair share of challenges.
That said, cloud services like Google have introduced data storage solutions like Google BigQuery that maps well to the way we think about and use data at work.
For one, Google BigQuery has improved data accessibility across organizations. Empowering more teams to generate more analytics through the use of Google BigQuery processing.
Not only this, but the costs of data warehousing are also becoming more and more cost effective with the introduction of the Cloud. So for more organizations, it's not a matter of whether or not to transition, but when and how.
When it comes down to streamlining Google BigQuery as part of your organizations operations, what work needs to go into it? If it's new to your data engineers, what challenges will they face and what's a realistic timeline to deliver the final transition?
While we cannot answer all of these questions tailored to the use case of your organization, here at Kloudio, we have our fair share of insights on Google BigQuery and reporting.
Kloudio has a great team of data experts, and for many years, we've work with data solutions; experiencing the good, bad and ugly.
As we recently integrated the Kloudio platform with Google BigQuery, we're motivated to share out learnings with all of you. Through the journey of building out the the Google BigQuery integration, we kept in mind what it meant for data engineers.
Here are the top 6 things we uncovered working with Google BigQuery:
1. Classic and standard SQL
We're not quite sure why but Google BigQuery provides two forms of SQL, but if there's one you prefer, there's the option to choose. Some love this, others not so much.
If you're interested in the classic SQL similar to that of the BigQueryWeb, that's one option. For those that prefer standard SQL, you may also do so.
2. Cannot save your own functions
It's Google BigQuery's way or the highway! There's no easy way to go about saving your own procedures.
When you take the time to create functions, there's no need to perfect it because they will only exist temporarily for your session. Overall, this makes Google BigQuery a more limited application.
3. Upper case sensitive
Unlike other data warehouses you might be used to, Google BigQuery is case sensitive for strings, object names and more.
You can get rid of this when using string comparison if you use the upper function for both sides. However, object names are not case sensitive with column names.
4. Cannot add projects to object browser
For any project you're working on in Google BigQuery, ensure that you have named access. If not, you'll be unable to add this to your object browser, limiting your overall convenience.
5. No query estimates
This has to be one of the greatest downfalls to using Google BigQuery. Basically, you're not sure whether your complex query will work before its being performed.
This can create delays as you're running a large number of queries. Once complete, the next steps are easy but this is done, Google BigQuery doesn't estimate the outcome.
6. Cannot edit existing tables
There's little explanation needed, but there's little to no way you can edit existing tables. Unfortunately, you can't add or remove columns. Nope, no renaming your table or other fields either.
We can see how this can easily peeve some users. Keeping in this in mind will help you avoid the mistake of "I'll fix up this table later".
From listing the above pitfalls, it may appear to you that we're saying Google BigQuery isn't a great choice for data warehousing. But that couldn't be further from the truth. No one cloud solution comes without its own challenges when data engineers work with them for the first time.
In reality, there's a ton of benefits to using Google BigQuery and that's why your organization is currently implementing it. That's why we found it important to share our findings so that you can save time and avoid these surprises on Google BigQuery related tasks.
To learn more about the fabulous upsides to using Google BigQuery, stay tuned for next week's article where we'll be sharing the less-than-obvious benefits we uncovered.
If you're currently in the process of working on your first Google BigQuery project, trust the process as there's a few pitfalls to get used to. That is, more or less related to save functions. When completed, you'll find that the benefits of cloud data warehousing and analytical reporting far outweighs the negatives.
Have you been using Google BigQuery for some time? If so, share with us in the comments below new techniques you've learned and challenges encountered in your day-to-day.