VizCanvaz

  This is the first post in a 5-part series on data engineering fundamentals. We'll go from SQL, to Python, to ELT/ETL tools, to data warehousing with Snowflake, and finally to visualisation. Part 2 covers Python basics, link at the bottom. Why SQL, even in 2026? Every modern data platform, Snowflake, Databricks, Fabric, Matillion, dbt, still expects you to think in SQL underneath the interface. Cloud platforms change every few years. SQL has stayed the primary way to query and reason about structured data for decades. If you only master one skill in data engineering, make it this one. Important concepts you actually need to know Joins, properly understood. Not just INNER vs LEFT, but what happens to row counts when a join key isn't unique on one side, and why an unexpected LEFT JOIN can silently multiply your row count. Window functions. ROW_NUMBER, RANK, DENSE_RANK, and aggregate functions used with OVER() let you calculate running totals, rankings, and comparisons across row...

I get asked a lot by people looking to break into data engineering, or move up in it, what they should actually focus on. After 15 years in this space, from writing SAP BEx queries to architecting Snowflake and Matillion pipelines today, here's the roadmap I'd give my younger self, based on what's really changed and what hasn't. Start with SQL, not a fancy tool. This still surprises people, but SQL remains the single most important skill in data engineering, more than any cloud platform or orchestration tool. Get comfortable with joins, window functions, aggregations, and query optimization before you touch anything else. Every tool you'll use later, Snowflake, dbt, Matillion, Databricks, still expects you to think in SQL underneath the interface. Learn Python well enough to automate and transform data. You don't need to be a software engineer, but you do need enough Python to write scripts, work with APIs, and understand basic data structures. Most modern ELT t...

I recently cleared the DP-700, Microsoft's Fabric Data Engineer Associate certification, and wanted to share what actually worked, in case it helps someone else preparing for it. A few basics before you book it. The exam has 40 to 60 questions, including a case study section of around 10 questions, and you get 100 minutes to complete it, plus an extra 20 minutes upfront for setup and the non-disclosure agreement, so plan for about 120 minutes total at the test center or on your webcam. It's proctored and not open-book, so no notes, no second monitor, and no wandering off camera. You'll need a government-issued photo ID that matches your Microsoft Learn profile exactly. One thing a lot of people don't realize going in, Microsoft's exam interface includes a Reference option that lets you pull up relevant Microsoft Learn documentation during the exam. It's genuinely useful, but treat it as a last resort, not a first move. Searching documentation mid-question eats i...

Snowflake has become the backbone of a lot of the data platforms I've worked on, and along the way I've picked up a few habits that make a real difference between a warehouse that scales gracefully and one that quietly burns through credits. Here they are. Warehouses are not one-size-fits-all. It's tempting to run everything, ETL loads, BI dashboards, ad hoc analyst queries, through a single virtual warehouse. Don't. Separate warehouses by workload so a heavy transformation job doesn't queue behind someone's Tableau refresh, and so you can size and auto-suspend each one independently. A small warehouse for BI traffic and a larger one for batch loads will almost always cost less than one oversized warehouse trying to do both. Auto-suspend aggressively, auto-resume freely. Snowflake bills by the second, so there's rarely a good reason to leave a warehouse running idle. Set auto-suspend to a minute or two for most workloads. The resume time is fast enough that ...

After working with Matillion across different enterprise projects, here are a few practical lessons that have saved me a lot of debugging time. Sharing them in case they help you too. - Design for reusability from day one. It's tempting to build a job that solves today's problem and move on, but don't. Break your transformation logic into Shared Jobs early, even if you're only using it once right now. Matillion's Shared Job feature lets you parameterize common patterns so the next pipeline takes hours instead of days. - Keep orchestration and transformation jobs separate. Orchestration jobs should control flow, sequencing, and error handling. Transformation jobs should do the actual data manipulation. Mixing the two makes jobs hard to read and hard to hand off to a teammate. - Use grid variables instead of hardcoding loops. Instead of duplicating a job for every table or source system, drive it from a grid variable populated by a metadata query. This turns 50 nearly...

The nice way to create KPI's in Tableau with a good or bad indicator is to use the custom format option. Follow the below steps to create a cool looking KPI. At first, create a calculated field in Tableau based on your KPI definition. In my case, I am using the Profit Ratio calculated field in the Super Store sample dataset. Custom format used : ▲ +0"%"; ▼ -0"%"; ► 0"%" Happy Tableauing :)

The easiest way to fetch data from databases to Excel is using the Get Data option available under the Data Tab, Excel creates a query using the ODBC connection and fetches the data without the hassle of coding. But coding makes the process more flexible and is very dynamic. Here goes the details... 1) Download and install the snowflake ODBC driver either 32 or 64 bit depending on your MS Office Installation & OS from the SnowFlake portal, available under the Help menu. 2) Create a DSN with the correct connection properties. 3) Add the below reference if required in the visual basic editor and also change the DSN name in the VBA code. VBA Code...  '----------------------------------------------------------------------------------------------------------------------------------------- Sub VBA_SnowFlake_Connect() Dim sSQLQry As String Dim ReturnArray Dim Conn As ADODB.Connection Dim rs As ADODB.Recordset Dim DBPath As String, sconne...