Author: cfinlan

Advanced Data Analysis with Power BI: Leveraging Statistical Functions

Microsoft Power BI is a powerful tool that helps businesses and individuals transform their raw data into actionable insights. One of its most powerful features is the ability to perform advanced data analysis through its comprehensive suite of statistical functions. This blog post will delve into using these functions effectively, giving you a better understanding of your data, and improving your decision-making process.

Let’s start by understanding Power BI a bit better.

Power BI: A Brief Overview

Power BI is a business analytics tool suite that provides interactive visualizations with self-service business intelligence capabilities. Users can create reports and dashboards without any technical knowledge, making it easier for everyone to understand the data. Power BI offers data extraction from multiple heterogeneous data sources, including Excel files, SQL Server, and cloud-based sources like Azure SQL Database, Salesforce, etc.

Leveraging Statistical Functions in Power BI

Power BI is capable of conducting high-level statistical analysis thanks to DAX (Data Analysis Expressions) – a library of functions used in Power BI, Analysis Services, and Power Pivot in Excel. DAX includes a variety of functions such as aggregation functions, date and time functions, mathematical functions, statistical functions, and more.

To start with, we will discuss some of the commonly used statistical functions and how to apply them.

1. AVERAGE and AVERAGEA

The AVERAGE function calculates the mean of a column of numbers. AVERAGEA does the same, but it evaluates TRUE and FALSE as 1 and 0, respectively.

Here’s an example:

AVERAGE ( Sales[Quantity] )
AVERAGEA ( Sales[Quantity] )

The first expression calculates the average of the Quantity column in the Sales table, ignoring any TRUE or FALSE values. The second expression, however, will include these boolean values.

2. COUNT and COUNTA

COUNT function counts the number of rows in a column that contain a number or an expression that evaluates to a number. On the other hand, COUNTA counts the number of rows in a column that are not blank.

COUNT ( Sales[Quantity] )
COUNTA ( Sales[Product] )

The first expression counts the number of rows in the Quantity column of the Sales table that contains a number. The second one counts the number of non-blank rows in the Product column of the Sales table.

3. MIN and MAX

MIN and MAX return the smallest and largest numbers in a numeric dataset, respectively.

MIN ( Sales[Price] )
MAX ( Sales[Price] )

The first expression finds the smallest price in the Price column of the Sales table. The second expression returns the highest price.

4. STDEV.P and STDEV.S

STDEV.P function calculates standard deviation based on the entire population given as arguments. STDEV.S calculates standard deviation based on a sample.

STDEV.P ( Sales[Price] )
STDEV.S ( Sales[Price] )

The first expression calculates the standard deviation of the entire population of prices in the Price column of the Sales table. The second calculates the standard deviation based on a sample.

Implementing Statistical Functions in Power BI: An Example

Let’s demonstrate the implementation of these statistical functions in Power BI with a hypothetical data set. Let’s assume we have a “Sales” table with the following columns: OrderID, Product, Quantity, and Price.

To calculate the average quantity sold, we would create a new measure:

Average Quantity = AVERAGE ( Sales[Quantity] )

We can then use this measure in our reports to get the average quantity of products sold.

To find out the number of unique products sold, we would use the COUNTA function:

Number of Products = COUNTA ( Sales[Product] )

Finally, to find out the standard deviation of prices, we would use the STDEV.P function:

Price Standard Deviation = STDEV.P ( Sales[Price] )

We can now use these measures in our reports and dashboards to provide a statistical analysis of our sales data.

Conclusion

Understanding statistical functions in Power BI can provide meaningful insights into data. With a broad range of statistical functions available in DAX, you can perform advanced data analysis with ease. This blog post has introduced you to the concept and shown you how to leverage these functions. However, the scope of Power BI’s statistical capabilities goes far beyond these basics. As you get more comfortable, you can explore more complex statistical functions and techniques to gain deeper insights into your data.

Remember, it’s not about the complexity of the analysis you’re performing but about how well you’re able to use that analysis to derive actionable insights for your business or organization. Happy analyzing!

This blogpost was created with help from ChatGPT Pro

Unlocking the Power of Power Query: Advanced Data Transformations in Power BI

Business intelligence is no longer the domain of large corporations alone. Thanks to tools like Microsoft Power BI, even small and mid-sized businesses can gain powerful insights from their data. At the heart of Power BI’s data handling capabilities lies Power Query – a potent data transformation tool. This blog post aims to explore some of the advanced features of Power Query, demonstrating how you can manipulate data to fit your needs, accompanied by usable code examples.

What is Power Query?

Power Query is an ETL (Extract, Transform, Load) tool that facilitates data discovery, connection, transformation, and integration tasks. It’s an integral part of the Power BI suite, but it can also be found in Excel and some other Microsoft products. The power of Power Query lies in its ability to connect to a variety of data sources, and more importantly, its transformative capabilities.

Advanced Data Transformations

1. Merging Queries

One common operation in data transformations is merging queries. The Merge Queries feature in Power Query allows you to join two tables similar to SQL. Here’s a simple example:

let
    Source = Excel.Workbook(File.Contents("C:\YourData\Customers.xlsx"), null, true),
    CustomerSheet = Source{[Item="Customer",Kind="Sheet"]}[Data],
    #"Changed Type" = Table.TransformColumnTypes(CustomerSheet,{{"Column1", type text}, {"Column2", type text}}),
    Source2 = Excel.Workbook(File.Contents("C:\YourData\Sales.xlsx"), null, true),
    SalesSheet = Source2{[Item="Sales",Kind="Sheet"]}[Data],
    #"Changed Type2" = Table.TransformColumnTypes(SalesSheet,{{"Column1", type text}, {"Column2", type text}}),
    MergedQueries = Table.NestedJoin(#"Changed Type", {"Column1"}, #"Changed Type2", {"Column1"}, "NewColumn", JoinKind.Inner)
in
    MergedQueries

In this example, Power Query fetches data from two Excel workbooks, Customers.xlsx and Sales.xlsx, and merges the two based on a common column (“Column1”).

2. Conditional Columns

Power Query also allows the creation of conditional columns. These columns generate values based on specific conditions in other columns:

let
    Source = Excel.Workbook(File.Contents("C:\YourData\Customers.xlsx"), null, true),
    CustomerSheet = Source{[Item="Customer",Kind="Sheet"]}[Data],
    #"Changed Type" = Table.TransformColumnTypes(CustomerSheet,{{"Column1", type text}, {"Column2", type text}}),
    #"Added Conditional Column" = Table.AddColumn(#"Changed Type", "Customer Type", each if [Column2] > 1000 then "Gold" else "Silver")
in
    #"Added Conditional Column"

In this scenario, a new column “Customer Type” is added to the Customers table. If the value in Column2 is greater than 1000, the customer is classified as “Gold”; otherwise, they’re classified as “Silver”.

3. Grouping Rows

Grouping rows is another powerful feature provided by Power Query. It allows you to summarize or aggregate your data:

let
    Source = Excel.Workbook(File.Contents("C:\YourData\Sales.xlsx"), null, true),
    SalesSheet = Source{[Item="Sales",Kind="Sheet"]}[Data],
    #"Changed Type" = Table.TransformColumnTypes(SalesSheet,{{"Column1", type text}, {"Column2", type text}}),
    #"Grouped Rows" = Table.Group(#"Changed Type", {"Column1"}, {{"Total", each List.Sum([Column2]), type number}})
in
    #"Grouped Rows"

In this code snippet, the data from Sales is grouped by Column1 (for instance, it could be a product category), and the total sum for each category is calculated and stored in the “Total” column.

Conclusion

These examples merely scratch the surface of what’s possible with Power Query. The platform is extremely flexible and powerful, allowing you to handle even the most complex data transformation tasks with relative ease. Unlocking its potential can drastically increase your efficiency in data analysis and make your Power BI reports more insightful.

With Power Query, the power to manipulate, transform, and visualize your data is literally at your fingertips. So, take the plunge and explore the powerful capabilities this tool has to offer. You’ll find that with a little bit of practice, you can take your data analysis to an entirely new level.

This blogpost was created with help from ChatGPT Pro

Using OpenAI and ElevenLabs APIs to Generate Compelling Voiceover Content: A Step-by-Step Guide

Voice technology has taken the world by storm, enabling businesses and individuals to bring text to life in a whole new way. In this blog post, we’ll walk you through how you can use OpenAI’s language model, GPT-3, in conjunction with ElevenLabs’ Text-to-Speech (TTS) API to generate compelling voiceover content.

Step 1: Setting Up Your Environment

First things first, you’ll need to make sure you have Python installed on your system. You can download it from the official Python website if you don’t have it yet. Once Python is set up, you’ll need to install the necessary libraries.

You can install the ElevenLabs and OpenAI Python libraries using pip:

pip install openai elevenlabs

Now that we have everything set up, let’s get started!

Step 2: Generating Text with OpenAI

We’ll start by using OpenAI’s GPT-3 model to generate some text. Before you can make API calls, you’ll need to sign up on the OpenAI website and get your API key.

Once you have your key, use it to set your API key in your environment:

import openai

openai.api_key = 'your-api-key'

Now you can generate some text using the openai.Completion.create function:

response = openai.Completion.create(
  engine="text-davinci-002",
  prompt="Translate the following English text to French: '{}'",
  max_tokens=60
)

The above code generates translations of English text to French. You can replace the prompt with any text you’d like to generate.

Step 3: Setting Up ElevenLabs API

Now that we have our text, we need to turn it into speech. That’s where ElevenLabs comes in.

Firstly, get your ElevenLabs API key from the ElevenLabs website. Then set up your environment:

from elevenlabs import set_api_key

set_api_key("<your-elevenlabs-api-key>")

Step 4: Adding a New Voice

Before we can generate speech, we need a voice. ElevenLabs allows you to add your own voices. Here’s how you can do it:

from elevenlabs import clone

voice = clone(
    name="Voice Name",
    description="A description of the voice",
    files=["./sample1.mp3", "./sample2.mp3"],
)

This code creates a new voice using the provided MP3 files. Be sure to replace Voice Name with a name for your voice, and A description of the voice with a fitting description.

Step 5: Generating Speech

Now that we have our voice, we can generate some speech:

from elevenlabs import generate

# Retrieve the generated text from the OpenAI's GPT-3 API
generated_text = response.choices[0].text.strip()

# Generate speech from the text using the created voice
audio = generate(text=generated_text, voice=voice)

In this code, generated_text is the text that was generated by OpenAI’s GPT-3 in Step 2. We then use that text to generate speech using the voice we created in Step 4 with ElevenLabs’ API.

And that’s it! You’ve now successfully used OpenAI’s GPT-3 and ElevenLabs’ TTS APIs to generate voiceover content from text created by a language model. You can now use this content in your applications, or just have some fun generating different voices and texts!

This blogpost was created with help from ChatGPT Pro

Calling the OpenAI API from a Microsoft Fabric Notebook

Microsoft Fabric notebooks are a versatile tool for developing Apache Spark jobs and machine learning experiments. They provide a web-based interactive surface for writing code with rich visualizations and Markdown text support.

In this blog post, we’ll walk through how to call the OpenAI API from a Microsoft Fabric notebook.

Preparing the Notebook

Start by creating a new notebook in Microsoft Fabric. Notebooks in Fabric consist of cells, which are individual blocks of code or text that can be run independently or as a group. You can add a new cell by hovering over the space between two cells and selecting ‘Code’ or ‘Markdown’.

Microsoft Fabric notebooks support four Apache Spark languages: PySpark (Python), Spark (Scala), Spark SQL, and SparkR. For this guide, we’ll use PySpark (Python) as the primary language.

You can specify the language for each cell using magic commands. For example, you can write a PySpark query using the %%pyspark magic command in a Scala notebook. But since our primary language is PySpark, we won’t need a magic command for Python cells.

Microsoft Fabric notebooks are integrated with the Monaco editor, which provides IDE-style IntelliSense for code editing, including syntax highlighting, error marking, and automatic code completions.

Calling the OpenAI API

To call the OpenAI API, we’ll first need to install the OpenAI Python client in our notebook. Add a new cell to your notebook and run the following command:

!pip install openai

Next, in a new cell, write the Python code to call the OpenAI API:

import openai

openai.api_key = 'your-api-key'

response = openai.Completion.create(
  engine="text-davinci-002",
  prompt="Translate the following English text to French: '{}'",
  max_tokens=60
)

print(response.choices[0].text.strip())

Replace 'your-api-key' with your actual OpenAI API key. The prompt parameter is the text you want the model to generate from. The max_tokens parameter is the maximum length of the generated text.

You can run the code in a cell by hovering over the cell and selecting the ‘Run Cell’ button or bypressing Ctrl+Enter. You can also run all cells in sequence by selecting the ‘Run All’ button.

Wrapping Up

That’s it! You’ve now called the OpenAI API from a Microsoft Fabric notebook. You can use this method to leverage the powerful AI models of OpenAI in your data science and machine learning experiments.

Always remember that if a cell is running for a longer time than expected, or you wish to stop execution for any reason, you can select the ‘Cancel All’ button to cancel the running cells or cells waiting in the queue.

I hope this guide has been helpful. Happy coding!

Please note that OpenAI’s usage policies apply when using their API. Be sure to understand these policies before using the API in your projects. Also, keep in mind that OpenAI’s API is a paid service, so remember to manage your usage to control costs.

Finally, it’s essential to keep your API key secure. Do not share it publicly or commit it in your code repositories. If you suspect that your API key has been compromised, generate a new one through the OpenAI platform.

This blogpost was created with help from ChatGPT Pro

The “eBay 1/1” Sports Cards Illusion: A Disingenuous Trend

When it comes to sports cards, there’s nothing quite as exciting as landing a true 1/1 card. For the uninitiated, 1/1 cards are unique in that they are one of a kind – the only one of their kind in the world. These cards hold a significant value in the hobby and are a dream for any serious collector. However, there is a new, disturbing trend emerging on eBay which threatens to cheapen the allure of these unique cards: the mislabelling of cards as an “eBay 1/1.”

What is an “eBay 1/1,” you ask? This is a moniker given by sellers to sports cards that are serial numbered but aren’t genuine 1/1 cards. The number on the card might be unique in some way – like 01/99 or 99/99 – but that’s a far cry from a true 1/1. Why is this happening? Quite simply, it’s a tactic to manipulate eBay’s search algorithm and show up in search results for actual 1/1 cards, attempting to cash in on the value and desirability of these genuinely unique pieces.

As a long-time collector and enthusiast of sports cards, I find this practice nothing short of disingenuous. It undermines the unique charm and allure that a true 1/1 card holds. The thrill of owning a card that is truly one-of-a-kind is diminished when the market is flooded with these pseudo 1/1 cards, sold under the pretense of rarity.

What’s more, it is incredibly frustrating for collectors who are seeking actual 1/1 cards. The search results are swamped with these misleading listings, making it an arduous task to filter through them and find the genuine article. This misrepresentation is a disservice to the serious collector who is willing to invest their time and money into acquiring these special pieces.

I fear the long-term ramifications of this trend. If this practice continues unchecked, it risks devaluing the whole concept of 1/1 cards. And that, dear fellow collectors, is something we simply cannot allow.

The responsibility lies with us, the community of collectors and enthusiasts, to discourage this trend. Sellers should uphold the integrity of the hobby and label their cards correctly. eBay, as the marketplace, also has a role to play in setting and enforcing listing standards to combat this. But we, as buyers, also hold power. It is essential that we be vigilant and discerning, and call out these deceptive listings when we see them.

It’s also crucial to educate newcomers to the hobby about this trend, so they can make informed decisions when building their collections. After all, the appeal of sports card collecting lies in its authenticity, and its value should be derived from genuine rarity and historical significance, not deceptive marketing tactics.

To conclude, let’s cherish the magic of true 1/1 cards and not let this deceptive “eBay 1/1” trend dilute the joy of this hobby we so love. A collective effort to uphold integrity can ensure the tradition of card collecting stays pure, preserving the thrill of the hunt and the joy of owning a piece of sporting history that is truly one-of-a-kind.

This blogpost was created with help from ChatGPT Pro

Reassessing the Presidency: Gerald Ford’s Underrated Tenure

When it comes to the list of U.S. Presidents, Gerald R. Ford is often lost in the shadows of his more historically influential predecessors and successors. Sandwiched between the infamous Richard Nixon and the charismatic Jimmy Carter, Ford’s short and seemingly unremarkable tenure as the 38th president is often overlooked. However, upon closer examination, it’s clear that Ford’s presidency marked a crucial period in American history.

Gerald Ford, the only U.S. President never elected to the vice presidency or presidency, assumed office during one of the nation’s most tumultuous times. Following the Watergate Scandal and the Vietnam War, public trust in the government was at an all-time low. This article aims to shed light on why Gerald Ford’s tenure, although brief, deserves more credit than it often receives.

Unprecedented Stability During Unstable Times

First and foremost, Ford’s stabilizing influence during a time of national uncertainty cannot be overstated. After Nixon’s resignation, the nation was reeling. Ford’s first task was to restore faith in the executive branch and bring stability back to the White House. He did this not with grandiose speeches or flashy policies, but with his quiet integrity and straightforward approach to governance. His words during his inauguration, “Our long national nightmare is over”, succinctly addressed the nation’s troubled psyche, offering hope and a fresh start.

Pardoning Nixon: A Controversial but Necessary Act

One of the most controversial decisions of Ford’s presidency was the pardoning of Richard Nixon. Initially, this was seen as a betrayal, causing a significant drop in his approval rating. However, in retrospect, it is generally viewed as a necessary act. The country was already battered by the Watergate scandal, and a drawn-out trial would have only perpetuated the public’s focus on the ordeal. By pardoning Nixon, Ford intended to facilitate national healing and redirect the country’s focus to pressing issues such as the economy and foreign policy.

The Helsinki Accords: A Triumph in Foreign Policy

Ford’s diplomatic acumen was evident in his handling of the Cold War tensions. The Helsinki Accords, signed in 1975, was a major diplomatic victory for the Ford administration. The Accords significantly improved East-West relations and laid the groundwork for greater human rights recognition within the Soviet Union. It also bolstered the United States’ position as a global peacemaker, a role which was severely tested in the aftermath of the Vietnam War.

Economic Policies and Fiscal Responsibility

Amid an era marked by “stagflation”, Ford demonstrated fiscal prudence and innovative economic management. His ‘WIN’ (Whip Inflation Now) program, while often criticized, demonstrated a commitment to involving the American public in economic solutions. Although its immediate success was limited, it represented an early recognition of the need for public-private partnerships in tackling complex issues.

Additionally, Ford’s decision to veto numerous spending bills showcased his fiscal responsibility, a principle he staunchly believed in. Despite criticism, his commitment to reducing the federal deficit should be appreciated as an early, if not fully successful, attempt to rein in government spending.

Conclusion: A Man of Integrity in the Oval Office

While his presidency may lack the defining moments that shape popular perception, Gerald Ford’s time in office was marked by steady leadership, careful decision-making, and a commitment to the American people. His approach to foreign policy, economic challenges, and national healing following the Watergate scandal reveals a president who prioritized the country’s needs above personal political gain.

It’s time to reassess Gerald Ford’s legacy. His tenure, characterized by integrity, courage, and an unwavering commitment to the nation, merits greater recognition. In these divisive times, we could all stand to learn a thing or two from President Ford’s understated but impactful leadership. Perhaps then, we can appreciate why Gerald Ford was, indeed, an underrated president.

This blogpost was created with help from ChatGPT Pro.

In Praise of Gilius Thunderhead: The Unrivalled Hero of Golden Axe

Golden Axe, an iconic video game that graced our arcades and homes in the late 80s and early 90s, gifted us with three memorable characters to choose from: the mighty warrior Ax Battler, the powerful amazon Tyris Flare, and the seemingly diminutive dwarf, Gilius Thunderhead. Yet, while each character brought unique skills to the battlefield, there’s a compelling argument to be made for Gilius Thunderhead being the best character of the trio. Despite his small stature, Gilius embodies the true essence of a hero and steals the spotlight with his distinct advantages.

Firstly, Gilius Thunderhead was unique in his ability to strike a fine balance between speed and power. While Ax Battler was strong, he could often be too slow, especially against nimble enemies. Tyris, while fast, often lacked the raw power required to deal with larger foes. Gilius Thunderhead, on the other hand, walked the middle path, exhibiting both strength and agility in equal measure. His compact size gave him the advantage of being difficult to hit, while his axe swung with a force that could be rivalled only by Ax Battler himself. This perfect blend of speed and power made Gilius an ideal choice for players who wanted the best of both worlds.

Secondly, Gilius Thunderhead boasted the most effective magic in the game. Although he had fewer magic pots than the other characters, the power he commanded with his thunder magic was unparalleled. His magic was not only visually stunning but also devastating to enemies. Each spell was a spectacle, a flash of light followed by a screen-wide assault that wiped out enemies in a single strike. Tyris and Ax may have had more magic pots, but they often needed to use all of them to achieve the same level of destruction that Gilius could with just a couple.

Moreover, Gilius Thunderhead’s character design and personality were as impactful as his abilities. His small stature and fierce demeanor belied a strength and determination that were truly inspiring. His gruff, no-nonsense attitude, combined with his unwavering dedication to vanquishing evil, made him a truly compelling character. Gilius was the underdog who rose above his limitations, a testament to the fact that true strength comes not from physical prowess alone, but from the courage and determination within.

Lastly, Gilius Thunderhead’s gameplay offered a unique challenge that made Golden Axe even more enjoyable. Mastering Gilius required a strategic approach, as players had to make the best use of his speed, power, and magic to overcome the game’s various obstacles and enemies. This added a layer of depth to the game that made playing as Gilius both challenging and rewarding.

In conclusion, Gilius Thunderhead is a testament to the fact that size doesn’t always matter in the realm of heroes. His balanced attributes, formidable magic, and indomitable spirit make him a character worth celebrating in Golden Axe. Whether you’re revisiting this classic or experiencing it for the first time, remember: underestimate the dwarf, and you may just find yourself on the wrong end of a thunderbolt.

This blogpost was created with help from ChatGPT Pro

Unveiling Microsoft OneLake: A Unified Intelligent Data Foundation

Microsoft recently introduced OneLake, a part of Microsoft Fabric, designed to accelerate data potential for the era of AI. One Lake provides a unified intelligent data foundation for all analytic workloads, integrating Power BI, Data Factory, and the next generation of Synapse. This solution offers customers a high-performing and easy-to-manage modern analytics solution.

OneLake: The OneDrive for All Your Data

OneLake provides a single data lake for your entire organization. For every Fabric tenant, there will always be exactly one OneLake, never two, never zero. There is no infrastructure to manage or set up. The concept of a tenant is a unique benefit of a SaaS service. It allows Microsoft to automatically provide a single management and governance boundary for the entire organization, which is ultimately under the control of a tenant admin.

Breaking down Data Silos with OneLake

OneLake aims to provide a data lake as a service without you needing to build it yourself. It enables different business groups to work independently without going through a central gatekeeper. Different workspaces allow different parts of the organization to work independently while still contributing to the same data lake. Each workspace can have its own administrator, access control, region, and capacity for billing.

OneLake: Spanning the Globe

OneLake covers this by spanning the globe as well. Different workspaces can reside in different regions. This means that any data stored in those workspaces will also reside in those countries. OneLake is built on top of Azure Data Lake Storage Gen2 under the covers. It will use multiple storage accounts in different regions, however, OneLake will virtualize them into one logical lake.

OneLake: Open Data Lake

OneLake is not just a Fabric data lake or a Microsoft data lake, it is an open data lake. In addition to being built on ADLS Gen2, OneLake supports the same ADLS Gen2 APIs and SDKs, making it compatible with existing ADLs applications, including Azure Databricks and Azure HDInsights.

OneLake: One Copy

OneLake with One Copy aims to get the most value possible out of a single copy of data without data movement or duplication. It allows data to be virtualized into a single data product without data movement, data duplication, or changing the ownership of the data.

OneLake: One Security

One Security is a feature in active development that aims to let you secure the data once and use it anywhere. One Security will bring a shared universal security model which you will define in OneLake. These security definitions will live alongside the data itself. This is an important detail. Security will live with the data rather than living downstream in the serving or presentation layers.

OneLake Data Hub

The OneLake Data Hub is the central location within Fabric to discover, manage, and reuse data. It serves all users from data engineer to business user. Data can easily be discovered by its domain, for example, Finance, HR, or Sales, so users find what actually matters to them.

In conclusion, OneLake is a game-changer in the world of data management and analytics. It provides a unified, intelligent data foundation that breaks down data silos, enabling organizations to harness the full potential of their data in the era of AI.

This blogpost was created with help from ChatGPT Pro.

Building a Lakehouse Architecture with Microsoft Fabric: A Comprehensive Guide

Microsoft Fabric is a powerful tool for data engineers, enabling them to build out a lakehouse architecture for their organizational data. In this blog post, we will walk you through the key experiences that Microsoft Fabric.

Creating a Lakehouse

A lakehouse is a new experience that combines the power of a data lake and a data warehouse. It serves as a central repository for all Fabric data. To create a lakehouse, you start by creating a new lakehouse artifact and giving it a name. Once created, you land in the empty Lakehouse Explorer.

Importing Data into the Lakehouse

There are several ways to bring data into the lakehouse. You can upload files and folders from your local machine, use data flows (a low-code tool with hundreds of connectors), or leverage the pipeline copy activity to bring in petabytes of data at scale. Most of the marketing data in the lakehouse is in Delta tables, which are automatically created with no additional effort. You can easily explore the tables, see their schema, and even view the underlying files.

Adding Unstructured Data

In addition to structured data, you might want to add some unstructured customer reviews to accompany your campaign data. If this data already exists in storage, you can simply point to it with no data movement necessary. This is done by adding a new shortcut, which allows you to create a virtual table and virtual files inside your lakehouse. Shortcuts enable you to select from a variety of sources, including lakehouses and warehouses in Fabric, but also external storage like ADLS Gen 2 and even Amazon S3.

Leveraging the Data

Once all your data is ready in the lakehouse, there are many ways to use it. As a data engineer or data scientist, you can open up the lakehouse in a notebook and leverage Spark to continue transforming the data or build a machine learning model. As a SQL professional, you can navigate to the SQL endpoint of the lakehouse where you can write SQL queries, create views and functions, all on top of the same Delta tables. As a business analyst, you can navigate to the built-in modeling view and start developing your BI data model directly in the same warehouse experience.

Configuring your Spark Environment

As an administrator, you can configure the Spark environment for your data engineers. This is done in the capacity admin portal, where you can access the Spark compute settings for data engineers and data scientists. You can set a default runtime and default Spark properties, and also turn on the ability for workspace admins to configure their own custom Spark pools.

Collaborative Data Development

Microsoft Fabric also provides a rich developer experience, enabling users to collaborate easily, work with their lakehouse data, and leverage the power of Spark. You can view your colleagues’ code updates in real time, install ML libraries for your project, and use the built-in charting capabilities to explore your data. The notebook has a built-in resource folder which makes it easy to store scripts or other code files you might need for the project.

In conclusion, Microsoft Fabric provides a frictionless experience for data engineers building out their enterprise data lakehouse and can easily democratize this data for all users in an organization. It’s a powerful tool that combines the power of a data lake and a data warehouse, providing a comprehensive solution for data engineering tasks.

This blogpost was created with help from ChatGPT Pro

How Spark Compute Works in Microsoft Fabric

Spark Compute is a key component of Microsoft Fabric, the end-to-end, unified analytics platform that brings together all the data and analytics tools that organizations need. Spark Compute enables data engineering and data science scenarios on a fully managed Spark compute platform that delivers unparalleled speed and efficiency.

What is Spark Compute?

Spark Compute is a way of telling Spark what kind of resources you need for your data analysis tasks. You can give your Spark pool a name, and choose how many and how big the nodes (the machines that do the work) are. You can also tell Spark how to adjust the number of nodes depending on how much work you have.

Spark Compute operates on OneLake, the data lake service that powers Microsoft Fabric. OneLake provides a single place to store and access all your data, whether it is structured, semi-structured, or unstructured. OneLake also supports data from other sources, such as Amazon S3 and (soon) Google Cloud Platform³.

Spark Compute supports both batch and streaming scenarios, and integrates with various tools and frameworks, such as Azure OpenAI Service, Azure Machine Learning, Databricks, Delta Lake, and more. You can use Spark Compute to perform data ingestion, transformation, exploration, analysis, machine learning, and AI tasks on your data.

How to use Spark Compute?

There are two ways to use Spark Compute in Microsoft Fabric: starter pools and custom pools.

Starter pools

Starter pools are a fast and easy way to use Spark on the Microsoft Fabric platform within seconds. You can use Spark sessions right away, instead of waiting for Spark to set up the nodes for you. This helps you do more with data and get insights quicker.

Starter pools have Spark clusters that are always on and ready for your requests. They use medium nodes that will dynamically scale-up based on your Spark job needs. Starter pools also have default settings that let you install libraries quickly without slowing down the session start time.

You only pay for starter pools when you are using Spark sessions to run queries. You don’t pay for the time when Spark is keeping the nodes ready for you.

Custom pools

A custom pool is a way of creating a tailored Spark pool according to your specific data engineering and data science requirements. You can customize various aspects of your custom pool, such as:

  • Node size: You can choose from different node sizes that offer different combinations of CPU cores, memory, and storage.
  • Node count: You can specify the minimum and maximum number of nodes you want in your custom pool.
  • Autoscale: You can enable autoscale to let Spark automatically adjust the number of nodes based on the workload demand.
  • Dynamic allocation: You can enable dynamic allocation to let Spark dynamically allocate executors (the processes that run tasks) based on the workload demand.
  • Libraries: You can install libraries from various sources, such as Maven, PyPI, CRAN, or your workspace.
  • Properties: You can configure custom properties for your custom pool, such as spark.executor.memory or spark.sql.shuffle.partitions.

Creating a custom pool is free; you only pay when you run a Spark job on the pool. If you don’t use your custom pool for 2 minutes after your job is done, Spark will automatically delete it. This is called the \”time to live\” property, and you can change it if you want.

If you are a workspace admin, you can also create default custom pools for your workspace, and make them the default option for other users. This way, you can save time and avoid setting up a new custom pool every time you run a notebook or a Spark job.

Custom pools take about 3 minutes to start, because Spark has to get the nodes from Azure.

Conclusion

Spark Compute is a powerful and flexible way of using Spark on Microsoft Fabric. It enables you to perform various data engineering and data science tasks on your data stored in OneLake or other sources. It also offers different options for creating and managing your Spark pools according to your needs and preferences.

If you want to learn more about Spark Compute in Microsoft Fabric, check out these resources:

This blogpost was created with help from ChatGPT Pro and Bing