Build Your Own Spark Job Doctor in Microsoft Fabric

Microsoft Fabric makes it incredibly easy to spin up Spark workloads: notebooks, Lakehouse pipelines, dataflows, SQL + Spark hybrid architectures—the whole buffet.

What’s still hard?
Knowing why a given Spark job is slow, expensive, or flaky.

  • A Lakehouse pipeline starts timing out.
  • A notebook that used to finish in 5 minutes is now taking 25.
  • Costs spike because one model training job is shuffling half the lake.

You open the Spark UI, click around a few stages, stare at shuffle graphs, and say the traditional words of Spark debugging:

“Huh.”

This is where an AI assistant should exist.

In this post, we’ll walk through how to build exactly that for Fabric Spark: a Job Doctor that:

  • Reads Spark telemetry from your Fabric environment
  • Detects issues like skew, large shuffles, spill, and bad configuration
  • Uses a large language model (LLM) to explain what went wrong
  • Produces copy-pasteable fixes in Fabric notebooks / pipelines
  • Runs inside Fabric using Lakehouses, notebooks, and Azure AI models

This is not a fake product announcement. This is a blueprint you can actually build.


What Is the Fabric “Job Doctor”?

At a high level, the Job Doctor is:

A Fabric-native analytics + AI layer that continuously reads Spark job history, detects common performance anti-patterns, and generates human-readable, prescriptive recommendations.

Concretely, it does three main things:

  1. Collects Spark job telemetry from Fabric
    • Spark application metrics (tasks, stages, shuffles, spills)
    • Spark logs & events (Driver/Executor/Event logs)
    • Optional query plans
    • Spark session configs
  2. Analyzes jobs using rules + metrics
    • Identifies skew, large shuffles, spill, etc.
    • Scores each job run and surfaces the top issues.
  3. Uses an LLM to generate a “diagnosis sheet”
    • Root cause in plain English
    • Fixes with code + config snippets for Fabric Spark
    • Expected impact on performance/cost

Let’s build it step by step, Fabric-style.


Part 1: Getting Spark Telemetry Out of Fabric

Before you can diagnose anything, you need the raw signals. In Fabric, there are three main ways to see what Spark is doing:

  1. Fabric Apache Spark diagnostic emitter → logs/metrics for each application
  2. Spark application details (UI / REST)
  3. In-job logging from notebooks/pipelines (e.g., configs, query plans)

You don’t have to use all three, but combining them gives you enough for a very capable Job Doctor.


1. Configure the Fabric Apache Spark Diagnostic Emitter

The core telemetry pipeline starts with the Fabric Apache Spark diagnostic emitter, configured on a Fabric environment.

At a high level, you:

  1. Create or use an environment for your Spark workloads.
  2. Configure one or more diagnostic emitters on that environment.
  3. Point each emitter to a sink such as:
    • Azure Storage (Blob, ADLS)
    • Azure Log Analytics
    • Azure Event Hubs

For example, an emitter to Azure Storage might be configured (conceptually) like this:

spark.synapse.diagnostic.emitters: MyStorageEmitter
spark.synapse.diagnostic.emitter.MyStorageEmitter.type: AzureStorage
spark.synapse.diagnostic.emitter.MyStorageEmitter.categories: DriverLog,ExecutorLog,EventLog,Metrics
spark.synapse.diagnostic.emitter.MyStorageEmitter.uri: https://<account>.blob.core.windows.net/<container>/<folder>
spark.synapse.diagnostic.emitter.MyStorageEmitter.auth: AccessKey
spark.synapse.diagnostic.emitter.MyStorageEmitter.secret: <storage-access-key>

Once this is in place:

  • Every Spark application (notebook, job, pipeline activity that spins up Spark) will emit diagnostic records.
  • Those records land as JSON lines describing driver logs, executor logs, Spark listener events, and metrics.

From there, you can:

  • If using Storage: Create a shortcut in a Lakehouse pointing at the container/folder.
  • If using Log Analytics: Build KQL queries or export into Fabric (e.g., into a KQL DB or as files you later hydrate into a Lakehouse).

We’ll assume the storage pattern for the rest of this post:

Spark app → Fabric environment with diagnostic emitter → Azure Storage → OneLake shortcut → Lakehouse.


2. Shape of the Raw Logs (and Why You’ll Normalize Them)

The emitter doesn’t give you a nice stageId / taskId table out of the box. Instead, you’ll see records like:

{
  "timestamp": "2024-05-01T12:34:56Z",
  "category": "Metrics",
  "applicationId": "app-20240501123456-0001",
  "properties": {
    "metricName": "executorRunTime",
    "stageId": 4,
    "taskId": 123,
    "value": 9182,
    "otherFields": "..."
  }
}

Or an EventLog record with a payload that looks like the Spark listener event.

To build a Job Doctor, you’ll:

  1. Read the JSON lines into Fabric Spark
  2. Explode / parse the properties payload
  3. Aggregate per-task metrics into per-stage metrics for each application

We’ll skip the exact parsing details (they depend on how you set up the emitter and which events/metrics you enable) and assume that after a normalization job, you have a table with one row per (applicationId, stageId, taskId).

That’s what the next sections use.


3. Capturing Query Plans in Fabric (Optional, but Powerful)

Spark query plans are gold when you’re trying to answer why a stage created a huge shuffle or why a broadcast join didn’t happen.

There isn’t yet a first-class “export query plan as JSON” API in PySpark, but in Fabric notebooks you can use a (semi-internal) trick that works today:

import json

df = ...  # some DataFrame you care about

# Advanced / internal: works today but isn't a public, stable API
plan_json = json.loads(df._jdf.queryExecution().toJSON())

You can also log the human-readable plan:

df.explain(mode="formatted")  # documented mode, prints a detailed plan

To persist the JSON plan for the Job Doctor, tie it to the Spark application ID:

from pyspark.sql import Row

app_id = spark.sparkContext.applicationId

spark.createDataFrame(
    [Row(applicationId=app_id, query_plan_json=plan_json)]
).write.mode("append").saveAsTable("job_doctor.query_plans")

A couple of caveats you should mention in a real blog:

  • _jdf.queryExecution().toJSON() is not guaranteed to be stable across Spark versions. It’s an advanced, “use at your own risk” trick.
  • You don’t need to capture plans for every single query—just key bottleneck notebooks or critical pipelines.

Even capturing a subset massively improves the quality of LLM explanations.


4. Capture Spark Config for Each Run

Fabric Spark lets you set configs at:

  • Environment / pool level (resource profiles, environment settings)
  • Notebook / job level (spark.conf.set(...))
  • Pipeline activity level (Spark job settings)

Inside the running Spark job, you can capture the effective session config like this:

from pyspark.sql import Row

app_id = spark.sparkContext.applicationId
conf_dict = dict(spark.conf.getAll())  # session-level config

config_rows = [
    Row(applicationId=app_id, key=k, value=v)
    for k, v in conf_dict.items()
]

spark.createDataFrame(config_rows).write.mode("append").saveAsTable("job_doctor.spark_conf")

Now the Job Doctor can say things like:

  • “AQE was disabled for this job.”
  • “Shuffle partitions was left at default 200, which is low for your data size.”

You’re building a small “Job Doctor mart” inside Fabric:

  • job_doctor.raw_logs (from emitter)
  • job_doctor.stage_metrics (aggregated)
  • job_doctor.stage_issues (rule engine output)
  • job_doctor.spark_conf (per-application configs)
  • job_doctor.query_plans (optional)

All keyed by applicationId.


Part 2: Loading and Normalizing Spark Logs in a Fabric Lakehouse

Let’s assume you’ve done one-time wiring:

  • Azure Storage container with Spark diagnostics
  • OneLake shortcut from that container into a Lakehouse
  • A Fabric Spark notebook attached to that Lakehouse

From that notebook:

logs_df = spark.read.json("Tables/spark_diagnostics_raw")  # or your shortcut path
display(logs_df.limit(10))

You’ll see something like:

  • timestamp
  • category (DriverLog, ExecutorLog, EventLog, Metrics, …)
  • applicationId
  • properties (nested JSON with stage/task/metric detail)

The normalization step (which you can run as a scheduled pipeline) should:

  1. Filter down to metrics/events relevant for performance (e.g. task / stage metrics)
  2. Extract stageId, taskId, executorRunTime, shuffleReadBytes, etc., into top-level columns
  3. Persist the result as job_doctor.task_metrics (or similar)

For the rest of this post, we’ll assume you’ve already done that and have a table with columns:

  • applicationId
  • stageId
  • taskId
  • executorRunTime
  • shuffleReadBytes
  • shuffleWriteBytes
  • memoryBytesSpilled
  • diskBytesSpilled

Aggregating Stage Metrics in Fabric

Now we want to collapse per-task metrics into per-stage metrics per application.

In a Fabric notebook:

from pyspark.sql import functions as F

task_metrics = spark.table("job_doctor.task_metrics")

stage_metrics = (
    task_metrics
    .groupBy("applicationId", "stageId")
    .agg(
        F.countDistinct("taskId").alias("num_tasks"),
        F.sum("executorRunTime").alias("total_task_runtime_ms"),
        # Depending on Spark version, you may need percentile_approx instead
        F.expr("percentile(executorRunTime, 0.95)").alias("p95_task_runtime_ms"),
        F.max("executorRunTime").alias("max_task_runtime_ms"),
        F.sum("shuffleReadBytes").alias("shuffle_read_bytes"),
        F.sum("shuffleWriteBytes").alias("shuffle_write_bytes"),
        F.sum("memoryBytesSpilled").alias("memory_spill_bytes"),
        F.sum("diskBytesSpilled").alias("disk_spill_bytes"),
    )
    .withColumn(
        "skew_ratio",
        F.col("max_task_runtime_ms") /
        F.when(F.col("p95_task_runtime_ms") == 0, 1).otherwise(F.col("p95_task_runtime_ms"))
    )
    .withColumn("shuffle_read_mb", F.col("shuffle_read_bytes") / (1024**2))
    .withColumn("shuffle_write_mb", F.col("shuffle_write_bytes") / (1024**2))
    .withColumn(
        "spill_mb",
        (F.col("memory_spill_bytes") + F.col("disk_spill_bytes")) / (1024**2)
    )
)

stage_metrics.write.mode("overwrite").saveAsTable("job_doctor.stage_metrics")

This gives you a Fabric Lakehouse table with:

  • skew_ratio
  • shuffle_read_mb
  • shuffle_write_mb
  • spill_mb
  • p95_task_runtime_ms
  • num_tasks, total_task_runtime_ms, etc.

You can run this notebook:

  • On a schedule via a Data Pipeline
  • Or as a Data Engineering job configured in the workspace

Part 3: Adding a Rule Engine Inside Fabric

Now that the metrics are in a Lakehouse table, let’s add a simple rule engine in Python.

This will run in a Fabric notebook (or job) and write out issues per stage.

from pyspark.sql import Row, functions as F

stage_metrics = spark.table("job_doctor.stage_metrics")

# For simplicity, we'll collect to the driver here.
# This is fine if you don't have thousands of stages.
# For very large workloads, you'd instead do this via a UDF / mapInPandas / explode.
stage_rows = stage_metrics.collect()

Define some basic rules:

def detect_issues(stage_row):
    issues = []

    # 1. Skew detection
    if stage_row.skew_ratio and stage_row.skew_ratio > 5:
        issues.append({
            "issue_id": "SKEWED_STAGE",
            "severity": "High",
            "details": f"Skew ratio {stage_row.skew_ratio:.1f}"
        })

    # 2. Large shuffle
    total_shuffle_mb = (stage_row.shuffle_read_mb or 0) + (stage_row.shuffle_write_mb or 0)
    if total_shuffle_mb > 10_000:  # > 10 GB
        issues.append({
            "issue_id": "LARGE_SHUFFLE",
            "severity": "High",
            "details": f"Total shuffle {total_shuffle_mb:.1f} MB"
        })

    # 3. Excessive spill
    if (stage_row.spill_mb or 0) > 1_000:  # > 1 GB
        issues.append({
            "issue_id": "EXCESSIVE_SPILL",
            "severity": "Medium",
            "details": f"Spill {stage_row.spill_mb:.1f} MB"
        })

    return issues

Apply the rules and persist the output:

issue_rows = []

for r in stage_rows:
    for issue in detect_issues(r):
        issue_rows.append(Row(
            applicationId=r.applicationId,
            stageId=r.stageId,
            issue_id=issue["issue_id"],
            severity=issue["severity"],
            details=issue["details"]
        ))

issues_df = spark.createDataFrame(issue_rows)

issues_df.write.mode("overwrite").saveAsTable("job_doctor.stage_issues")

Now you have a table of Spark issues detected per run inside your Lakehouse.

Later, the LLM will use these as structured hints.


Part 4: Bringing in the LLM — Turning Metrics into Diagnosis

So far, everything has been pure Spark in Fabric.

Now we want a model (e.g., Azure AI “Models as a Service” endpoint or Azure OpenAI) to turn:

  • job_doctor.stage_metrics
  • job_doctor.stage_issues
  • job_doctor.spark_conf
  • job_doctor.query_plans

into an actual diagnosis sheet a human can act on.

In Fabric, this is simplest from a Spark notebook using a Python HTTP client.

Below, I’ll show the pattern using an Azure AI serverless model endpoint (the one that uses model: "gpt-4.1" in the body).


1. Prepare the Prompt Payload

First, fetch the data for a single Spark application:

import json
from pyspark.sql import functions as F

app_id = "app-20240501123456-0001"  # however you pick which run to diagnose

stages_df = spark.table("job_doctor.stage_metrics").where(F.col("applicationId") == app_id)
issues_df = spark.table("job_doctor.stage_issues").where(F.col("applicationId") == app_id)
conf_df   = spark.table("job_doctor.spark_conf").where(F.col("applicationId") == app_id)
plans_df  = spark.table("job_doctor.query_plans").where(F.col("applicationId") == app_id)

stages_json = stages_df.toPandas().to_dict(orient="records")
issues_json = issues_df.toPandas().to_dict(orient="records")
conf_json   = conf_df.toPandas().to_dict(orient="records")
plans_json  = plans_df.toPandas().to_dict(orient="records")  # likely 0 or 1 row

Then build a compact but informative prompt:

prompt = f"""
You are an expert in optimizing Apache Spark jobs running on Microsoft Fabric.

Here is summarized telemetry for one Spark application (applicationId={app_id}):

Stage metrics (JSON):
{json.dumps(stages_json, indent=2)}

Detected issues (JSON):
{json.dumps(issues_json, indent=2)}

Spark configuration (key/value list):
{json.dumps(conf_json, indent=2)}

Query plans (optional, may be empty):
{json.dumps(plans_json, indent=2)}

Your tasks:
1. Identify the top 3–5 performance issues for this run.
2. For each, explain the root cause in plain language.
3. Provide concrete fixes tailored for Fabric Spark, including:
   - spark.conf settings (for notebooks/jobs)
   - suggestions for pipeline settings where relevant
   - SQL/DataFrame code snippets
4. Estimate likely performance impact (e.g., "30–50% reduction in runtime").
5. Call out any risky or unsafe changes that should be tested carefully.

Return your answer as markdown.
"""


2. Call an Azure AI Model from Fabric Spark

For the serverless “Models as a Service” endpoint, the pattern looks like this:

import os
import requests

# Example: using Azure AI Models as a Service
# AZURE_AI_ENDPOINT might look like: https://models.inference.ai.azure.com
AZURE_AI_ENDPOINT = os.environ["AZURE_AI_ENDPOINT"]
AZURE_AI_KEY      = os.environ["AZURE_AI_KEY"]

MODEL = "gpt-4.1"  # or whatever model you've enabled

headers = {
    "Content-Type": "application/json",
    "api-key": AZURE_AI_KEY,
}

body = {
    "model": MODEL,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant for optimizing Spark jobs on Microsoft Fabric."},
        {"role": "user", "content": prompt},
    ],
}

resp = requests.post(
    f"{AZURE_AI_ENDPOINT}/openai/chat/completions",
    headers=headers,
    json=body,
)

resp.raise_for_status()
diagnosis = resp.json()["choices"][0]["message"]["content"]

If you instead use a provisioned Azure OpenAI resource, the URL shape is slightly different (you call /openai/deployments/<deploymentName>/chat/completions and omit the model field), but the rest of the logic is identical.

At this point, diagnosis is markdown you can:

  • Render inline in the notebook with displayHTML
  • Save into a Lakehouse table
  • Feed into a Fabric semantic model for reporting

Part 5: What the Job Doctor’s Output Looks Like in Fabric

A good Job Doctor output for Fabric Spark might look like this (simplified):


🔎 Issue 1: Skewed Stage 4 (skew ratio 12.3)

What I see

  • Stage 4 has a skew ratio of 12.3 (max task runtime vs. p95).
  • This stage also reads ~18.2 GB via shuffle, which amplifies the imbalance.

Likely root cause

A join or aggregation keyed on a column where a few values dominate (e.g. a “default” ID, nulls, or a small set of hot keys). One partition ends up doing far more work than the others.

Fabric-specific fixes

In your notebook or job settings, enable Adaptive Query Execution and skew join handling:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

If the query is in SQL (Lakehouse SQL endpoint), enable AQE at the session/job level through Spark configuration.

If one side of the join is a small dimension table, add a broadcast hint:

SELECT /*+ BROADCAST(dim) */ f.*
FROM fact f
JOIN dim
  ON f.key = dim.key;

Estimated impact:
30–50% reduction in total job runtime, depending on how skewed the key distribution is.


📦 Issue 2: Large Shuffle in Stage 2 (~19.7 GB)

What I see

  • Stage 2 reads ~19.7 GB via shuffle.
  • Shuffle partitions are set to 200 (Spark default).

Likely root cause

A join or aggregation is shuffling nearly the full dataset, but parallelism is low given the data volume. That leads to heavy tasks and increased risk of spill.

Fabric-specific fixes

Increase shuffle partitions for this job:

spark.conf.set("spark.sql.shuffle.partitions", "400")

For pipelines, set this at the Spark activity level under Spark configuration, or through your Fabric environment’s resource profile if you want a new default.

Also consider partitioning by the join key earlier in the pipeline:

df = df.repartition("customer_id")

Estimated impact:
More stable runtimes and reduced likelihood of spill; wall-clock improvements if your underlying capacity has enough cores.


💾 Issue 3: Spill to Disk (~1.8 GB) in Stage 3

What I see

  • Stage 3 spills ~1.8 GB to disk.
  • This correlates with under-parallelism or memory pressure.

Fabric-specific fixes

  • Adjust cluster sizing via Fabric capacity / resource profiles (enough cores + memory per core).
  • Increase spark.sql.shuffle.partitions as above.
  • Avoid wide transformations producing huge intermediate rows early in the job; materialize smaller, more selective intermediates first.

You can persist the diagnosis text into a table:

from pyspark.sql import Row

spark.createDataFrame(
    [Row(applicationId=app_id, diagnosis_markdown=diagnosis)]
).write.mode("append").saveAsTable("job_doctor.diagnoses")

Then you can build a Power BI report in Fabric bound to:

  • job_doctor.diagnoses
  • job_doctor.stage_metrics
  • job_doctor.stage_issues

to create a “Spark Job Health” dashboard where:

  • Rows = recent Spark runs
  • Columns = severity, duration, shuffle size, spill, etc.
  • A click opens the AI-generated diagnosis for that run

All inside the same workspace.


Part 6: Stitching It All Together in Fabric

Let’s recap the full Fabric-native architecture.

1. Telemetry Ingestion (Environment / Emitter)

  • Configure a Fabric environment for your Spark workloads.
  • Add a Fabric Apache Spark diagnostic emitter to send logs/metrics to:
    • Azure Storage (for Lakehouse shortcuts), or
    • Log Analytics / Event Hubs if you prefer KQL or streaming paths.
  • (Optional) From notebooks/pipelines, capture:
    • Spark configs → job_doctor.spark_conf
    • Query plans → job_doctor.query_plans

2. Normalization Job (Spark / Data Pipeline)

  • Read raw diagnostics from Storage via a Lakehouse shortcut.
  • Parse and flatten the records into per-task metrics.
  • Aggregate per-stage metrics → job_doctor.stage_metrics.
  • Evaluate rule engine → job_doctor.stage_issues.
  • Persist all of this into Lakehouse tables.

3. AI Diagnosis Job (Spark + Azure AI Models)

  • For each new (or most expensive / slowest) application:
    • Pull stage metrics, issues, configs, and query plans from Lakehouse.
    • Construct a structured prompt.
    • Call your Azure AI / Azure OpenAI endpoint from a Fabric Spark notebook.
    • Store the markdown diagnosis in job_doctor.diagnoses.

4. User Experience

  • Fabric Notebook
    • A “Run Job Doctor” cell or button that takes applicationId, calls the model, and displays the markdown inline.
  • Data Pipeline / Job
    • Scheduled daily to scan all runs from yesterday and generate diagnoses automatically.
  • Power BI Report in Fabric
    • “Spark Job Health” dashboard showing:
      • Top slowest/most expensive jobs
      • Detected issues (skew, large shuffle, spill, config problems)
      • AI recommendations, side-by-side with raw metrics

Everything lives in one Fabric workspace, using:

  • Lakehouses for data
  • Spark notebooks / pipelines for processing
  • Azure AI models for reasoning
  • Power BI for visualization

Why a Fabric-Specific Job Doctor Is Worth Building

Spark is Spark, but in Fabric the story is different:

  • Spark jobs are tied closely to Lakehouses, Pipelines, Dataflows, and Power BI.
  • You already have a single control plane for capacity, governance, cost, and monitoring.
  • Logs, metrics, and reports can live right next to the workloads they describe.

That makes Fabric an ideal home for a Job Doctor:

  • No extra infrastructure to stand up
  • No random side services to glue together
  • The telemetry you need is already flowing; you just have to catch and shape it
  • AI can sit directly on top of your Lakehouse + monitoring data

With some Spark, a few Lakehouse tables, and an LLM, you can give every data engineer and analyst in your organization a “Spark performance expert” that’s always on call.

I’ve included a sample notebook you can use to get started on your Job Doctor today!


This post was created with help from (and suggested to me) by ChatGPT Pro using the 5.1 Thinking Model

Time to Automate: Why Sports Card Grading Needs an AI Revolution

As I head to the National for the first time, this is a topic I have been thinking about for quite some time, and a recent video inspired me to put this together with help from ChatGPT’s o3 model doing deep research. Enjoy!

Introduction: Grading Under the Microscope

Sports card grading is the backbone of the collectibles hobby – a PSA 10 vs PSA 9 on the same card can mean thousands of dollars of difference in value. Yet the process behind those grades has remained stubbornly old-fashioned, relying on human eyes and judgment. In an age of artificial intelligence and computer vision, many are asking: why hasn’t this industry embraced technology for more consistent, transparent results? The sports card grading industry is booming (PSA alone graded 13.5 million items in 2023, commanding ~78% of the market), but its grading methods have seen little modernization. It’s a system well overdue for a shakeup – and AI might be the perfect solution.

The Human Element: Trusted but Inconsistent

For over 30 years, Professional Sports Authenticator (PSA) has set the standard in grading, building a reputation for expertise and consistency . Many collectors trust PSA’s human graders to spot subtle defects and assess a card’s overall appeal in ways a machine allegedly cannot. This trust and track record are why PSA-graded cards often sell for more than those graded by newer, tech-driven companies. Human graders can apply nuanced judgment – understanding vintage card print idiosyncrasies, knowing how an odd factory cut might affect eye appeal, etc. – which some hobbyists still value.

However, the human touch has undeniable downsides. Grading is inherently subjective: two experienced graders might assign different scores to the same card. Mood, fatigue, or unconscious bias can creep in. And the job is essentially a high-volume, low-wage one, meaning even diligent graders face burnout and mistakes in a deluge of submissions. Over the pandemic boom, PSA was receiving over 500,000 cards per week, leading to a backlog of 12+ million cards by early 2021. They had to suspend submissions for months and hire 1,200 new employees to catch up. Relying purely on human labor proved to be a bottleneck – an expensive, slow, and error-prone way to scale. Inconsistencies inevitably arise under such strain, frustrating collectors who crack cards out of their slabs and resubmit them hoping for a higher grade on a luckier day. This “grading lottery” is accepted as part of the hobby, but it shouldn’t be.

Anecdotes of inconsistency abound: Collectors tell stories of a card graded PSA 7 on one submission coming back PSA 8 on another, or vice versa. One hobbyist recounts cracking a high-grade vintage card to try his luck again – only to have it come back with an even lower grade, and eventually marked as “trimmed” by a different company. While such tales may be outliers statistically, they underscore a core point: human grading isn’t perfectly reproducible. As one vintage card expert put it, in a high-volume environment “mistakes every which way will happen” . The lack of consistency not only erodes collector confidence but actively incentivizes wasteful behavior like repeated resubmissions.

Published Standards, Unpredictable Results

What’s ironic is that the major grading companies publish clear grading standards. PSA’s own guide, for instance, specifies that a Gem Mint 10 card must be centered 55/45 or better on the front (no worse than 60/40 for a Mint 9), with only minor flaws like a tiny print spot allowed. Those are numeric thresholds that a computer can measure with pixel precision. Attributes like corner sharpness, edge chipping, and surface gloss might seem more subjective, but they can be quantified too – e.g. by analyzing images for wear patterns or gloss variance. In other words, the criteria for grading a card are largely structured and known.

If an AI system knows that a certain scratch or centering offset knocks a card down to a 9, it will apply that rule uniformly every time. A human, by contrast, might overlook a faint scratch at 5pm on a Friday or be slightly lenient on centering for a popular rookie card. The unpredictability of human grading has real consequences: collectors sometimes play “submitter roulette,” hoping their card catches a grader on a generous day. This unpredictability is so entrenched that an entire subculture of cracking and resubmitting cards exists, attempting to turn PSA 9s into PSA 10s through persistence. It’s a wasteful practice that skews population reports and costs collectors money on extra fees – one that could be curbed if grading outcomes were consistent and repeatable.

A Hobby Tailor-Made for AI

Trading cards are an ideal use-case for AI and computer vision. Unlike, say, comic books or magazines (which have dozens of pages, staples, and complex wear patterns to evaluate), a sports card is a simple, two-sided object of standard size. Grading essentially boils down to assessing four sub-criteria – centering, corners, edges, surface – according to well-defined guidelines. This is exactly the kind of structured visual task that advanced imaging systems excel at. Modern AI can scan a high-resolution image of a card and detect microscopic flaws in an instant. Machine vision doesn’t get tired or biased; it will measure a border centering as 62/38 every time, without rounding up to “approximately 60/40” out of sympathy.

In fact, several companies have proven that the technology is ready. TAG Grading (Technical Authentication & Grading) uses a multi-patented computer vision system to grade cards on a 1,000-point scale that maps to the 1–10 spectrum. Every TAG slab comes with a digital report pinpointing every defect, and the company boldly touts “unrivaled accuracy and consistency” in grading. Similarly, Arena Club (co-founded by Derek Jeter) launched in 2022 promising AI-assisted grading to remove human error. Arena Club’s system scans each card and produces four sub-grades plus an overall grade, with a detailed report of flaws. “You can clearly see why you got your grade,” says Arena’s CTO, highlighting that AI makes grading consistent across different cards and doesn’t depend on the grader. In other words, the same card should always get the same grade – the ultimate goal of any grading process.

Even PSA itself has dabbled in this arena. In early 2021, PSA acquired Genamint Inc., a tech startup focused on automated card diagnostics. The idea was to integrate computer vision that could measure centering, detect surface issues or alterations, and even “fingerprint” each card to track if the same item gets resubmitted. PSA’s leadership acknowledged that bringing in technology would allow them to grade more cards faster while improving accuracy. Notably, one benefit of Genamint’s card fingerprinting is deterring the crack-and-resubmit cycle by recognizing cards that have been graded before. (One can’t help but wonder if eliminating resubmissions – and the extra fees they generate – was truly in PSA’s financial interest, which might explain why this fingerprinting feature isn’t visibly advertised to collectors.)

The point is: AI isn’t some far-off fantasy for card grading – it’s here. Multiple firms have developed working systems that scan cards, apply the known grading criteria, and produce a result with blinding speed and precision. A newly launched outfit, Zeagley Grading, showcased in 2025 a fully automated AI grading platform that checks “thousands of high-resolution checkpoints” on each card’s surface, corners, and edges. Zeagley provides a QR-coded digital report with every slab explaining exactly how the grade was determined, bringing transparency to an area long criticized for its opacity. The system is so confident in its consistency that they’ve offered a public bounty: crack a Zeagley-slabbed card and resubmit it – if it doesn’t come back with the exact same grade, they’ll pay you $1,000. That is the kind of repeatability collectors dream of. It might sound revolutionary, but as Zeagley’s founders themselves put it, “What we’re doing now isn’t groundbreaking at all – it’s what’s coming next that is.” In truth, grading a piece of glossy cardboard with a machine should be straightforward in 2025. We have the tech – it’s the will to use it that’s lagging.

Why the Slow Adoption? (Ulterior Motives?)

If AI grading is so great, why haven’t the big players fully embraced it? The resistance comes from a mix of practical and perhaps self-serving reasons. On the practical side, companies like PSA and Beckett have decades of graded cards in circulation. A sudden shift to machine-grading could introduce slight changes in standards – for example, the AI might technically grade tougher on centering or surface than some human graders have historically. This raises a thorny question: would yesterday’s PSA 10 still be a PSA 10 under a new automated system? The major graders are understandably cautious about undermining the consistency (or at least continuity) of their past population reports. PSA’s leadership has repeatedly stated that their goal is to assist human graders with technology, not replace them. They likely foresee a gradual integration where AI catches the easy stuff – measuring centering, flagging obvious print lines or dents – and humans still make the final judgment calls, keeping a “human touch” in the loop.

But there’s also a more cynical view in hobby circles: the status quo is just too profitable. PSA today is bigger and more powerful than ever – flush with record revenue from the grading boom and enjoying market dominance (grading nearly 4 out of every 5 cards in the hobby ). The lack of consistency in human grading actually drives more business for them. Think about it: if every card got a perfectly objective grade, once and for all, collectors would have little reason to ever resubmit a card or chase a higher grade. The reality today is very different. Many collectors will crack out a PSA 9 and roll the dice again, essentially paying PSA twice (or more) for grading the same card, hoping for that elusive Gem Mint label. There’s an entire cottage industry of group submitters and dealers who bank on finding undergraded cards and bumping them up on resubmission. It’s not far-fetched to suggest that PSA has little incentive to eliminate that lottery aspect of grading. Even PSA’s own Genamint acquisition, which introduced card fingerprinting to catch resubmissions, could be a double-edged sword – if they truly used it to reject previously-graded cards, it might dry up a steady stream of repeat orders. As one commentator wryly observed, “if TAG/AI grading truly becomes a problem [for PSA], PSA would integrate it… but for now it’s not, so we have what we get.” In other words, until the tech-savvy upstarts start eating into PSA’s market share, PSA can afford to move slowly.

There’s also the human factor of collector sentiment. A segment of the hobby simply prefers the traditional approach. The idea of a seasoned grader, someone who has handled vintage Mantles and modern Prizm rookies alike, giving their personal approval still carries weight. Some collectors worry that an algorithm might be too severe, or fail to appreciate an intangible “eye appeal” that a human might allow. PSA’s brand is built not just on plastic slabs, but on the notion that people – trusted experts – are standing behind every grade. Handing that over entirely to machines risks alienating those customers who aren’t ready to trust a computer over a well-known name. As a 2024 article on the subject noted, many in the hobby still see AI grading as lacking the “human touch” and context for certain subjective calls. It will take time for perceptions to change.

Still, these concerns feel less convincing with each passing year. New collectors entering the market (especially from the tech world) are often stunned at how low-tech the grading process remains. Slow, secretive, and expensive is how one new AI grading entrant described the incumbents – pointing to the irony that grading fees can scale up based on card value (PSA charges far more to grade a card worth $50,000 than a $50 card), a practice seen by some as a form of price-gouging. An AI-based service, by contrast, can charge a flat rate per card regardless of value, since the work and cost to the company are the same whether the card is cheap or ultra-valuable. These startups argue they have no conflicts of interest – the algorithm doesn’t know or care what card it’s grading, removing any unconscious bias or temptation to cut corners for high-end clients. In short, technology promises an objective fairness that the current system can’t match.

Upstart Efforts: Tech Takes on the Titans

In the past few years, a number of new grading companies have popped up promising to disrupt the market with technology. Hybrid Grading Approach (HGA) made a splash in 2021 by advertising a “hybrid” model: cards would be initially graded by an AI-driven scanner, then verified by two human graders. HGA also offered flashy custom labels and quicker turnaround times. For a moment, it looked like a strong challenger, but HGA’s momentum stalled amid reports of inconsistent grades and operational missteps (underscoring that fancy tech still needs solid execution behind it).

TAG Grading, mentioned earlier, took a more hardcore tech route – fully computerized grading with proprietary methods and a plethora of data provided to the customer. TAG’s system, however, launched with limitations: initially they would only grade modern cards (1989-present) and standard card sizes, likely because their imaging system needed retraining or reconfiguration for vintage cards, thicker patch cards, die-cuts, etc. This highlights a challenge for any AI approach: it must handle the vast variety of cards in the hobby, from glossy Chrome finish to vintage cardboard, and even odd-shaped or acetates. TAG chose to roll out methodically within its comfort zone. The result has been rave reviews from a small niche – those who tried TAG often praise the “transparent grading report” showing every flaw – but TAG remains a tiny player. Despite delivering what many consider a better mousetrap, they have not come close to denting PSA’s dominance.

Arena Club, backed by a sports icon’s star power, also discovered how tough it is to crack the market. As Arena’s CFO acknowledged, “PSA is dominant, which isn’t news to anyone… it’s definitely going to be a longer road” to convince collectors. Arena pivoted to position itself not just as a grading service but a one-stop marketplace (offering vaulting, trading, even “Slab Pack” digital reveal products). In doing so, they tacitly recognized that trying to go head-to-head purely on grading technology wasn’t enough. Collectors still gravitate to PSA’s brand when it comes time to sell big cards – even if the Arena Club slab has the same card graded 10 with an AI-certified report, many buyers simply trust PSA more. By late 2024, Arena Club boasted that cards in their AI-grade slabs “have sold for almost the same prices as cards graded by PSA” , but “almost the same” implicitly concedes a gap. The market gives PSA a premium, deservedly or not.

New entrants continue to appear. Besides TAG and Arena, we’ve seen firms like AGS (Automated Grading Systems) targeting the Pokémon and TCG crowd with a fully automated “Robograding” service. AGS uses lasers and scanners to find microscopic defects “easily missed by even the best human graders,” and provides sub-scores and images of each flaw. Their pitch is that they grade 10x faster, more accurately, and cheaper – yet their footprint in the sports card realm is still small. The aforementioned Zeagley launched in mid-2025 with a flurry of press, even offering on-site instant grading demos at card shows. Time will tell if they fare any better. So far, each tech-focused upstart has either struggled to gain trust or found itself constrained to a niche, while PSA is grading more cards than ever (up 21% in volume last year ) and even raising prices for premium services. In effect, the incumbents have been able to watch these challengers from a position of strength and learn from their mistakes.

PSA: Bigger Than Ever, But Is It Better?

It’s worth noting that PSA hasn’t been entirely tech-averse. They use advanced scanners at intake, have implemented card fingerprinting and alteration-detection algorithms (courtesy of Genamint) behind the scenes, and likely use software to assist with centering measurements. Nat Turner, who leads PSA’s parent company, is a tech entrepreneur himself and clearly sees the long-term importance of innovation. But from an outsider’s perspective, PSA’s grading process in 2025 doesn’t look dramatically different to customers than it did a decade ago: you send your cards in, human graders assign a 1–10 grade, and you get back a slab with no explanation whatsoever of why your card got the grade it did. If you want more info, you have to pay for a higher service tier and even then you might only get cursory notes. This opacity is increasingly hard to justify when competitors are providing full digital reports by default. PSA’s stance seems to be that its decades of experience are the secret sauce – that their graders’ judgment cannot be fully replicated by a machine. It’s a defensible position given their success, but also a conveniently self-serving one. After all, if the emperor has ruled for this long, why acknowledge any need for a new way of doing things?

However, cracks (no pun intended) are showing in the facade. The hobby has not forgotten the controversies where human graders slipped up – like the scandal a few years ago where altered cards (trimmed or recolored) managed to get past graders and into PSA slabs, rocking the trust in the system. Those incidents suggest that even the best experts can be duped or make errors that a well-trained AI might catch via pattern recognition or measurement consistency. PSA has since leaned on technology more for fraud detection (Genamint’s ability to spot surface changes or match a card to a known altered copy is likely in play), which is commendable. But when it comes to the routine task of assigning grades, PSA still largely keeps that as an art, not a science.

To be fair, PSA (and rivals like Beckett and SGC) will argue that their human-led approach ensures a holistic assessment of each card. A grader might overlook one tiny print dot if the card is otherwise exceptional, using a bit of reasonable discretion, whereas an algorithm might deduct points rigidly. They might also argue that collectors themselves aren’t ready to accept a purely AI-driven grade, especially for high-end vintage where subtle qualities matter. There’s truth in the notion that the hobby’s premium prices often rely on perceived credibility – and right now, PSA’s brand carries more credibility than a newcomer robot grader in the eyes of many auction bidders. Thus, PSA can claim that by sticking to (and refining) their human grading process, they’re actually protecting the market’s trust and the value of everyone’s collections. In short: if it ain’t broke (for them), why fix it?

The Case for Change: Consistency, Transparency, Trust

Despite PSA’s dominance, the case for an AI-driven shakeup in grading grows stronger by the day. The hobby would benefit enormously from grading that is consistent, repeatable, and explainable. Imagine a world where you could submit the same card to a grading service twice and get the exact same grade, with a report detailing the precise reasons. That consistency would remove the agonizing second-guessing (“Should I crack this 9 and try again?”) and refocus everyone on the card itself rather than the grading lottery. It would also level the playing field for collectors – no more wondering if a competitor got a PSA 10 because they’re a bulk dealer who “knows a guy” or just got lucky with a lenient grader. Every card, every time, held to the same standard.

Transparency is another huge win. It’s 2025 – why are we still largely in the dark about why a card got a 8 vs a 9? With AI grading, detailed digital grading reports are a natural output. Companies like TAG and Zeagley are already providing these: high-res imagery with circles or arrows pointing out each flaw, sub-scores for each category, and even interactive web views to zoom in on problem areas. Not only do these reports educate collectors on what to look for, they also keep the grading company honest. If the report says your card’s surface got an 8.5/10 due to a scratch and you, the collector, don’t see any scratch, you’d have grounds to question that grade immediately. In the current system, good luck – PSA simply doesn’t answer those questions beyond generic responses. Transparency would greatly increase trust in grading, ironically the very thing PSA prides itself on. It’s telling that one of TAG’s slogans is creating “transparency, accuracy, and consistency for every card graded.” Those principles are exactly what collectors have been craving.

Then there’s the benefit of speed and efficiency. AI grading systems can process cards much faster than humans. A machine can work 24/7, doesn’t need coffee breaks, and can ramp up throughput just by adding servers or scanners (whereas PSA had to physically expand to a new 130,000 sq ft facility and hire dozens of new graders to increase capacity ). Faster grading means shorter turnaround times and fewer backlogs. During the pandemic, we saw how a huge backlog can virtually paralyze the hobby’s lower end – people stopped sending cheaper cards because they might not see them back for a year. If AI were fully deployed, the concept of a months-long queue could vanish. Companies like AGS brag about “grading 10,000 cards in a day” with automation; even if that’s optimistic, there’s no doubt an algorithm can scale far beyond what manual grading ever could.

Lastly, consider cost. A more efficient grading process should eventually reduce costs for both the company and the consumer. Some of the new AI graders are already undercutting on price – e.g. Zeagley offering grading at $9.99 a card for a 15-day service – whereas PSA’s list price for its economy tier floats around $19–$25 (and much more for high-value or faster service). Granted, PSA has the brand power to charge a premium, but in a competitive market a fully automated solution should be cheaper to operate per card. That savings can be passed on, which encourages more participation in grading across all value levels.

The ChatGPT Experiment: DIY Grading with AI

Perhaps the clearest proof that card grading is ripe for automation is that even hobbyists at home can now leverage AI to grade their cards in a crude way. Incredibly, thanks to advances in AI like OpenAI’s ChatGPT, a collector can snap high-resolution photos of a card (front and back), feed them into an AI model, and ask for a grading opinion. Some early adopters have done just that. One collector shared that he’s “been using ChatGPT to help hypothetically grade cards” – he uploads pictures and asks, “How does the centering look? What might this card grade on PSA’s scale?” The result? “Since I’ve started doing this, I have not received a grade lower than a 9” on the cards he chose to submit. In other words, the AI’s assessment lined up with PSA’s outcomes well enough that it saved him from sending in any card that would grade less than mint. It’s a crude use of a general AI chatbot, yet it highlights something powerful: even consumer AI can approximate grading if given the standards and some images.

Right now, examples like this are more curiosities than commonplace. Very few collectors are actually using ChatGPT or similar tools to pre-grade on a regular basis. But it’s eye-opening that it’s even possible. As image recognition AI improves and becomes more accessible, one can imagine a near-future app where you scan your card with your phone and get an instantaneous grade estimate, complete with highlighted flaws. In fact, some apps and APIs already claim to do this for pre-grading purposes. It’s not hard to imagine a scenario where collectors start publicly verifying or challenging grades using independent AI tools – “Look, here’s what an unbiased AI thinks of my card versus what PSA gave it.” If those two views diverge often enough, it could pressure grading companies to be more transparent or consistent. At the very least, it empowers collectors with more information about their own cards’ condition.

Embracing the Future: It’s Time for Change

The sports card grading industry finds itself at a crossroads between tradition and technology. PSA is king – and by many metrics, doing better than ever in terms of business – but that doesn’t mean the system is perfect or cannot be improved. Relying purely on human judgment in 2025, when AI vision systems are extraordinarily capable, feels increasingly antiquated. The hobby deserves grading that is as precise and passion-driven as the collectors themselves. Adopting AI for consistent and repeatable standards should be an easy call: it would eliminate so many pain points (inconsistency, long waits, lack of feedback) that collectors grumble about today.

Implementing AI doesn’t have to mean ousting the human experts entirely. A hybrid model could offer the best of both worlds – AI for objectivity and humans for oversight. For example, AI could handle the initial inspection, quantifying centering to the decimal and finding every tiny scratch, then a human grader could review the findings, handle any truly subjective nuances (like eye appeal or print quality issues that aren’t easily quantified), and confirm the final grade. The human becomes more of a quality control manager rather than the sole arbiter. This would massively speed up the process and tighten consistency, while still keeping a human in the loop to satisfy those who want that assurance. Over time, as the AI’s track record builds trust, the balance could shift further toward full automation.

Ultimately, the adoption of AI in grading is not about devaluing human expertise – it’s about capturing that expertise in a reproducible way. The best graders have an eye for detail; the goal of AI is to have 1000 “eyes” for detail and never blink. Consistency is king in any grading or authentication field. Imagine if two different coin grading experts could look at the same coin and one says “MS-65” and the other “MS-67” – coin collectors would be up in arms. And yet, in cards we often tolerate that variability as normal. We shouldn’t. Cards may differ subtly in how they’re produced (vintage cards often have rough cuts that a computer might flag as edge damage, for instance), so it’s important to train the AI on those nuances. But once trained, a machine will apply the standard exactly, every single time. That level of fairness and predictability would enhance the hobby’s integrity.

It might take more time – and perhaps a serious competitive threat – for the giants like PSA to fully embrace an AI-driven model. But the winds of change are blowing. A “technological revolution in grading” is coming; one day we’ll look back and wonder how we ever trusted the old legacy process, as one tech expert quipped. The smarter companies will lead that revolution rather than resist it. Collectors, too, should welcome the change: an AI shakeup would make grading more of a science and less of a gamble. When you submit a card, you should be confident the grade it gets is the grade it deserves, not the grade someone felt like giving it that day. Consistency. Transparency. Objectivity. These shouldn’t be revolutionary concepts, but in the current state of sports card grading, they absolutely are.

The sports card hobby has always been a blend of nostalgia and innovation. We love our cardboard heroes from the past, but we’ve also embraced new-age online marketplaces, digital card breaks, and blockchain authentication. It’s time the critical step of grading catches up, too. Whether through an industry leader finally rolling out true AI grading, or an upstart proving its mettle and forcing change, collectors are poised to benefit. The technology is here, the need is obvious, and the hobby’s future will be brighter when every slabbed card comes with both a grade we can trust and the data to back it up. The sooner we get there, the better for everyone who loves this game

Harnessing the Power of Code Interpreter Beta in ChatGPT Plus: A Deep Dive into eBay Purchases

As we continue to navigate the digital age, data has become an integral part of our lives, and understanding this data is more important than ever. One of the recent advancements in data analysis is the Code Interpreter Beta feature in ChatGPT Pro, a powerful tool that brings programming and data analysis to your fingertips. To illustrate its power, let’s dive into an example where we analyze a dataset of eBay purchases.

Imagine that you’ve been given a CSV file containing information about eBay transactions. The dataset includes the date of purchase, the title of the listing, the total price of the item, and the name of the seller. At first glance, it may seem like a daunting task to extract meaningful information from this raw data. However, with the Code Interpreter Beta feature, we can easily navigate this data and gain valuable insights.

Data Cleaning

The first step in our analysis involves cleaning our data. We noticed that the ‘Total Price’ column in our dataset was stored as text rather than numerical values, which prevents us from performing numerical computations. The power of the Code Interpreter Beta feature shines here as it enables us to quickly convert the ‘Total Price’ column into a numerical format using a few lines of Python code.

Descriptive Statistics

Once our data is cleaned, we can start delving into the interesting stuff: gaining insights from our data. We can use the Code Interpreter Beta feature to easily compute descriptive statistics for the ‘Total Price’ column. With a few lines of code, we can determine the average purchase price, the variability in prices, and the range of prices.

  • Count: There are 2988 transactions in the dataset with a valid ‘Total Price’.
  • Mean: The average price of a purchase is approximately $42.29.
  • Standard Deviation: The standard deviation, a measure of price variability, is approximately $339.88. This high value suggests there’s a large variation in purchase prices.
  • Minimum: The least expensive purchase in the dataset cost $0.01.
  • 25% (1st Quartile): 25% of the purchases were priced at $5.00 or less.
  • Median (50% / 2nd Quartile): The median price, which separates the higher half and the lower half of the purchase prices, is $11.96. This means that 50% of the purchases were less than $11.96, and 50% were more.
  • 75% (3rd Quartile): 75% of the purchases were priced at $27.00 or less.
  • Maximum: The most expensive purchase in the dataset cost $15,200.00.

Purchase Trends Over Time

Next, we wanted to investigate the trends in eBay purchases over time. Utilizing the date and time functionalities offered by the Code Interpreter Beta feature, we were able to group our purchases by month and year. This allowed us to visualize the number of purchases over time, revealing an increasing trend in purchases from November 2001 to January 2022.

Price Distribution

Finally, we looked into the distribution of purchase prices. Through the visualization tools available in the Code Interpreter Beta feature, we could easily generate histograms to visualize this data. We found that the majority of the purchases were in the lower price range, with a few purchases significantly more expensive. To focus on the majority of transactions, we created a histogram for purchases priced at $200 or less, revealing that most purchases were in the $0-$50 range.

Conclusion

In conclusion, the Code Interpreter Beta feature is a powerful tool that opens up a world of possibilities for data analysis. With its help, we were able to transform a raw eBay transactions dataset into meaningful insights, uncovering trends in purchase prices and their distribution. Its seamless integration of data cleaning, statistical computation, and visualization capabilities makes it a potent tool for any data enthusiast. Whether you’re a seasoned data scientist or a curious beginner, the Code Interpreter Beta feature brings data analysis to your fingertips.

This blogpost was created with help from ChatGPT Pro

Leverage ChatGPT to Debug and Refine Code Snippets in Blog Posts

While the internet is an incredibly rich resource for programmers and developers of all levels, it’s not always a flawless one. You’ve likely found yourself in situations where you’ve sourced a snippet of code from a blog post, only to find it doesn’t work quite as expected. Luckily, ChatGPT, a state-of-the-art language model developed by OpenAI, is here to help debug and refine code snippets you encounter in blog posts.

Introduction to ChatGPT

ChatGPT is a variant of the GPT-4 architecture, a powerful, AI-based language model. It can understand, respond to, and create human-like text based on prompts given to it. Thanks to OpenAI’s extensive training process, ChatGPT has learned from a wide variety of internet text. But, while it has a wealth of knowledge, it doesn’t know everything and should not be considered infallible or completely up-to-date.

However, despite its limitations, ChatGPT can be an incredibly valuable tool when it comes to understanding, correcting, and working through coding issues.

Using ChatGPT for Code Debugging

  1. Describe the Problem: Begin by describing the issue you’re facing in as much detail as possible. Be sure to mention the language you’re using, the code you’re trying to run, and any error messages you’re receiving. Here’s an example: "I'm trying to run a Python script that should sort a list of numbers in descending order. But, it keeps returning the list in ascending order. Here's the code: `my_list.sort()`. What am I doing wrong?"

    ChatGPT will then provide a response that should guide you to a solution, for instance:

    "In Python, the `sort()` method sorts a list in ascending order by default. If you want to sort the list in descending order, you need to add the reverse parameter to the `sort()` method like this: `my_list.sort(reverse=True)`. Try that and see if it fixes your issue."
  2. Review Code Snippets: When you come across code snippets in blog posts, you can present them to ChatGPT and ask for an explanation of what the code does. It can help you understand complex code structures and algorithms.
  3. Ask for Alternatives: If the code you’ve found doesn’t fit your needs perfectly, you can ask ChatGPT for an alternative approach. For example, if a Java code snippet uses a for-loop, but you’re more comfortable with while-loops, ChatGPT can help rewrite the code to suit your comfort zone.
  4. Error Messages: If a certain piece of code is giving you error messages, sharing those with ChatGPT could lead to a more effective solution. Error messages usually point to the part of the code where something is wrong, and ChatGPT can often provide guidance on what the error message means and how to fix it.
  5. Learn Best Practices: ChatGPT can also provide advice on coding best practices. Whether you’re looking to understand the most efficient way to write a certain piece of code, or you want to make sure your code is as readable as possible, you can ask ChatGPT for tips.

Some Caveats

While ChatGPT can be incredibly helpful, there are a few things to keep in mind:

  1. Not Always Up-to-date: As of now, ChatGPT’s training only includes data up until September 2021. As such, it might not be aware of more recent language updates or coding practices.
  2. Doesn’t Execute Code: ChatGPT doesn’t execute code—it makes predictions based on the information it was trained on. Thus, while it can often provide useful guidance, it won’t be able to catch runtime errors or issues that arise from specific environmental setups.
  3. Check Multiple Sources: AI can be a powerful tool, but it’s essential to cross-verify the information. Always consider consulting official documentation, forums, or other resources as well.

All things considered, ChatGPT can be a great tool to help debug and refine code snippets from blog posts. Whether you’re a beginner looking to understand new concepts or an experienced developer looking for a quick solution, interacting with ChatGPT can often lead you in the right direction.

This blogpost was created with help from ChatGPT Pro

Using OpenAI and ElevenLabs APIs to Generate Compelling Voiceover Content: A Step-by-Step Guide

Voice technology has taken the world by storm, enabling businesses and individuals to bring text to life in a whole new way. In this blog post, we’ll walk you through how you can use OpenAI’s language model, GPT-3, in conjunction with ElevenLabs’ Text-to-Speech (TTS) API to generate compelling voiceover content.

Step 1: Setting Up Your Environment

First things first, you’ll need to make sure you have Python installed on your system. You can download it from the official Python website if you don’t have it yet. Once Python is set up, you’ll need to install the necessary libraries.

You can install the ElevenLabs and OpenAI Python libraries using pip:

pip install openai elevenlabs

Now that we have everything set up, let’s get started!

Step 2: Generating Text with OpenAI

We’ll start by using OpenAI’s GPT-3 model to generate some text. Before you can make API calls, you’ll need to sign up on the OpenAI website and get your API key.

Once you have your key, use it to set your API key in your environment:

import openai

openai.api_key = 'your-api-key'

Now you can generate some text using the openai.Completion.create function:

response = openai.Completion.create(
  engine="text-davinci-002",
  prompt="Translate the following English text to French: '{}'",
  max_tokens=60
)

The above code generates translations of English text to French. You can replace the prompt with any text you’d like to generate.

Step 3: Setting Up ElevenLabs API

Now that we have our text, we need to turn it into speech. That’s where ElevenLabs comes in.

Firstly, get your ElevenLabs API key from the ElevenLabs website. Then set up your environment:

from elevenlabs import set_api_key

set_api_key("<your-elevenlabs-api-key>")

Step 4: Adding a New Voice

Before we can generate speech, we need a voice. ElevenLabs allows you to add your own voices. Here’s how you can do it:

from elevenlabs import clone

voice = clone(
    name="Voice Name",
    description="A description of the voice",
    files=["./sample1.mp3", "./sample2.mp3"],
)

This code creates a new voice using the provided MP3 files. Be sure to replace Voice Name with a name for your voice, and A description of the voice with a fitting description.

Step 5: Generating Speech

Now that we have our voice, we can generate some speech:

from elevenlabs import generate

# Retrieve the generated text from the OpenAI's GPT-3 API
generated_text = response.choices[0].text.strip()

# Generate speech from the text using the created voice
audio = generate(text=generated_text, voice=voice)

In this code, generated_text is the text that was generated by OpenAI’s GPT-3 in Step 2. We then use that text to generate speech using the voice we created in Step 4 with ElevenLabs’ API.

And that’s it! You’ve now successfully used OpenAI’s GPT-3 and ElevenLabs’ TTS APIs to generate voiceover content from text created by a language model. You can now use this content in your applications, or just have some fun generating different voices and texts!

This blogpost was created with help from ChatGPT Pro

Leveraging OpenAI for Creating Compelling Sample Datasets for Microsoft Fabric and Power BI

Data analysis and visualization are key components of business intelligence, and Power BI stands as a leading platform in this domain. A pivotal part of working with Power BI involves dealing with datasets. Unfortunately, it isn’t always easy to access or generate datasets that perfectly illustrate the capabilities of Power BI. This is where ChatGPT, OpenAI’s powerful language model, can lend a hand. Today, we’ll delve into how you can use ChatGPT to create intriguing sample datasets for use in Power BI.

Step 1: Understanding the Desired Data Structure

Before generating your data, it’s essential to understand the structure you require. In Power BI, data is often organized into tables that consist of rows (records) and columns (fields). For example, a simple customer database could contain fields such as CustomerID, Name, Email, Country, and Purchase Amount.

You can sketch out your desired table and decide the kind of data you need for each column. For instance, for a column like “Country,” you might want a mix of countries worldwide, while for “Purchase Amount,” you may need a range of numerical values.

Step 2: Defining the Data Parameters with ChatGPT

Once you understand the structure of the data, the next step is to translate it into a form that ChatGPT can generate. This would typically involve providing the model with examples or templates of what you want. For instance, if you are creating a dataset for customer analysis, you can instruct ChatGPT as follows:

    data_template = """
    {
    "CustomerID": "random alphanumeric string of length 6",
    "Name": "random human name",
    "Email": "random email",
    "Country": "random country",
    "Purchase Amount": "random number between 100 and 5000"
    }
    """

Remember, your instructions need to be as clear and specific as possible to generate the right type of data.

Step 3: Generating the Data

After setting the data parameters, you can now instruct ChatGPT to generate the data. If you’re using the OpenAI API, you can use the openai.ChatCompletion.create() method, passing in the model you’re using (for instance, ‘text-davinci-002’) and the data template you’ve defined. Your code may look something like this:

    import openai
    import json

    openai.api_key = 'your-api-key'
    
    response = openai.ChatCompletion.create(
      model="text-davinci-002",
      messages=[
          {"role": "system", "content": "You are a helpful assistant that's generating a data sample."},
          {"role": "user", "content": data_template},
      ]
    )

    data_sample = json.loads(response['choices'][0]['message']['content'])

    print(data_sample)

This code will generate a single record. If you want to generate more records, you can loop through the data generation process as many times as you need.

Step 4: Compiling and Formatting the Data

Now that you have the data generated, you can compile it into a dataset. Each generated record can be appended to a list which can later be converted into a DataFrame using pandas. Here is how it might look:

    import pandas as pd

    data_records = []

    # Assume you have generated n number of records
    for i in range(n):
        data_records.append(generate_data()) # generate_data function includes the data generation code from step 3

    # Convert the list to DataFrame
    df = pd.DataFrame(data_records)

    # Save the DataFrame as a CSV file for use in Power BI
    df.to_csv('sample_dataset.csv', index=False)

Step 5: Importing the Dataset into Power BI

After your CSV file is ready, you can now import it into Power BI. In Power BI Desktop, you can import your CSV file by navigating to “Home” > “External Data” > “CSV”. From here, you can start creating your visualizations and dashboards.

Here is the complete code as a single block for easier reference:

import openai
import json
import pandas as pd

def generate_data():
    # Define your data template
    data_template = """
    {
    "CustomerID": "random alphanumeric string of length 6",
    "Name": "random human name",
    "Email": "random email",
    "Country": "random country",
    "Purchase Amount": "random number between 100 and 5000"
    }
    """

    # Initialize the OpenAI API
    openai.api_key = 'your-api-key'
    
    # Create a chat completion with the model and data template
    response = openai.ChatCompletion.create(
      model="text-davinci-002",
      messages=[
          {"role": "system", "content": "You are a helpful assistant that's generating a data sample."},
          {"role": "user", "content": data_template},
      ]
    )
    # Parse the response to JSON and return
    return json.loads(response['choices'][0]['message']['content'])

# Initialize a list for storing your data
data_records = []

# Decide the number of records you want to generate
n = 100

# Generate n number of records
for i in range(n):
    data_records.append(generate_data())

# Convert the list to a DataFrame
df = pd.DataFrame(data_records)

# Save the DataFrame as a CSV file
df.to_csv('sample_dataset.csv', index=False)

This script will generate 100 records based on the data template, compile them into a DataFrame, and save it as a CSV file. You can then import this CSV file into Power BI. Remember to replace 'your-api-key' with your actual OpenAI API key. Also, ensure that you have installed the openai and pandas libraries, which you can do with pip:

pip install openai pandas

Wrapping Up

Creating compelling sample datasets for Power BI is crucial for demonstrating its capabilities and experimenting with various features. By leveraging ChatGPT, you can create datasets that are tailored to your specific needs and can offer varied insights when analyzed in Power BI.

It’s important to remember that while ChatGPT is a powerful tool, it’s not perfect. Be sure to verify and clean the generated data before using it in your Power BI projects to ensure accuracy in your data visualizations and analysis.

This blogpost was created with help from ChatGPT Pro

Why ChatGPT Won’t Replace the Need for Data Analysts in the Future

Introduction

Artificial Intelligence (AI) has come a long way in recent years, thanks to groundbreaking research and technological advancements. One of the most notable AI innovations is ChatGPT, a large language model developed by OpenAI. With its advanced capabilities in natural language processing and understanding, ChatGPT has significantly influenced many industries, including data analysis.

However, despite the impressive performance of ChatGPT, it is crucial to understand that it will not replace the need for data analysts in the future. In this blog post, we will explore the reasons behind this assertion and discuss the unique value that data analysts bring to the table.

  1. Human Insight and Intuition

While ChatGPT is highly proficient in understanding and processing language, it lacks the human intuition and insight that data analysts possess. Data analysts are not only trained to interpret complex patterns and trends but also to provide context and reasoning behind the data. This level of understanding goes beyond simply recognizing patterns and requires a deep knowledge of the domain and the ability to make informed decisions based on that understanding. ChatGPT, as powerful as it is, cannot replicate the human touch that data analysts provide.

  1. The Art of Asking the Right Questions

Data analysts are experts in asking the right questions to drive actionable insights. They know how to tailor their approach to suit the specific needs of their clients, and they understand the importance of asking probing questions to uncover hidden trends and opportunities. ChatGPT, as an AI language model, is inherently limited in this regard, as it can only respond to the questions it is given, rather than proactively identifying areas of interest or potential pitfalls.

  1. Domain-Specific Expertise

Data analysts often specialize in specific industries or domains, bringing a wealth of knowledge and expertise to their work. They are familiar with the unique challenges and trends that characterize their chosen fields and are well-equipped to provide tailored solutions to these problems. While ChatGPT can process and analyze vast amounts of information, it lacks the domain-specific expertise that makes data analysts invaluable assets to their organizations.

  1. Data Quality and Data Preparation

A large part of a data analyst’s job involves cleaning, preparing, and transforming raw data into a format that can be easily analyzed. This process requires a deep understanding of the data, its sources, and its limitations, as well as the ability to identify and address any inconsistencies or inaccuracies. ChatGPT, on the other hand, is not designed to handle this crucial aspect of data analysis. It is, therefore, necessary to have data analysts in place to ensure that the data being used is accurate, relevant, and reliable.

  1. Ethical Considerations

Data analysts are trained to consider the ethical implications of their work, ensuring that data is collected, analyzed, and presented in a responsible and unbiased manner. This ethical awareness is particularly important given the increasing concerns surrounding data privacy and the potential for misuse of information. ChatGPT, while an impressive tool, is not equipped to navigate these complex ethical issues and cannot replace the thoughtful, human-driven approach that data analysts bring to their work.

Conclusion

Although ChatGPT has undoubtedly revolutionized the way we interact with and process data, it is essential to recognize that it cannot replace the need for data analysts in the future. Data analysts offer a unique blend of human insight, domain-specific expertise, and ethical awareness that simply cannot be replicated by an AI language model. By working together, ChatGPT and data analysts can complement each other’s strengths and drive more efficient and effective data-driven decision-making processes.

This blogpost was created with help from ChatGPT Pro.

Empowering Individuals with ADHD: The Benefits of ChatGPT

Introduction

Attention Deficit Hyperactivity Disorder (ADHD) is a neurodevelopmental disorder that affects millions of people worldwide. Characterized by difficulty paying attention, impulsivity, and hyperactivity, it can make everyday tasks and routines a challenge. In today’s fast-paced world, where there is an abundance of information and distractions, managing ADHD symptoms can be even more daunting.

Enter ChatGPT, an artificial intelligence language model developed by OpenAI. This cutting-edge technology has the potential to assist people with ADHD in a variety of ways. In this blog post, we will explore how ChatGPT can empower individuals with ADHD by enhancing organization, improving focus, and facilitating communication.

  1. Enhancing Organization

One of the most common challenges faced by people with ADHD is staying organized. ChatGPT can help in several ways:

  • Task management: ChatGPT can serve as a digital assistant, helping users create and manage to-do lists, set reminders, and prioritize tasks.
  • Structuring information: By providing clear and concise summaries of lengthy articles or documents, ChatGPT can make it easier for individuals with ADHD to process and retain information.
  • Habit formation: ChatGPT can be programmed to provide consistent reminders and positive reinforcement, assisting users in establishing and maintaining healthy habits and routines.
  1. Improving Focus

Maintaining focus can be difficult for individuals with ADHD, but ChatGPT can help users stay on track:

  • Filtering distractions: ChatGPT can be used to monitor incoming messages and notifications, only alerting users to high-priority items, allowing them to remain focused on the task at hand.
  • Mindfulness and relaxation techniques: ChatGPT can provide guided meditation and breathing exercises to help users reduce anxiety, improve focus, and achieve a calmer state of mind.
  • Customized focus strategies: ChatGPT can offer personalized suggestions and techniques for improving focus based on users’ preferences and needs.
  1. Facilitating Communication

People with ADHD may struggle with communication, whether it’s expressing themselves clearly or interpreting complex information. ChatGPT can help bridge these gaps:

  • Clear expression: ChatGPT can assist users in organizing their thoughts and presenting them in a clear, coherent manner, making communication more effective.
  • Active listening: By summarizing conversations, ChatGPT can help users maintain focus during discussions, ensuring they are better able to engage and respond.
  • Emotional support: ChatGPT can offer non-judgmental listening and encouragement, providing emotional support to users who may feel overwhelmed or isolated due to their ADHD.
  1. Capturing Creativity and Overcoming “Blank Page Syndrome”

People with ADHD often experience bursts of creativity, but may struggle to capture and develop these ideas due to challenges with focus and organization. Additionally, facing a blank page can be daunting for anyone, but particularly for individuals with ADHD who might struggle with initiating tasks. ChatGPT can support users in these areas:

  • Idea capturing: ChatGPT can serve as a readily available tool to record and store creative ideas as they arise. Users can easily input their thoughts, knowing that ChatGPT will save and organize them for future reference.
  • Idea development: ChatGPT can help users expand on their creative ideas by offering suggestions, asking thought-provoking questions, and providing relevant information or resources to fuel their brainstorming process.
  • Overcoming “blank page syndrome”: ChatGPT can help users break through the initial barrier of starting a project by providing writing prompts, outlines, or even drafting a brief introduction. By giving users a starting point, ChatGPT helps mitigate the anxiety and procrastination often associated with beginning a task.

Conclusion

ChatGPT offers valuable tools that can help individuals with ADHD better manage their symptoms and improve their daily lives. By enhancing organization, improving focus, facilitating communication, capturing creativity, and overcoming the challenges of starting with a blank page, this AI-powered technology can empower people with ADHD to thrive in today’s fast-paced world. As we continue to refine and develop ChatGPT, the potential applications and benefits for individuals with ADHD and other neurodevelopmental disorders will only continue to grow.

This blogpost was created with help from ChatGPT Pro.

Integrating OpenAI with Power BI Paginated Reports using Azure Functions

Introduction: Power BI Paginated Reports are ideal for creating highly formatted, pixel-perfect layouts optimized for printing or PDF generation. By integrating OpenAI with Power BI Paginated Reports using Azure Functions, you can enhance your reports with AI-generated insights and content. This blog post provides a step-by-step guide on how to integrate OpenAI with Power BI Paginated Reports using Azure Functions and an intermediary SQL Server database.

Prerequisites:

  • An OpenAI API key
  • An Azure account
  • Power BI Report Builder
  • Basic knowledge of Power BI Paginated Reports, Azure Functions, and C#

Step 1: Create a SQL Server database

  1. Set up a SQL Server database or use an existing one.
  2. Create a new table to store the AI-generated content:
CREATE TABLE OpenAI_Responses (
ID INT PRIMARY KEY IDENTITY(1,1),
Prompt NVARCHAR(MAX),
GeneratedText NVARCHAR(MAX),
DateGenerated DATETIME
);

Step 2: Create an Azure Function to call the OpenAI API and store the AI-generated content in the SQL Server database

  1. Set up an Azure Function App with an HTTP trigger and follow the instructions to create a new function.
  2. Add the necessary NuGet packages to call the OpenAI API (e.g., OpenAI) and connect to SQL Server (e.g., System.Data.SqlClient).
  3. Modify the Azure Function code to call the OpenAI API, and insert the AI-generated content into the SQL Server table.
using System.Data.SqlClient;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using OpenAI;

public static class OpenAIIntegrationFunction
{
[FunctionName("OpenAIIntegrationFunction")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");

// Get the 'prompt' parameter from the query string
string prompt = req.Query["prompt"];

// Use OpenAI API Key
string openaiApiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY");

// Initialize the OpenAI API client
var apiClient = new OpenAIApiClient(apiKey: openaiApiKey);

// Set up the completion request
var completions = await apiClient.Completions.CreateAsync(
engine: "text-davinci-002",
new CompletionRequest
{
Prompt = prompt,
MaxTokens = 50,
N = 1,
Stop = null,
Temperature = 0.7,
}
);

string generated_text = completions.Choices[0].Text.Trim();

// Replace with your SQL Server connection string
string connectionString = "your_sql_server_connection_string";

using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
using (SqlCommand command = new SqlCommand("INSERT INTO OpenAI_Responses (Prompt, GeneratedText, DateGenerated) VALUES (@Prompt, @GeneratedText, @DateGenerated)", connection))
{
command.Parameters.AddWithValue("@Prompt", prompt);
command.Parameters.AddWithValue("@GeneratedText", generated_text);
command.Parameters.AddWithValue("@DateGenerated", DateTime.UtcNow);

await command.ExecuteNonQueryAsync();
}
}

return new OkObjectResult("Data saved to the database.");
}
}

Copy and paste this code into your Azure Function App. Replace the your_sql_server_connection_string placeholder with your actual SQL Server connection string. This code assumes you have already set up an OpenAI API key as an environment variable within your Azure Function App.

  1. Save the function and test it to ensure it inserts the AI-generated content into the SQL Server table.

Step 3: Create a Power BI Paginated Report

  1. Open Power BI Report Builder.
  2. Create a new report or open an existing one.
  3. Add a new “Parameter” to the report:
    • In the “Report Data” pane, right-click “Parameters” and click “Add Parameter.”
    • Name the parameter “Prompt.”
    • Set the data type to “Text.”
    • Provide a default value or leave it blank.
  4. Add a “Textbox” to the report and set its value to the “Prompt” parameter: =Parameters!Prompt.Value

Step 4: Connect the Power BI Paginated Report to the SQL Server database

  1. In the “Report Data” pane, right-click “Data Sources” and click “Add Data Source.”
  2. Choose “Microsoft SQL Server” as the connection type and provide a name for the data source.
  3. In the “Connection string” field, enter your SQL Server connection string.
  4. Click “OK” to add the data source.
  5. In the “Report Data” pane, right-click “Datasets” and click “Add Dataset.”
  6. Choose the SQL Server data source you just created and click “Query Designer.”
  7. In the “Query Designer,” enter a SQL query to fetch the latest AI-generated content for the given prompt:
SELECT TOP 1 GeneratedText
FROM OpenAI_Responses
WHERE Prompt = @Prompt
ORDER BY DateGenerated DESC

8. Add the “Prompt” parameter to the query by clicking

“Add Parameter” in the “Query Designer.” 9. Close the “Query Designer” and click “OK” to add the dataset.

  1. Add a “Textbox” to the report and set its value to the AI-generated text: =First(Fields!GeneratedText.Value, "Dataset1")

Conclusion: You now have a Power BI Paginated Report that displays AI-generated content based on the prompt parameter. When the report is run, it will retrieve the latest AI-generated content for the given prompt from the SQL Server database and display it in the report. To update the AI-generated content in the SQL Server database, you can manually call the Azure Function with the specified prompt, or you can create a separate application to automate this process. The Azure Function will then call the OpenAI API, generate the text, and insert it into the SQL Server table.

This approach allows you to leverage the Power BI Paginated Report’s native support for SQL Server as a data source while still incorporating AI-generated content from the OpenAI API. It involves additional steps and requires an intermediary database, but it provides a viable solution for integrating OpenAI with Power BI Paginated Reports

This blogpost was created with help from ChatGPT Pro and Paginated Report Bear.

Leveraging OpenAI for Automated Data Storytelling in Power BI

Introduction

Automated data storytelling is a powerful way to transform complex data visualizations into meaningful narratives. By leveraging OpenAI’s natural language generation capabilities, you can create engaging and informative stories based on your Power BI data visualizations. In this blog post, we’ll discuss the importance of automated data storytelling and guide you through the process of using OpenAI to generate narratives and summaries for your Power BI reports.

Note – Please be aware that this solution involves interacting with OpenAI’s API. I encourage users to familiarize themselves with OpenAI’s data usage policy (https://platform.openai.com/docs/data-usage-policy) and take necessary precautions to ensure the privacy and security of their data.

  1. The Importance of Automated Data Storytelling

Data visualizations in Power BI enable users to analyze and gain insights from their data. However, interpreting these visualizations can be challenging, especially for users without a background in data analysis. Automated data storytelling bridges this gap by:

  • Making data insights accessible: Narratives help users understand the context and significance of the data, making insights more accessible to a broader audience.
  • Enhancing decision-making: Clear and concise narratives can help users grasp the implications of the data, leading to better-informed decisions.
  • Saving time and resources: Generating data stories automatically reduces the time and effort required to create manual reports and analyses.
  1. Prerequisites and Setup

Before we begin, you’ll need the following:

  • Power BI data visualizations: Ensure that you have a Power BI report or dashboard with data visualizations that you’d like to generate narratives for.
  • OpenAI API key: Sign up for an OpenAI API key if you haven’t already. You’ll use this to access OpenAI’s natural language generation capabilities. Visit https://beta.openai.com/signup/ to sign up.
  • A development environment: You can use any programming language and environment that supports HTTP requests. For this tutorial, we’ll use Python and the requests library.
  1. Accessing Power BI Data

In order to generate narratives based on your Power BI data visualizations, you’ll first need to extract the data from the visualizations. You can do this using the Power BI API. Follow the instructions in the “Accessing Power BI Data” section of our previous blog post on creating a Power BI chatbot to set up the necessary API access and create a Python function to query the Power BI API: https://christopherfinlan.com/?p=1921

  1. Generating Narratives with OpenAI

Once you have access to your Power BI data, you can use OpenAI’s API to generate narratives based on the data. Create a Python function to send data to the OpenAI API, as demonstrated in the “Building the Chatbot with OpenAI” section of our previous blog post: https://christopherfinlan.com/?p=1921

  1. Crafting Data Stories

To create data stories, you’ll need to design prompts for the OpenAI API that effectively convey the context and purpose of the data visualizations. The prompts should include relevant data points, visualization types, and any specific insights you’d like the narrative to highlight. Here’s an example of a prompt for a sales report:

openai_prompt = f"""
Create a narrative based on the following sales data visualization:

- Data: {sales_data}
- Visualization type: Bar chart
- Time period: Last 12 months
- Key insights: Top 3 products, monthly growth rate, and seasonal trends
"""

narrative = chat_with_openai(openai_prompt)

Remember to replace {sales_data} with the actual data you’ve extracted from your Power BI visualization.

  1. Integrating Narratives into Power BI Reports

With the generated narratives, you can enhance your Power BI reports by embedding the narratives as text boxes or tooltips. Although Power BI doesn’t currently support direct integration with OpenAI, you can use the following workaround:

  • Manually copy the generated narrative and paste it into a text box or tooltip within your Power BI report.

For a more automated approach, you can build a custom web application that combines both Power BI data visualizations and generated narratives. To achieve this, follow these steps:

  1. Embed Power BI visuals using Power BI Embedded: Power BI Embedded allows you to integrate Power BI visuals into custom web applications. Follow the official documentation to learn how to embed Power BI reports and dashboards in your web application: https://docs.microsoft.com/en-us/power-bi/developer/embedded/embedding
  2. Create a web application with a user interface: Design a user interface for your web application that displays Power BI visuals alongside the generated narratives. You can use HTML, CSS, and JavaScript to create the user interface.
  3. Fetch narratives using JavaScript: When a user interacts with your Power BI visuals or requests a narrative, use JavaScript to send a request to your OpenAI-powered Python backend. The backend should return the generated narrative, which can then be displayed in your web application.

Here’s a simple example using JavaScript to fetch a narrative from your Python backend:

async function getNarrative() {
    const response = await fetch('http://localhost:5000/narrative', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({ data: yourData }),
    });

    const responseData = await response.json();
    document.getElementById('narrative-container').innerText = responseData.narrative;
}

Remember to replace yourData with the data you’ve extracted from your Power BI visualization.

Conclusion

Automated data storytelling enhances the value of Power BI data visualizations by providing users with engaging narratives that help them better understand their data. By leveraging OpenAI’s natural language generation capabilities, you can automatically generate insightful narratives and summaries based on your Power BI visuals. Although direct integration with Power BI is not currently available, you can still utilize OpenAI-generated narratives in your reports or create custom web applications to combine Power BI visuals with automated storytelling.

This blogpost was created with help from ChatGPT Pro.