Microsoft Fabric notebooks are a versatile tool for developing Apache Spark jobs and machine learning experiments. They provide a web-based interactive surface for writing code with rich visualizations and Markdown text support.
In this blog post, we’ll walk through how to call the OpenAI API from a Microsoft Fabric notebook.
Preparing the Notebook
Start by creating a new notebook in Microsoft Fabric. Notebooks in Fabric consist of cells, which are individual blocks of code or text that can be run independently or as a group. You can add a new cell by hovering over the space between two cells and selecting ‘Code’ or ‘Markdown’.
Microsoft Fabric notebooks support four Apache Spark languages: PySpark (Python), Spark (Scala), Spark SQL, and SparkR. For this guide, we’ll use PySpark (Python) as the primary language.
You can specify the language for each cell using magic commands. For example, you can write a PySpark query using the %%pyspark magic command in a Scala notebook. But since our primary language is PySpark, we won’t need a magic command for Python cells.
Microsoft Fabric notebooks are integrated with the Monaco editor, which provides IDE-style IntelliSense for code editing, including syntax highlighting, error marking, and automatic code completions.
Calling the OpenAI API
To call the OpenAI API, we’ll first need to install the OpenAI Python client in our notebook. Add a new cell to your notebook and run the following command:
!pip install openai
Next, in a new cell, write the Python code to call the OpenAI API:
import openai
openai.api_key = 'your-api-key'
response = openai.Completion.create(
engine="text-davinci-002",
prompt="Translate the following English text to French: '{}'",
max_tokens=60
)
print(response.choices[0].text.strip())
Replace 'your-api-key' with your actual OpenAI API key. The prompt parameter is the text you want the model to generate from. The max_tokens parameter is the maximum length of the generated text.
You can run the code in a cell by hovering over the cell and selecting the ‘Run Cell’ button or bypressing Ctrl+Enter. You can also run all cells in sequence by selecting the ‘Run All’ button.
Wrapping Up
That’s it! You’ve now called the OpenAI API from a Microsoft Fabric notebook. You can use this method to leverage the powerful AI models of OpenAI in your data science and machine learning experiments.
Always remember that if a cell is running for a longer time than expected, or you wish to stop execution for any reason, you can select the ‘Cancel All’ button to cancel the running cells or cells waiting in the queue.
I hope this guide has been helpful. Happy coding!
Please note that OpenAI’s usage policies apply when using their API. Be sure to understand these policies before using the API in your projects. Also, keep in mind that OpenAI’s API is a paid service, so remember to manage your usage to control costs.
Finally, it’s essential to keep your API key secure. Do not share it publicly or commit it in your code repositories. If you suspect that your API key has been compromised, generate a new one through the OpenAI platform.
This blogpost was created with help from ChatGPT Pro
Christopher Finlan 