- 2 days ago
Category
🤖
TechTranscript
00:00In this video, we're going to explore how we can use our own functions and modules in Microsoft Fabric.
00:10So most often, if you come from a company environment, then you probably have a lot of functions and code written already.
00:17And this code you want to use now in Fabric. So how can you do this?
00:20Now, the easiest and best option probably would be, if you have a notebook in here, to import it as a resource.
00:27What I mean by that is, currently, we are here in the lake house.
00:30And if we go back to all sources, there's an option resources.
00:34And there we can upload, in this case, modules and functions.
00:37Now, the one I'm going to use now, and which you can also use if you want, because it's available in the resource section, is my grade function.
00:44And I'm going to show you how it looks like. It's this one.
00:47So basically, the hotdog function. And the hotdog function simply checks a string.
00:51And if the string is hotdog, it returns, this is a hotdog.
00:54And if it's not a string, it's a hotdog, then it returns no hotdog, right?
00:58So that's basically the idea behind that.
01:00And if you have watched the famous sitcom Silicon Valley, you know this function probably.
01:04Otherwise, ignore it for now. But this is where I actually copied it from.
01:08So to get this function into Fabric and then use it there, let me just minimize that, we need to upload it first.
01:15So that just means inside this built-in option in this resource folder, we can click the three dots.
01:20And then we can upload files. Click on this option.
01:23Then here you get a little information that must be small at 100 megabytes.
01:27So keep this in mind. But click on next.
01:30And then I just choose here my custom function. So it's my hotdog function.
01:33I click on open. And now it's already uploaded.
01:36You can see it here. It's small, so it shouldn't take too long.
01:39Now, how can I use it now, here, this module in my notebook?
01:43So the idea is, you can, of course, easily, best way, I would say, is actually to drag and drop it inside.
01:49You can see, you can drag and drop it in a new cell, drop it there.
01:52And now you can see, this is how it actually, you need to write it.
01:56So import built-in.custom as custom.
01:59So you reference the built-in folder here, then .custom, that's the file.
02:03And then you can just give it an abbreviation or a short name.
02:07So, for instance, also c, if I want to do that.
02:09But in this case, it's like that, right?
02:11And we can execute this just to check it.
02:13And then we should see that direction should actually give us some information.
02:16And there you can see, there, beside these ones, we have here the hotdog function.
02:20So what we can do now is we can check it.
02:22So simply, the results, the result is equal to, and now, custom and .hotdog.
02:28Hotdog is a function.
02:29And now we just put in some kind of string here.
02:32So let's say, in this case, banana.
02:35Okay.
02:36And then just print the results.
02:38So in this case, print result, like that.
02:41And of course, if we run this, then we get no hotdog.
02:43So you see that the function works.
02:45And that's basically how we actually can get the modules from our company or what we have created inside Fabric.
02:51So kind of easy.
02:52And all you need to do is then either you drag and drop it or you can see here now the code.
02:55You just import built-in dot and then your package name.
02:58And then you just give an abbreviation as you probably are familiar with in Python and then you can use it in here.
03:03So that's basically it for that.
03:05So hopefully that was clear.
03:06If you've got questions related to that, feel free to ask.
03:09Otherwise, I'll see you not in the next video because I have another interesting thing for you.
03:14And this is the concept of environments.
03:16Now, what is an environment?
03:18Well, you might already have spotted.
03:20At the top here, we have the environment and this is currently workspace default.
03:25Meaning, here we can see that this is workspace settings.
03:29It's the default runtime and so on.
03:31But this is the default environment.
03:33What does this mean, default environment?
03:34Well, default environment means this is the environment which is automatically used when we create notebooks in this specific workspace.
03:43And that just means that currently the environment is kind of plain.
03:46Meaning, whenever we want to use a new module, we would have to install it.
03:50As an example, if I want to use the YFinance package, I cannot simply write import YFinance S and then YF to get some kind of stock news and stock related prices.
04:01If I try to run this like that, you see that I run into an error because there is no module named YFinance.
04:07This just means in order to use it, I would have to install it.
04:10Meaning, for instance, I could write, in this case, exclamation mark and then PIP install and then YFinance like that.
04:18And now if I press enter, like this, then the module would be installed and then I could use the YFinance package.
04:24So, you can see here, I should have the quiet flag.
04:27If you type in "-q", you should not see this collection, all these things here.
04:31But for now, that's totally fine.
04:33And then wait until this is done.
04:36So, just a second.
04:37And then we can use this package.
04:39So, after we install it in the notebook, it is available for us and then we can use it.
04:43So, now we have installed it.
04:45So, let me maybe just for now, just use the hash symbol to comment this out.
04:52Now we can use it.
04:53So, if I say now import YFinance S, YF, abbreviation, and I run this, you see that now I don't get any issues anymore.
05:01Because now we have installed it inside the notebook, inside this current session.
05:05Meaning, if we just check it, for instance, in this case, let's say the ticker is equal to Microsoft, which is MSFT, the stock ticker.
05:14And then we say, in this case, ticker equals YF dot ticker.
05:20And then the ticker is the ticker.
05:22So, actually we should maybe use a different name here.
05:24So, let's say TK.
05:26TK is a ticker.
05:27And then we want to get the information for the ticker.
05:30And then we just say a print in this case.
05:33And let's say ticker dot info dot get.
05:37And let's say we would like to have the current price, current price.
05:41That's actually the results.
05:43And maybe normally we should actually store this in a verb as well.
05:46So, let's do that.
05:47So, let's copy it out.
05:48And let's say here the result.
05:51Result is equal to you.
05:53So, it's equal to you.
05:54And now this.
05:55And now let's actually print this.
05:56So, print in result.
05:57Like that.
05:58And if we run this, we should actually get the stock price.
06:01You can see here.
06:02408 dollars and 43 cents currently.
06:04So, basically you can see that the package itself works.
06:07This is great.
06:08So, we can of course import all the packages we want into our current session.
06:11And we can use them inside our notebook.
06:13But this is actually not that good.
06:15Because think about it.
06:17If you work in a company, then of course all your colleagues and so on may also want to use this package.
06:21And everyone has to manually install it in his or her workbook session.
06:26Also, what if the session closes?
06:29So, if I go on this button and stop the session, then the next time I start a new session, again I would have to install it.
06:36Because Fabric by default doesn't remember that we have installed this specific package in this environment.
06:42We have to reinstall it again and again.
06:44And this is something we like to avoid.
06:46And for that, there we have the concept of environments in Microsoft Fabric.
06:51Environments allow us to install, for instance, specific packages.
06:54And then those packages will always be available when we start a new notebook session.
06:59So, we just then have the specific environment set up which we need and also our colleagues need to do their jobs.
07:07So, hopefully it is clear why we want to have environments.
07:11And now let's see how we can create them.
07:13So, for that, let's go back inside the workspace we're currently in.
07:17Which is, in this case, the data render itself is inside the Fabric Trial.
07:20So, go to Fabric Trial.
07:22And then we can create a new environment specifically for this workspace.
07:25Which just means that when we create the new notebooks, we can reference this specific environment.
07:31And an environment itself is also a Fabric item.
07:34Meaning, we should go to the new item section here.
07:37And then search for the filter here.
07:39ENV.
07:40So, ENV.
07:41And there are the environments.
07:43Right?
07:44Under Analyst and Trained Data.
07:45So, for the data scientists as well as Develop for Data Engineers.
07:48So, let's go to one of them.
07:50It doesn't matter which one you click.
07:51Go to Environment.
07:52And then we just wait until this item gets created.
07:54So, let's say this is my ENV.
07:58Of course, you can name it properly in your organization.
08:01Click on Create.
08:02And then we just wait until this environment gets created.
08:04And there it is.
08:06And then we can see there are various libraries.
08:08There are built-in libraries, which are these ones here.
08:10But there are also additional ones like public libraries.
08:13They can add libraries from PyPy.
08:16For instance, let's use our example.
08:18Right?
08:19So, add from PyPy.
08:20And what I want to add is the Y Finance.
08:22You can see that if you type in, you already find it in here.
08:25Click this option.
08:26And then you can also choose different kinds of versions which are available.
08:29I'm choosing now this one here.
08:30But you can also choose another one.
08:32So, and this is just one library.
08:34But of course, you can add several libraries here.
08:36Right?
08:37You can add additional libraries here.
08:38But these are the public libraries which you can install.
08:40So, other options.
08:41You also find them up here.
08:42So, add from PyPy.
08:43You can click this option.
08:44Then you can add a new library.
08:45You can also remove this.
08:46You can also click down there.
08:47And then add from Conda.
08:48This is also available.
08:49Or add from my YAML file.
08:50So, if you have a YAML file, you can also add the library here.
08:55So, these are the public libraries.
08:57Right?
08:58The built-ins we already have seen.
08:59So, these exist for Java and Scala, for Python and for R.
09:03Because remember, we have different kinds of coding syntaxes we can use in these notebooks.
09:08So, depending on what we need, we can actually add them here in public libraries as well.
09:13But you can also clearly see that there are already a lot of installed by default.
09:16So, that's great.
09:17But as I said, the MyFinance was just an example.
09:20But oftentimes, you also want to use additional libraries.
09:23Or you want to use custom libraries.
09:25So, we have seen that because we can of course import, as we've seen, with our hotdot function,
09:31this into one notebook.
09:32But again, this is only available for this specific notebook.
09:35And not for all the other colleagues of us.
09:37So, that's why we can also upload here custom libraries.
09:40So, you go to Upload here.
09:41And then you can choose what is custom.py.
09:43I click on Open.
09:44I click on Open.
09:45And now, this would mean that now the hotdog function would be available for everyone.
09:49So, also, this is possible here.
09:51Then there are also Spark compute settings which you can set for the environment.
09:54So, by default, you have here the option to acceleration.
09:57You have the option here to choose a different size of compute if you want to do that for the environment itself.
10:02But I leave the default settings now.
10:04This is just something you can also enable here as well.
10:07And as soon as you're done, what you need to do is you need to save the environment.
10:11If I go to save here.
10:12Yes, I like to save the changes.
10:14And then, of course, you basically now have saved it.
10:17And now I want to publish it.
10:19So, yes, I want to publish all.
10:21And yes, I would like to publish all.
10:23And that's basically what we need to do.
10:25And this just means that now this environment is created.
10:28You can see it's in progress.
10:30It's publishing.
10:31It's progress.
10:32And as soon as it's done, we should see a new item in our workspace in here.
10:37So, let's go back to the workspace already, fabric trial.
10:40And there should be then a new item which is called my ELE.
10:43So, my environment or a different name if you gave it.
10:46And this then they are available.
10:48And this just means as soon as the environment is available in here.
10:51This means that we can go back to our notebooks in the data wrangler section here.
10:55And then under the environment here, up here, instead of using workspace default,
11:00we can then choose, for instance, you can already see it here, my environment.
11:03So, I can choose a different environment here when I want to run this notebook.
11:06And finally, if you want to configure it for your workspace itself,
11:09what you need to do is you go back to the workspace.
11:13You need to go in the workspace settings, click here.
11:16And then under the data engineering, data science part,
11:19and click on the spark settings here.
11:21And there you can also find, up here, environment.
11:24And there you can specify, in this case, set default environment.
11:27You can take this option and then you can choose, instead of using workspace default,
11:31you can choose my environment.
11:33And if you take this option, then each notebook created in this specific workspace
11:37will always start, if you run it, with your own environment configuration.
11:41And that just means that all the libraries, which you have installed in this environment,
11:44will be available by default.
11:46And that's basically how you would handle this in a big company.
11:49So, you ensure that the colleagues, all the data engineers, data scientists, and so on,
11:54always have their specific libraries available.
11:56Also, your own custom ones developed in your organization,
11:59when they start actually a notebook in Microsoft Fabric.
12:01So, that's it for this video.
12:03Hopefully, you enjoyed it.
12:04Hopefully, that was helpful.
12:05And if you got questions, let me know.
12:07Otherwise, thanks for watching, and hopefully see you in the next video.
12:10Until then, best guys.
Recommended
13:32
|
Up next
1:16:45
0:33
12:08
12:23
0:36
24:07
0:46
Be the first to comment