- 1 year ago
Welcome to this AI-900 lab session, where we explore how to evaluate the performance of generative AI models using Azure AI. With the rapid advancements in large language models (LLMs) and AI-driven applications, it’s crucial to measure and optimize their accuracy, efficiency, and reliability. This hands-on tutorial will guide you through various techniques to analyze, test, and improve generative AI models in Azure AI Foundry.
🔍 What You’ll Learn in This Video:
1️⃣ Key Metrics for Evaluating Generative AI Performance
2️⃣ Understanding Model Accuracy, Bias, and Responsiveness
3️⃣ Using Azure AI Foundry for AI Model Testing
4️⃣ Evaluating Text Quality, Coherence, and Relevance
5️⃣ Performance Benchmarking: Latency, Cost, and Scalability
6️⃣ Best Practices for Optimizing AI Model Outputs
🛠️ Who Is This For?
AI & ML Enthusiasts looking to optimize AI models
Developers & data scientists working with LLMs & generative AI
Professionals preparing for the Microsoft AI-900 Certification
Businesses seeking reliable AI solutions for real-world applications
📌 Key Highlights:
✅ Hands-on demo of AI performance evaluation techniques
✅ How to assess AI-generated content for quality & bias
✅ Using Azure AI tools for testing & optimizing generative AI
✅ Best practices for improving AI efficiency & cost-effectiveness
💡 Learn how to build safer AI applications with Azure AI Foundry today!
Explore Our other Courses and Additional Resources on: https://www.youtube.com/@skilltechclub
🔍 What You’ll Learn in This Video:
1️⃣ Key Metrics for Evaluating Generative AI Performance
2️⃣ Understanding Model Accuracy, Bias, and Responsiveness
3️⃣ Using Azure AI Foundry for AI Model Testing
4️⃣ Evaluating Text Quality, Coherence, and Relevance
5️⃣ Performance Benchmarking: Latency, Cost, and Scalability
6️⃣ Best Practices for Optimizing AI Model Outputs
🛠️ Who Is This For?
AI & ML Enthusiasts looking to optimize AI models
Developers & data scientists working with LLMs & generative AI
Professionals preparing for the Microsoft AI-900 Certification
Businesses seeking reliable AI solutions for real-world applications
📌 Key Highlights:
✅ Hands-on demo of AI performance evaluation techniques
✅ How to assess AI-generated content for quality & bias
✅ Using Azure AI tools for testing & optimizing generative AI
✅ Best practices for improving AI efficiency & cost-effectiveness
💡 Learn how to build safer AI applications with Azure AI Foundry today!
Explore Our other Courses and Additional Resources on: https://www.youtube.com/@skilltechclub
Category
🤖
TechTranscript
00:00Generative AI can help you to generate new contents with the help of the large language
00:16models well we all know this thing but then there is a question can we evaluate a performance of
00:22generative AI if yes then how exactly we can do that well that's what exactly we are going to see
00:28today my name is Maruti and I'm back with another dose of AI with Azure cloud today we are going to
00:35focus on generative AI performance and how we can evaluate that as usual we are again going to use
00:41Azure AI Foundry portal which is one of the newest service available in Azure AI cloud and let's get
00:47started so I'm in my Azure AI Foundry portal as usual the first thing which I'm going to do is
00:54I'm going to create a new project this new project is going to have some meaningful name so let's say
00:59I'm giving a name Maruti underscore Gen AI performance and then in the customize section the hub name
01:10is going to be also something like Maruti hub with some number I'm okay with all the other things the
01:17only thing which I want to change is I want to change the location to East US remember this location is
01:22very important for this particular project because when we are going to evaluate this performance
01:27somewhere in the last step of this particular lab we need to have some kind of a compute resources
01:32available for that evaluation now if you do not have enough quota available in a specific region
01:38then it can create an issue for you and you won't be able to do an automated evolution using that
01:43that's the reason I'm choosing East US because for me I believe East US is the one where I can have a
01:48maximum compute available I'm choosing this I'm going to click on next and I'll click on create
01:56okay the validation process is going on project creation is going to take few moments and while
02:00the project creation is going on I want to share some important things with you guys you already know
02:06my name I am a Microsoft certified trainer having 18 years of industry experience and I actually have an
02:13expertise on teaching Azure AI cloud to corporates okay my project creation is actually done it's just
02:20loading my project yes project is successfully created the first thing which I'm going to do is I'm going
02:26to click on my model catalog and I'm going to deploy my GPT-35 Turbo LLM
02:32yes I got my deployment model I'm going to click on deploy
02:51I'll choose customize and I want to make sure that we are going to reduce the tokens per minute to 5000
02:58if you are watching our videos regularly you know these are the common configuration steps which
03:02we have to do all the time I'll click on deploy and then within few seconds the GPT-35 Turbo model
03:08will be deployed successfully yes it is done
03:14with the rate limit of 5000 tokens per minute perfect now once this is done I'm going to open this in
03:21playground I want to make sure that this particular model is actually responding with the proper prompt so
03:28let me just provide some kind of a prompt like what can you do and if I send this prompt it's going to
03:36give me a response because it's an AI assistant which can help me with the information this is what they
03:42are actually trying to respond in this perfect now let me do the next thing I'm going to copy some model
03:50instructions here and then I'm going to paste it here in the system message box so I'm going to paste it
03:55here this is actually having some kind of an objective which is mentioned for this particular
04:00chatbot so I'm mentioning that you're going to assist users with travel related inquiries offering tips
04:06advices and recommendations as a knowledgeable travel agent also what kind of capabilities I want to
04:13include in this and what kind of instructions I want it to follow is all which we have mentioned here
04:20now for your kind information this is one of the official lab which is available on my maths learn
04:25you will find the link of this particular lab in the description of this particular video so please
04:29make sure you check out that and from that only you can copy paste these instructions that's what I
04:34have done I'm happy with this I'm going to click on apply changes and continue now when we do this the
04:40system message is going to control the chatbot properly so now this is applied successfully let me just
04:46ask the same question once again what can you do if I ask this time the answer is going to be different
04:54because it's having this additional system configuration message here is going to show
04:59me that I can assist you with all the travel related inquiries whatever you need destination
05:04recommendations travel tips accommodation suggestions this is actually changed now because of my system
05:10message which I have given which is perfect now once this is done I know that this is going to give
05:14me a travel related information but I do not want to use a chatbot instead of that I want to evaluate
05:21the performance of my generative AI model which is GPT-35 turbo which we have selected right now
05:27how can we do this well for that you have to go into this left side column where we have a section
05:33called evaluation in this evaluation you will find two different kind of evaluations you can do manual
05:39evolution or you can do an automated evolution also now as of now it's taking me to the page which is
05:46showing me that you can assess and compare AI application performance which is under automated
05:52evolution and then we have manual evolution also which is available here now I am going to specify a
06:00manual evolution first so let's check how manually we can evaluate our model I'm going to click on new manual evolution
06:09in the assistant setup in the system message I am going to change the system message in spite of this
06:15I'm going to provide the information which I have shared in that chatbot like this model is GPT-35 turbo
06:23max response temperature is all fine right now I do not want to change it here if you want to add
06:28your own data you can do that also while evaluating a model but we are not doing it right now and then in
06:34the below section we have manual evaluation result in which we have add input kind of a button now what
06:41I want to do is I want to provide a separate separate multiple inputs and then I want to also mention some
06:47kind of an expected output which are coming with that so multiple inputs which we will provide we'll see
06:52that what kind of expected responses we are getting because of that let's see that but let me add multiple
06:58input first so I'm going to add one two three four and five input boxes here and then in each one I'm
07:05going to provide an input also just to note this all the inputs are available in the same lab document
07:11so please make sure you follow the lab document in which you will find all the step-by-step instructions
07:18I have provided four inputs and now I'm going to do the last one which is a fifth one now obviously all
07:24these five different inputs are actually asking separate separate questions which are not related
07:28to each other like the first one is asking can you provide a list of the top rated budget hotels in
07:34Rome while in the second one we are asking similar kind of restaurant related questions in New York City
07:40and then same way in the coming questions we are asking for vacation for a family in Florida or a
07:45honeymoon trip to Maldives and then we are also asking for a guided tour for a Great Wall of China so
07:52these are five separate questions which I want to put into this particular model evaluation and then
07:58we'll see that what kind of evaluation result we are getting with this so once you are done with the
08:02inputs all you have to do is you have to click on this run button and wait now within few seconds you can
08:09see it's done it's showing me right now running evolution is in progress one of the five input is
08:14processed so far so basically this is actually started and it's just doing one out of five we have to wait
08:20till all five are done second one is also done let's wait for some more time
08:26okay sometimes this happens which is a common issue it's showing you the run resulted in one
08:31failure see the details in each row so basically in the first second and third it got executed but
08:37somewhere in the fifth one when I'm asking for a guided tool available for the Great Wall of China
08:41they are saying that this request to chat completion created operation under this is actually exceeded the
08:47token rate limit for this so maybe because of the response which is there it's not actually working
08:52with that so I can say rerun all field this is a very common cause because of the token limit because
08:58we have not used a maximum token limit we have used limited in that so I think I did I did rerun and then
09:05once I did rerun I think all five are actually now done now let's focus on the responses now because
09:10we are doing manual evaluation we need to check each response and then we need to make sure which one is
09:15appropriate as per us so when I was asking for hotels in Rome I think they have given me a five hotels
09:22highlighted with the proper details about those hotels so I think this is a proper response I like
09:28it so I'm just going to give a thumbs up the like inside that same way somewhere we have a New York City
09:35is a great place they are giving me something which is vegan friendly restaurant so they are giving me
09:40some of the vegan friendly restaurant names like oven garden is there and then also this another one
09:47which is by Chloe which is there so that is fine these are the things which they are giving me but I
09:52I'm still not happy with maybe this response I'm going to give thumbs down for this and then maybe the
09:57third one which is saying that a seven day itinerary for a family vacation in Orlando they have given me
10:05day wise itinerary where I'm going to go into arrival then Walt Disney World then animal kingdom
10:11then universal Orlando resort wow this is something which looks perfect I am happy with all the seven
10:16days kind of a segregation which they have done for the itinerary so I'm going to give a thumbs up
10:22by the way if you're also liking our videos you can give thumbs up to us also and that's going to
10:28encourage us so that will be great if you give thumbs up to us also let's move forward so the other two
10:34responses are also there I do not want to give thumbs up or down in this manual evolution we can see
10:39right now that the data which we have available here for the 40 percent we have given thumbs up
10:44for maybe 20 percent we have given thumbs down and the remaining data is still not we have not given
10:49thumbs up or downs maybe something which is average now once we are done with this now once we are done
10:57with this we will click on save results will give some kind of a meaningful name to this so let's say
11:02i'm giving a name manual evaluation results and i'm going to click on save and this is your manual
11:10evaluation which you have done first results are saved successfully which is great now i can go back
11:16to my evaluation tab and if i go into this this is going to show me manual evaluations and inside that
11:25my result which i have saved is going to be visible now this particular result i can check any point of
11:30times i can click on this i can go inside that and i can also check if i have some previously created
11:36evaluation results are saved inside that so this is actually showing me all this configuration here
11:42now let me go back to evaluation and show you the another way by which you can evaluate your model
11:47that is known as automated evaluations now before i do this let me show you third thing which is here
11:53there is something called evaluator library evaluator library is an asset that can be used to run
11:59evaluations you can define evaluation in sdk and you can run evaluation to generate scores of one or more
12:07metrics so basically this is going to be the one if you want to associate with the sdk then this evaluator
12:13library is actually going to help you in that now i do not want to create a new evaluator right now
12:18we will first try automated evaluations right now so let me click on automated evolutions there is option
12:26create a new evaluation now before i create a new evaluation let me tell you in this evaluation we
12:32will be able to evaluate with the help of our own data so you need to take your some kind of a data also
12:38based on which that is going to be treated as a data set and on that you can actually evaluate your model
12:44in my case i'm going to download that file first so let me just open the new tab in the browser
12:50i'll hit the url which is this and in this case this is actually my very simple data set which is
12:55helping me to ask some questions related to that i'm going to save this i'm going to save this in my
13:02download folder somewhere with the name travel qa json l just make sure that you are saving this thing
13:09with the json l kind of an extension and you click on save it's showing me the file name is not valid
13:18okay i need to change the type so let me just change back to a specific folder i'm changing this to all
13:25files and then in the all files i'm going to change it and save it to travel qa json l and i'm going to
13:30click on save okay so once we have downloaded our file which is travel qa dot json l we are going back
13:36to our azure ai foundry portal we'll create a new evaluation which is automated evolution and this
13:43is first asking you what do you want to evaluate do you want to evaluate model and prompt do you have
13:49a data set so you want to evaluate an existing output data set or you want to associate with the prompt flow
13:55and you want to evaluate prompt flow i'm going to choose that i'm going to select data set right now
14:00in the data set the evolution name can be anything i think this is perfectly fine a description can be
14:08anything i do not want to change anything in the first one i'm going to click on next and then in
14:13this section we are going to specify that we want to use an existing data set okay in this we are going
14:19to specify that we want to add our own data set by uploading a file so we are going to select add your
14:25data set we'll upload a file we will go to that specific location where we have travel qa json l file
14:34and then we'll upload that file make sure when you upload this json l file this is always going to take
14:40files in two different formats json l as well as csv so make sure you're just going to have any of
14:45these two formats which are supported here i have my file uploaded here successfully i'll click on next
14:52and now it is asking me what kind of metrics based on which you want to evaluate this model now these
14:58are all the quality metrics which are associated here some of the quality metrics are focusing on
15:02ai assisted some of the quality metrics are focusing on natural language processing so we have different
15:08kind of quality metrics i am going to select right now two metrics here which is coherence and fluency
15:15remember coherence measures how well the generative ai application can produce output that flows smoothly
15:23it's going to read naturally and it's going to resemble human-like language so i want to make sure it is
15:29actually evaluating based on coherence or not as well as we want to focus on fluency also which measures
15:36the language proficiency of a generative ai applications predicted answer so these are the two
15:42things based on which we will measure this and this is why i'm selecting these two metrics you can
15:47select other also what kind of a connection will be used for this evaluation i'm choosing my azure ai
15:53service and the only model which is deployed inside this is gpt35 turbo which is coming here once we are
16:00done with this we will click on next and then in the review and finish i am going to click on submit this is an
16:10automated evaluation so remember this can take some time and when you are submitting this evaluation
16:15make sure that it is actually going to successfully start the session it's going to configure some kind
16:21of a compute which is going to do this evaluation for that and then once it is successfully done then
16:27only you'll be able to see the result of that there are many things which they are going to check in the
16:32background so just make sure when you click on the submit you do not refresh this page and you just
16:37wait here on this page because this is going to create a separate cluster instance in the background
16:42you can see right now the status is where i have to focus it's showing me the status is not started
16:48i have to wait for some time after a few minutes is maybe going to show me that it is running and then
16:54after that it will be completed i have to wait till then remember the basic need for this evaluation to
17:00work successfully is you need to have enough quota available for your cpu usage in that particular
17:07region if your subscription is not having enough quota or maybe you have already used the quota then
17:13you won't be able to do this kind of an automated evaluation i'll wait for some time and then i'll just
17:19fast forward this video right now once this is done i'll show you what kind of result is coming also
17:24let's see for me if i have enough quota in east us region then this will work otherwise it won't work
17:30okay so let's just wait i'm keeping my fingers crossed let's hope that is going to work fine
17:35okay i have completed this evaluation and if you see the evaluation detail where they're showing me the
17:41status is failed and they're also showing me that we encountered a problem during your evaluation
17:47please review the raw json or visit the logs for that now this is a raw json which is available here
17:54if i go into that it's going to show me the details which are visible and somewhere inside this they are
18:00going to show me that log files path which are available inside this now why exactly this is failed
18:06i want to figure that out and that's the reason instead of this raw json i'm going into my logs
18:14if i go into execution logs there's nothing from flow automatic dot log
18:20it's showing me that okay they have run this thing for a couple of minutes they have configured
18:26some kind of a prompt flow runtime with the docker powershell commands it got executed with that and
18:32they created the containers also and somewhere after this they are saying app inside key is not set
18:38and then they are starting the ingress configuration with this now when we try to focus on this this
18:44whole configuration got executed actually so there's nothing wrong with the prompt flow automatic
18:48configuration in the prompt flow service also when we go and check it's showing me that from all this
18:55time it was actually running and then somewhere in this background they are saying the automated
19:00runtime is not ready in a fixed time stamp so error is container not started somewhere the issue is
19:07because of the container which is not started and as i said earlier container which is not started can
19:12be because of the issues which happen because of that region you can see right now the prompt flow service
19:18is showing me the status code 503 because of some internal configuration issues in that region
19:24it's not actually going through now this is a common issue which can happen because of the limited quota
19:30or limited privileges associated with that region but yeah this is how you can check you can understand
19:35that what exactly happening in that particular evaluation uh i am not going further in this
19:41right now so this is something which is the case if you are facing this issue in one region i suggest
19:46you to try with the different region instead of east us you try with the other one where maybe you're
19:51going to have enough quota and services sources available so you'll be able to do that thing i hope you
19:56understood how we can evaluate generative ai models automatically and manually automatic is little time
20:04consuming and cost expensive while manual is easy and straightforward so i hope you'll understand
20:09both and you'll try both thank you so much happy learning i'll see you tomorrow
Comments