Prompt Engineering and Microsoft AI Services

AI-103: Developing AI Apps and Agents on Azure

1. Generative AI vs Traditional AI

Before diving into prompt engineering, it helps to understand how generative AI differs from the traditional AI workloads that have been used in industry for years, well before tools like ChatGPT existed.

Analyzing Data vs Creating Content

The core distinction comes down to purpose:

Traditional AI Use Cases

Traditional AI has been solving well-defined problems for many years, including:

• Fraud detection - widely used in e-commerce and banking to flag suspicious transactions
• Spam filtering - email security solutions have used AI-based filtering for decades
• Image recognition - classifying or identifying objects within images. Example identifying cats in photos
• Product recommendations - suggesting products to users based on their behavior and preferences. Example: recommending books on an e-commerce site

None of these use cases require generative capabilities - they rely on models that analyze input and classify or score it.

Generative AI Use Cases

Generative AI, in contrast, focuses on producing new output, such as:

• Generating new text (writing assistance, summarization)
• Generating new images
• Generating new code

These are representative examples rather than an exhaustive list.

How Generative AI Creates New Content

Generative AI applications, and the large language models behind them, are trained on data sources. These sources can come from the public internet or from a private/organizational environment.

Once trained, the model can generate new content in response to a prompt - an instruction submitted by a user.

For example, a user could submit the prompt:

The generative AI application would then return the requested Python code as its output. This same prompt-driven interaction pattern is the foundation for the prompt engineering concepts covered next.

2. How Large Language Models Work

Before looking at prompt engineering in detail, it's worth understanding what a large language model (LLM) is actually doing when it responds to you.

Predicting the Next Token

An LLM does not "know" facts the way a person does. Instead, it works by repeatedly predicting the most likely next token (a word or part of a word), based on everything that came before it.

Each time the model generates a token, it calculates a probability for many possible next tokens, and picks (or samples from) the most likely one. It then repeats this process, using the growing sequence as new context, until it decides the response is complete.

A Simplified Example

Suppose a user asks a generative AI application:

Internally, the model doesn't look up an actual weather report. Instead, it estimates probabilities for candidate next words, something like:

Candidate word	Estimated probability
The	24%
It	20%
Today	17%
Currently	8%
Weather	5%

(These numbers are illustrative only - actual probabilities depend on the model and context.)

The model selects the highest-probability option, then repeats the process for the next token, and the next, until it produces a full response, for example:

The model has no awareness of the actual conditions outside. It has simply calculated that, given the prompt, this sequence of tokens is statistically the most likely continuation.

From Prompt to Response: A Code Example

The same mechanism applies to more complex prompts. Consider:

To respond, the model draws on patterns learned during training from large amounts of public data, such as:

• Official documentation (for example, Python's own docs)
• Public code repositories
• Community Q&A and discussion sites
• Other publicly available text

The prompt itself is processed by the model's underlying neural network architecture, which identifies relevant concepts within it, in this simplified example, something like "Python," "function," and "even number." These concepts shape the context the model uses when predicting tokens.

The model then generates the function token by token, in the same probability-driven way as the earlier example, resulting in working Python code returned as the response.

Key Takeaway

This is an important limitation to keep in mind: because responses are probability-based rather than fact-based, LLMs can generate plausible-sounding but incorrect output, commonly referred to as a hallucination. This is also part of why prompt engineering matters: giving the model clearer, more specific context narrows down the probability space and makes an accurate, relevant response more likely.

Generative AI applications are only as effective as the instructions we give them.

Whether you're asking an AI model to write code, summarize information, analyze data, or perform a task, the way you formulate your instructions can significantly affect the quality of the result.

This is where prompt engineering becomes important.

In this lesson, we'll cover two important topics:

• Prompt engineering
• Microsoft's AI ecosystem and the services available to AI developers

3. What Is a Prompt?

A prompt is the input or instruction that you provide to a generative AI application or large language model (LLM).

For example:

The model receives this instruction and generates an appropriate response.

The prompt is therefore one of the primary ways humans interact with generative AI systems.

4. What Is Prompt Engineering?

From here on we'll use a single running example: a hospital that wants an AI assistant to help with patient consultations.

Simple prompts can work well for simple tasks. For example:

There isn't much ambiguity here. But consider a more complex request:

What exactly should the model do? Should it:

• Summarize for the specialist, or for the patient?
• Include the full medication history?
• Cover every past visit, or only today's?
• Suggest a diagnosis?
• Recommend treatment?
• Flag abnormal readings?
• Return a paragraph, a letter, or a bulleted list?

The instruction is ambiguous. This is where prompt engineering becomes important.

Prompt engineering

Prompt engineering is the process of designing and refining instructions so that a generative AI model produces more useful, accurate, relevant, and consistent results.

In simple terms:

5. Why Is Prompt Engineering Important?

A well-designed prompt can improve the quality of an AI application's output. There are four important benefits.

Reduce Hallucinations

A vague prompt can leave the model with too much room to make assumptions.

Providing more specific instructions and relevant context can reduce the likelihood of incorrect or fabricated information.

Prompt engineering does not guarantee that a model will never hallucinate, but it can help reduce ambiguity and improve grounding when appropriate information is supplied.

Improve Accuracy and Relevance

A good prompt clearly explains what you want. Instead of:

you could say:

The second prompt gives the model much more direction - it narrows the audience, the scope, and the expected shape of the answer.

Save Time and Cost

If your first prompt produces a poor result, you may need several additional prompts to correct it. A well-designed prompt can reduce this back-and-forth.

For applications that make many model calls, fewer unnecessary interactions can also reduce token usage and therefore potentially reduce cost.

Improve Consistency

A clearly defined prompt gives the model explicit instructions about how it should respond. This can make outputs more predictable and consistent across repeated requests.

This becomes particularly important when integrating an LLM into an application or agent rather than simply using it interactively.

6. Components of a Strong Prompt

There are many prompt-engineering frameworks. However, the core principles can be summarized into five components:

1. Role
2. Task
3. Context
4. Input Data
5. Output Format

Let's examine each one, building up the physician prompt component by component.

7. Role

The role tells the model what perspective or expertise it should use when responding. For example:

This gives the model additional context for how it should approach the task. Compare that with:

The expected depth, vocabulary, and confidence of the answer could be very different - even though the question that follows may be identical.

You are an experienced general physician.

The role establishes the perspective from which the model should approach the rest of the request.

8. Task

The task describes what you want the model to accomplish. For example:

Now the model knows what it is supposed to do. A strong prompt should make the task as unambiguous as possible - note how much narrower this is than "Summarize this patient record."

You are an experienced general physician.

Summarize the patient's visit and list the recommended next steps.

The first sentence establishes the role. The second defines the task.

9. Context

Context provides additional information that affects how the model should solve the task. For example:

This is very different from:

The model now knows who the reader is. For a patient-facing summary, you might expect the response to consider things such as:

• Plain, non-technical language
• A reassuring tone
• No unexplained abbreviations
• Clearly stated next steps

Context can also set a boundary:

In regulated domains such as healthcare, finance, or law, that boundary is often the most important part of the whole prompt.

The more relevant context you provide, the less the model has to guess.

10. Input Data

The next component is the input data that the model should work with. For example, you might provide the consultation notes:

BP 150/95, complains of headaches for 2 weeks, no chest pain,
advised to reduce salt intake, review in 4 weeks.

But don't simply paste the data and expect the model to infer everything. Explain what the data represents. For example:

This gives the model both the data and the meaning of the data. Without that explanation, the model has to guess whether this is a current visit, a historical record, or just an example.

11. Output Format

Finally, specify exactly how you want the response returned. For example:

This is better than leaving the output format ambiguous. Without an explicit instruction, the model might return:

• A single dense paragraph
• A formal clinical letter
• A table
• A bulleted list only
• A mixture of all four

If your application needs machine-readable output - for example, to store the summary in a patient portal - specifying the format becomes even more important.

12. Putting the Components Together

We can combine everything into one prompt.

Component	Prompt text
Role	You are an experienced general physician.
Task	Summarize the patient's visit and list the recommended next steps.
Context	The summary will be read by the patient, not by medical staff, so avoid technical jargon. Do not provide a diagnosis.
Input Data	The notes below are the doctor's observations recorded during today's consultation.
Output Format	Return a short paragraph followed by a bulleted list of next steps.

Together:

You are an experienced general physician.

Summarize the patient's visit and list the recommended next steps.

The summary will be read by the patient, not by medical staff,
so avoid technical jargon. Do not provide a diagnosis.

The notes below are the doctor's observations recorded during
today's consultation:

  BP 150/95, complains of headaches for 2 weeks, no chest pain,
  advised to reduce salt intake, review in 4 weeks.

Return a short paragraph followed by a bulleted list of next steps.

This is much more precise than simply saying "Summarize this patient record."

13. Bad Prompt vs Better Prompt

Let's consider another example.

Bad Prompt

This prompt does provide a task, but many important details are missing. For example:

• What kind of note - clinical, referral, or patient-facing?
• Who is going to read it?
• Which visit should it cover?
• Is the model allowed to interpret the findings?
• How long should it be?
• Should it include a follow-up date?

The model has to make assumptions.

Better Prompt

This prompt is much more precise. The model now knows:

Element	Value
Role	General physician
Task	Write a follow-up note
Reader	The patient
Input	Today's consultation notes only
Constraint	No diagnosis
Tone	Plain language
Output	Note ending with the next review date

The fix in both cases is identical: replace what the model would otherwise have to assume with something you state explicitly.

14. Prompt Engineering in AI Applications

Prompt engineering becomes even more important when you move from manually chatting with an AI model to building an AI application.

Consider a customer-support application. A weak instruction might be:

A better system instruction could define:

• The assistant's role
• What information it can use
• What it should do when information is unavailable
• The tone of the response
• What actions it is allowed to perform
• The required response format

For AI agents, prompts and instructions can also define the agent's goals, constraints, and behavior. Microsoft Foundry's current agent architecture explicitly treats instructions as one of the core components of an agent, alongside the model and tools.

15. Microsoft AI Services

Microsoft's AI ecosystem contains several different offerings. This can initially be confusing because Microsoft provides AI capabilities through different products and platforms.

A useful way to think about the ecosystem is:

These products serve different types of users and development scenarios.

16. Microsoft Copilot

Microsoft Copilot refers to Microsoft's family of AI-powered assistants and experiences. It is not simply one product.

For example, Microsoft 365 Copilot helps users with work tasks inside Microsoft 365 and can use information from Microsoft 365 applications and other permitted work content.

Azure Copilot

Azure Copilot is designed specifically to help users work with Azure. It can help users:

• Learn about Azure
• Navigate Azure services
• Understand Azure resources
• Troubleshoot resources
• Gain insights into their Azure environment

Azure Copilot uses LLMs together with Azure control-plane information and knowledge about the user's Azure environment.

The key idea is:

17. Copilot Studio

Microsoft Copilot Studio is a graphical, low-code platform for building agents and agent flows.

This makes it different from simply using an existing Copilot. Instead of only consuming an AI assistant, you can build your own agent experience.

Microsoft describes Copilot Studio as a low-code tool for building agents and agent flows.

Copilot Studio Agents

An agent can:

• Interact with users
• Use instructions and context
• Access connected data
• Perform actions
• Handle complex conversations

You can also connect prebuilt or custom connectors to external data and services.

18. Copilot Studio Agent Flows

Copilot Studio also provides agent flows. Agent flows are useful for deterministic, step-by-step automation.

Microsoft describes agent flows as deterministic and capable of being triggered manually, by events, by agents, or on a schedule.

This makes them useful when you need predictable business processes rather than leaving every step to an AI model.

19. Microsoft Foundry

For developers and AI engineers building custom AI applications, Microsoft Foundry is a central platform for working with models, agents, tools, and related AI capabilities.

Microsoft currently describes Foundry as a platform that unifies agents, models, and tools with enterprise capabilities such as tracing, monitoring, evaluations, RBAC, networking, and policies.

20. Foundry Models

Foundry Models provides access to models through Microsoft's model catalog. Developers can select models based on the requirements of their application or agent.

The current Foundry platform provides access to models from Microsoft and multiple model providers.

The important concept for AI-103 is:

21. Foundry Agent Service

Microsoft Foundry Agent Service is a managed platform for building, deploying, and scaling AI agents.

An agent combines:

1. Model - provides reasoning and language capabilities.
2. Instructions - define the agent's goals and behavior.
3. Tools - allow the agent to access information or perform actions.

Foundry Agent Service supports both prompt agents and hosted agents, allowing developers to choose between a more declarative approach and bringing their own agent code.

22. Foundry IQ

Foundry IQ provides a knowledge layer that allows agents to retrieve information from organizational data.

A Foundry IQ knowledge base can connect to multiple knowledge sources and use agentic retrieval to find relevant information. It can work with sources such as:

• Azure Blob Storage
• SharePoint
• OneLake
• Public web data

The retrieved information can then be used to provide grounded responses.

This is particularly relevant to AI applications that need to work with organizational knowledge rather than relying only on the model's training data.

23. Foundry Tools

Foundry Tools are Microsoft's prebuilt AI capabilities for common AI scenarios.

Microsoft's current terminology uses Foundry Tools for what was previously commonly referred to as Azure AI Services.

Foundry Tool	Example capability
Speech	Speech-to-text and text-to-speech
Translator	Language translation
Language	Natural language processing
Vision	Image and video analysis
Document Intelligence	Extract information from documents
Content Understanding	Analyze and understand multimodal content
Content Safety	Detect potentially harmful content
Azure AI Search	Search and information retrieval

Microsoft provides these capabilities through APIs and SDKs for application development.

24. Foundry Local

Foundry Local is designed for running supported AI models directly on a user's device. This is different from a cloud-hosted model.

Foundry Local is useful when an application needs:

• Offline operation
• Low latency
• Local inference
• Data privacy
• No dependency on a cloud inference endpoint

Microsoft states that inference runs entirely on the device and that an Azure subscription is not required.

25. Copilot vs Copilot Studio vs Microsoft Foundry

This is one of the most useful comparisons to remember.

Offering	Primary purpose	Typical approach
Microsoft Copilot	Use existing AI assistants	Ready-to-use
Copilot Studio	Build agents and agent flows	Low-code
Microsoft Foundry	Build custom AI applications and agents	Developer/code-first
Foundry Tools	Add specific AI capabilities	APIs / SDKs
Foundry Local	Run supported models locally	On-device

26. Putting Everything Together

Imagine you're building an AI-powered customer-support application. You might use:

You could then expose the resulting experience through an application or supported publishing channel.

Alternatively, if your organization wants a low-code solution, Copilot Studio may be more appropriate.

The correct choice depends on the requirements, development skills, level of control, and architecture of the solution.

27. AI-103 Developer Perspective

For AI-103, it is useful to understand why you would choose one option over another.

Use Microsoft Copilot when:

You want an existing AI assistant for a particular productivity, development, security, or Azure scenario.

Use Copilot Studio when:

You want to create an agent or workflow using a graphical, low-code experience.

Use Microsoft Foundry when:

You are developing custom AI applications and agents and need developer-oriented control over models, agents, tools, evaluation, and deployment.

Use Foundry Tools when:

You need a specialized AI capability such as:

• Speech
• Vision
• Language
• Translation
• Document Intelligence
• Content Safety

Use Foundry Local when:

You need AI inference directly on the user's device, particularly for offline, low-latency, or privacy-sensitive scenarios.

28. AI-103 Exam Takeaways

Prompt Engineering

Remember the five core components:

A strong prompt should minimize ambiguity and clearly communicate what the model needs to accomplish.

Microsoft AI Ecosystem

Remember the high-level distinction:

Foundry Agent

An AI agent can be thought of as:

The model provides the reasoning capability. Instructions define what the agent should do. Tools allow the agent to access data and perform actions.

Quick Revision

Concept	Key Point
Prompt	Instruction/input given to an AI model
Prompt Engineering	Designing effective prompts for better results
Role	Defines the perspective or expertise
Task	Defines what the model should accomplish
Context	Provides relevant background
Input Data	Data the model should work with
Output Format	Specifies how the response should be returned
Microsoft Copilot	Ready-to-use AI experiences
Copilot Studio	Low-code agents and agent flows
Microsoft Foundry	Platform for developing AI apps and agents
Foundry Models	Models available through the Foundry model catalog
Foundry Agent Service	Managed platform for building, deploying, and scaling agents
Foundry IQ	Knowledge layer for agentic retrieval
Foundry Tools	Prebuilt AI capabilities
Foundry Local	Local/on-device AI inference

Final Takeaway

Prompt engineering is about giving the model enough information and direction to produce the desired result.

A simple prompt may be sufficient for a simple task. For more complex AI applications, however, you should think carefully about:

At the platform level, Microsoft provides different ways to build and consume AI.

Microsoft Copilot provides ready-to-use AI experiences.

Copilot Studio provides a low-code way to build agents and agent flows.

Microsoft Foundry provides the developer platform for building custom AI applications and agents, with models, tools, knowledge, evaluation, and enterprise capabilities.

Foundry Tools provide specialized AI capabilities such as Speech, Vision, Language, Document Intelligence, and Content Safety.

Foundry Local extends AI inference to the user's device.

Understanding these distinctions is an important foundation for the development scenarios covered by AI-103: Developing AI Apps and Agents on Azure.

AI-103 - Table of Contents

Develop AI apps and agents on Azure with study guides covering every exam domain.

AI-103 - Developing AI Apps and Agents on Azure Main page - all AI-103 topics → Azure Global Backbone, Regions, Region Pairs & Availability Zones Global infrastructure foundations for AI workloads Prompt Engineering and Microsoft AI Services Prompt components, Copilot, Copilot Studio, Foundry Microsoft Foundry Models, Agent Service, Foundry IQ, Tools, Foundry Local Foundry Models in Microsoft Foundry Pre-built tools, foundation models, RAG, fine-tuning Agentic AI: From Generative AI to Autonomous AI Agents Agent tools, multi-agent systems, the agentic loop