How to use Semantic Kernel for open source

I was building a customer support automation tool a few months ago, and I hit a wall. I had started the project using LangChain in Python because, well, that's what everyone was using. But as the project grew, the "wild west" nature of the library started biting me. Abstractions leaked everywhere, updates broke my code, and honestly, the lack of strong typing made refactoring a nightmare. I was spending more time fighting the framework than building features.

I needed something more structured. That's when I stumbled across Semantic Kernel (SK), Microsoft's open-source SDK. I was skeptical—Microsoft's AI tooling can sometimes feel like a walled garden pushing you toward Azure. But what I found was a genuinely model-agnostic, MIT-licensed toolkit that felt more like enterprise middleware than a hacky wrapper. Here's how I got it running with completely open-source models, no API keys required.

Why Semantic Kernel?

The core pitch of Semantic Kernel is simple: it's a lightweight middleware that sits between your code and AI models. Think of it as a translator. The model says "I need to look up a customer's order," and SK translates that into an actual function call in your codebase, executes it, and hands the result back to the model.

What sold me was the v1.0+ commitment. Across C#, Python, and Java, the SDK is stable and promises no breaking changes. After dealing with LangChain's constant churn, that reliability was a breath of fresh air. Plus, it's model-agnostic out of the box. You can use OpenAI, Azure OpenAI, Hugging Face, or—my favorite—local open-source models running through compatible endpoints.

Setting Up the Environment

I wanted to prove to myself that SK wasn't just an Azure on-ramp. So I decided to build a simple agent using a local LLM, completely offline. For this, I used LM Studio, a fantastic app that lets you download and run open-source models locally and exposes an OpenAI-compatible API server.

First, I downloaded LM Studio and grabbed a model. I went with Mistral-7B-Instruct because it's small enough to run on my 16GB RAM laptop and handles function calling reasonably well.

Once the model was downloaded in LM Studio, I started the local server:

1. Click the "Local Server" tab in LM Studio
2. Load your model
3. Hit "Start Server"

By default, it runs on http://localhost:1234/v1. This is the magic part—because it mimics the OpenAI API format, Semantic Kernel can talk to it without knowing it's not actually OpenAI.

Building the Kernel in Python

I chose the Python SDK for this project since my team's backend is Python. Installation is straightforward:

pip install semantic-kernel

Now, here's where I made my first mistake. I initially tried to configure the connection using the older kernel.add_text_service() method I found in an outdated tutorial. That method is deprecated in v1.0+. The correct approach uses an async setup with the OpenAI connector pointed at our local server.

Here's the working configuration:

import semantic_kernel as sk
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion

async def initialize_kernel():
    kernel = sk.Kernel()
    
    # Point to our local LM Studio server instead of OpenAI
    chat_service = OpenAIChatCompletion(
        ai_model_id="local-model",  # Can be any name, LM Studio ignores this
        endpoint="http://localhost:1234/v1",
        api_key="not-needed",  # LM Studio doesn't require a key, but SK expects one
    )
    
    kernel.add_service(chat_service)
    return kernel

That api_key="not-needed" part tripped me up. The SK OpenAI connector requires an API key parameter, even if your local server doesn't use one. Just pass any string and it works fine.

Creating Your First Plugin

The real power of Semantic Kernel comes from plugins. Plugins are how you expose your existing code to the AI model. You describe what your function does, and the model decides when to call it.

Let's say I want my agent to look up customer orders. I'd create a plugin like this:

from semantic_kernel.functions.kernel_function_decorator import kernel_function

class OrderPlugin:
    @kernel_function(
        description="Looks up a customer's order by their order ID",
        name="get_order"
    )
    def get_order(self, order_id: str) -> str:
        # In reality, this would query your database
        orders = {
            "ORD-123": "Status: Shipped - 2 items, arriving tomorrow",
            "ORD-456": "Status: Processing - 1 item, estimated ship date Friday",
            "ORD-789": "Status: Delivered - 3 items, delivered on Monday"
        }
        return orders.get(order_id, f"No order found with ID {order_id}")

The @kernel_function decorator is the key. The description is what the model sees—it uses that to decide whether to call this function. Be specific and clear in your descriptions; the model literally depends on them to understand when to use the tool.

Now, register the plugin with the kernel:

kernel = await initialize_kernel()
kernel.add_plugin(OrderPlugin(), plugin_name="orders")

Running the Agent

With the kernel configured and the plugin registered, let's actually run a conversation:

from semantic_kernel.contents.chat_history import ChatHistory
from semantic_kernel.functions.kernel_arguments import KernelArguments

async def chat_with_agent(user_input: str, chat_history: ChatHistory):
    chat_function = kernel.add_function(
        plugin_name="orders",
        function_name="get_order"
    )
    
    # Enable automatic function calling
    settings = kernel.get_prompt_execution_settings_from_service_id("local-model")
    settings.auto_invoke_kernel_functions = True
    
    chat_history.add_user_message(user_input)
    
    result = await kernel.invoke_prompt(
        prompt="{{$chat_history}}",
        arguments=KernelArguments(chat_history=chat_history),
        template_format="semantic-kernel",
        settings=settings
    )
    
    return str(result)

# Test it out
chat_history = ChatHistory()
chat_history.add_system_message(
    "You are a customer support agent. Help customers with their orders."
)

response = await chat_with_agent(
    "Can you check on my order ORD-123?",
    chat_history
)
print(response)

When I first ran this, nothing happened—the model just responded with text saying "I'll check your order" but never actually called the function. After some debugging, I realized I had forgotten to enable auto_invoke_kernel_functions. That setting is what tells SK to actually execute the plugin functions when the model requests them. Without it, the model just says it would call a function, but SK doesn't follow through.

Once I enabled that setting, the flow worked perfectly:

1. User asks about order ORD-123
2. Model recognizes it needs to call the get_order function
3. SK intercepts that request, calls OrderPlugin.get_order("ORD-123")
4. The result ("Status: Shipped...") gets passed back to the model
5. Model formulates a natural response: "Your order ORD-123 is currently shipped and will be arriving tomorrow!"

Honest Limitations

Let's be real about the trade-offs I've hit working with Semantic Kernel and open-source models:

Function calling with smaller models is hit or miss. Mistral-7B handles basic function calls, but it sometimes hallucinates parameters or calls the wrong function. If you're building something production-grade with tool use, you'll likely need a larger model like Mixtral-8x7B or Llama-3-70B, which means more hardware or renting GPU compute.

The documentation is still catching up. Microsoft's docs lean heavily toward Azure OpenAI examples. Finding the right configuration for local, open-source endpoints required digging through GitHub issues and source code. The community is active, but you'll spend time spelunking.

Python SDK feels like a port from C#. If you're a C# developer, SK will feel natural. In Python, some patterns feel verbose compared to native Python conventions. The async-everything approach can also be annoying if you're building simple synchronous scripts.

Prompt template syntax is different from LangChain. If you're migrating, you'll need to learn SK's {{$variable}} syntax and template format. It's not hard, but it's another thing to learn.

Practical Tips

After a few months with SK, here's what I'd recommend:

1. Start with LM Studio for local development. It eliminates API costs while you're experimenting, and the OpenAI-compatible server means your SK code works unchanged when you later point it at a real OpenAI or Azure endpoint.

2. Write extremely clear function descriptions. The model's ability to use your plugins correctly depends entirely on how well you describe them. "Gets order" is bad. "Retrieves the current status and details of a customer order using a unique order ID in the format ORD-XXX" is good.

3. Use the hooks and filters for observability. SK has built-in hooks that let you log every function call the model makes. This saved me hours of debugging when my agent went off the rails. Add logging hooks early, not after you have problems.

4. Don't fight the async. Just embrace that SK is async-first. If you need to call it from sync code, use asyncio.run(). Trying to work around it causes more headaches than it solves.

Semantic Kernel isn't the flashiest option in the AI SDK space, but it's become my go-to for projects that need to actually ship and keep working. The model-agnostic design means I can develop locally with open-source models and deploy to whatever endpoint makes sense for production, all without rewriting my orchestration logic. That alone makes it worth the learning curve.