Genkit is an open source framework for: Build full-stack, AI-powered agent applications for any platform Supports TypeScript, Go, Dart, and Python. Building production-ready agent applications and AI capabilities requires more than powerful models and careful prompts. Retries and fallbacks may be required to maximize reliability, human approval before invoking destructive tools, and observability across all layers.
Genkit solves this middleware: Composable hooks that intercept generation calls and inject custom behavior, including tool execution loops. This middleware system is currently available for TypeScript, Go, and Dart, with support for Python coming soon.
How Genkit middleware works
every generate() When you call Genkit, tool loop: The model produces output, the requested tools are executed, the results are fed back into new model calls, and the cycle repeats until the model is complete. Middleware hooks are connected to three layers of this loop.
|
hook |
runtime |
General usage |
|---|---|---|
|
generate |
Once per tool loop iteration |
Context injection, message rewriting, conversation level logic |
|
model |
Once per model API call |
Retries, fallbacks, caching, and delayed logging |
|
tool |
Once per tool run |
Human participation, sandboxing, and logging for each tool |
Pre-built middleware
Genkit provides several pre-built middleware solutions for common use cases. Here’s what’s available today:
1. Retry
Automatically retry failed model API calls when a temporary error occurs (RESOURCE_EXHAUSTED, UNAVAILABLEetc.) uses exponential backoff with jitter. Only model calls are retried. Surrounding tool loops are not played.
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Summarize the quarterly earnings report."),
ai.WithUse(&middleware.Retry{
MaxRetries: 3,
InitialDelayMs: 1000,
BackoffFactor: 2,
}),
)go
2. Fallback
Switch to an alternate model if the primary model fails with a specified set of error codes. Useful for falling back to an entirely different provider if your primary model exceeds its quota.
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Analyze this complex document..."),
ai.WithUse(&middleware.Fallback{
Models: []ai.ModelRef{
anthropic.ModelRef("claude-sonnet-4-6", nil), // fall back to Claude
},
Statuses: []core.StatusName{core.RESOURCE_EXHAUSTED},
}),
)go
3. Tool approval
Restricts tool execution to the allowed list. Tools not listed will trigger an interrupt and allow human confirmation before proceeding with the action.
resp, _ := genkit.Generate(ctx, g,
ai.WithPrompt("Delete the temp files"),
ai.WithTools(deleteFilesTool),
ai.WithUse(&middleware.ToolApproval{
AllowedTools: []string{}, // empty = every tool call interrupts
}),
)
if len(resp.Interrupts()) > 0 {
interrupt := resp.Interrupts()[0]
// Prompt the user for approval, then resume with the approval flag.
approved, _ := deleteFilesTool.RestartWith(interrupt,
ai.WithResumedMetadata[DeleteInput](map[string]any{"toolApproved": true}),
)
resp, err := genkit.Generate(ctx, g,
ai.WithMessages(resp.History()...),
ai.WithTools(deleteFilesTool),
ai.WithToolRestarts(approved),
ai.WithUse(&middleware.ToolApproval{}),
)
fmt.Println(resp.Text())
}go
4. Skills
scan the directory and SKILL.md Create a file and insert its contents into the system prompt. Also, use_skill Use tools to enable models to load specific skills on demand.
resp, err := genkit.Generate(ctx, g,
ai.WithPrompt("How do I deploy this service?"),
ai.WithUse(&middleware.Skills{SkillPaths: []string{"./skills"}}),
)go
5. File system
Injected tools (list_files, read_file, plus write_file and edit_file write enabled). Path safety is enforced, so the model cannot escape from the root directory.
resp, err := genkit.Generate(ctx, g,
ai.WithPrompt("Create a hello world program in the workspace"),
ai.WithUse(&middleware.Filesystem{
RootDir: "./workspace",
AllowWriteAccess: true,
}),
)go
Building custom middleware
Pre-built middleware covers common scenarios, but the real power of the system lies in creating your own scenarios. Imagine you’re building an agent-based customer support app and need to make sure that your model never mentions competitive products or internal pricing data. Rather than encoding these rules in every prompt, you can use middleware to apply them deterministically.
Custom middleware follows simple conventions across all languages. Specify the name and the factory function that returns the desired hook. The factory is called once at a time. generate() Implement only the hooks you need.
Below is a complete custom content filter consisting of about 20 lines of code.
// ContentFilter rejects model responses containing any forbidden term.
type ContentFilter struct {
ForbiddenTerms []string `json:"forbiddenTerms"`
}
func (ContentFilter) Name() string { return "app/contentFilter" }
func (f ContentFilter) New(ctx context.Context) (*ai.Hooks, error) {
return &ai.Hooks{
WrapModel: func(ctx context.Context, p *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
resp, err := next(ctx, p)
if err != nil {
return nil, err
}
text := strings.ToLower(resp.Text())
for _, term := range f.ForbiddenTerms {
if strings.Contains(text, strings.ToLower(term)) {
return nil, fmt.Errorf("content filter: response contains %q", term)
}
}
return resp, nil
},
}, nil
}go
You can also configure and stack different middleware solutions. Middleware is stacked from left to right, with the first listed being the outermost wrapper.
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("What CRM should our customer use?"),
ai.WithUse(
&middleware.Retry{MaxRetries: 3}, // outer: retries the inner stack
&ContentFilter{ // inner: validates model output
ForbiddenTerms: []string{"CompetitorCRM", "RivalCo", "internal price"},
},
),
)go
here Retry rap ContentFilterwraps the model call. Order matters, and Genkit makes it clear.
If you believe you have built middleware that is valuable to other developers, you can publish it as a package so that other developers can benefit from it.
Developer UI experience
You can use the Genkit Developer UI to inspect, test, and debug your applications, including running middleware. Once you register your middleware, it becomes visible in the development UI and you can inspect its configuration, trace its execution through each hook layer, and test different combinations.
Let’s get started
We’re excited about the capabilities Genkit middleware brings to your apps, and look forward to seeing what custom middleware you build to solve your use cases. Check out the middleware documentation for a deeper understanding, or get started with Genkit if you’re new to the framework.
Do you have an idea for new pre-built middleware? Submit an issue. We’d love to hear what helps us improve your development experience.
Have fun coding! 🚀
