Hallucination evaluation of language models using GraphEval

# introduction

hallucination This is one of the most well-known problems. Large-scale language model (LLM) This can occur when generating a response. These occur when the model produces responses that are factually incorrect, nonsensical, or simply fabricated, usually due to the model’s lack of internal knowledge about the problem.

Although many solutions have been generated to tackle the problem of model hallucinations in recent years, methodological evaluation frameworks for internally diagnosing them are relatively unexplored. one recent research Amazon researchers propose using knowledge graphs as a means to analyze and detect hallucinations that occur in LLM. The framework presented in the study is named: graph evaluation.

This article takes a gentle, hands-on approach to explain the conceptual building blocks of GraphEval through lightweight, simulation-based code examples that you can easily try on your machine.

# Overview of GraphEval

GraphEval leverages the knowledge graph to identify and signal hallucinations in the LLM-generated output. Unlike traditional performance metrics that provide a single score to evaluate aspects such as accuracy and certainty, GraphEval applies a two-step evaluation process that emphasizes explainability. That is, it provides insight into where exactly the hallucinations occur.

To do this, GraphEval considers two stages.

• Build a knowledge graph from the generated model responses. A graph consists of semantic triples of the form: (subject, relationship, object)Here, subjects and objects correspond to nodes, and relations correspond to edges connecting those nodes.
• Through a natural language inference (NLI) model, each triple in the constructed knowledge graph is evaluated against the source context (ground truth body of knowledge). According to the NLI engine, any triple that is out of context because it is contradictory or neutral will be flagged as hallucinatory.

# GraphEval explained with code example

Before you start code that simulates an application for the GraphEval framework, make sure that the required libraries are installed.

!pip install -q transformers networkx matplotlib torch

The purpose of the following code examples is to clearly explain how the GraphEval technique works. Therefore, we replace stages that require large computational loads in real-world settings with lightweight simulated alternatives.

Therefore, we simulate a ground truth knowledge base (context) that is assumed to contain factual information. In a production environment, this true knowledge can be obtained, for example, by retrieving relevant documents from a vector database in a search augmentation generation (RAG) system. For simplicity, we will directly create the ground truth context and save it. source_context.

# The ground-truth context provided to the LLM
source_context = (
    "GraphEval is a hallucination evaluation framework based on representing information "
    "in Knowledge Graph (KG) structures. It acts as a pre-processing step and utilizes "
    "out-of-the-box NLI models to detect factual inconsistencies."
)

Now let’s assume that the following is the original LLM response to a user prompt such as “Please briefly explain what GraphEval is.” To begin the first stage of the evaluation process, ask your co-LLM to build a knowledge graph from their responses. Both the responses and follow-up prompts used to obtain the knowledge graph are shown below.

# The generated response we want to evaluate (contains a hallucination)
llm_output = (
    "GraphEval is an evaluation framework that uses Knowledge Graphs. "
    "It requires a highly expensive, enterprise-level server farm to operate."
)

# Prompt template that would theoretically be passed to a local/free LLM (e.g. Mistral-7B)
KG_EXTRACTION_PROMPT = f"""
You are an expert information extractor. Extract the core information from the following text as a Knowledge Graph.
Return the output strictly as a Python list of tuples in the format: (Subject, Relationship, Object).

Text: {llm_output}
"""

Once again, for simplicity and to avoid the heavy computational load of running a large LLM locally, assume that the following graph triples are obtained:

# Simulated extraction to bypass the heavy computational load of running a massive LLM locally
extracted_triples = [
    ("GraphEval", "is", "evaluation framework"),
    ("GraphEval", "uses", "Knowledge Graphs"),
    ("GraphEval", "requires", "expensive enterprise server farm")
]

print("Extracted Triples:")
for t in extracted_triples:
    print

Extracted Triples:
('GraphEval', 'is', 'evaluation framework')
('GraphEval', 'uses', 'Knowledge Graphs')
('GraphEval', 'requires', 'expensive enterprise server farm')

from transformers import pipeline

# Loading the open-source NLI model
print("Loading DeBERTa NLI model...")
nli_evaluator = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-small")

def evaluate_triple(context, triple):
    subject, relation, obj = triple
    hypothesis = f"{subject} {relation} {obj}"

    # Checking if the context entails the hypothesis
    result = nli_evaluator({"text": context, "text_pair": hypothesis})

    # NLI models normally output: 'entailment', 'neutral', or 'contradiction'
    label = result['label'].lower()

    # In GraphEval, anything other than 'entailment' is flagged as a hallucination
    is_hallucinated = label != 'entailment'

    return is_hallucinated, label, hypothesis

# Running the evaluation pipeline
evaluation_results = []

print("\n--- GraphEval Results ---")
for t in extracted_triples:
    is_hallucinated, nli_label, hypothesis = evaluate_triple(source_context, t)
    evaluation_results.append((is_hallucinated, nli_label))

    status = "🚨 HALLUCINATION" if is_hallucinated else "✅ GROUNDED"
    print(f"{status} | Triple: {t} | NLI Output: {nli_label}")

--- GraphEval Results ---
✅ GROUNDED | Triple: ('GraphEval', 'is', 'evaluation framework') | NLI Output: entailment
✅ GROUNDED | Triple: ('GraphEval', 'uses', 'Knowledge Graphs') | NLI Output: entailment
🚨 HALLUCINATION | Triple: ('GraphEval', 'requires', 'expensive enterprise server farm') | NLI Output: neutral

import networkx as nx
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

def visualize_grapheval(triples, eval_results):
    G = nx.DiGraph()
    edge_colors = []

    for (triple, res) in zip(triples, eval_results):
        sub, rel, obj = triple
        is_hallucinated = res[0]

        G.add_node(sub)
        G.add_node(obj)
        G.add_edge(sub, obj, label=rel)

        # Color-code the edges based on the NLI evaluation
        edge_colors.append('red' if is_hallucinated else 'green')

    # Set up the plot
    plt.figure(figsize=(10, 6))
    pos = nx.spring_layout(G, seed=42)

    # Draw nodes
    nx.draw_networkx_nodes(G, pos, node_color="lightblue", node_size=2500)
    nx.draw_networkx_labels(G, pos, font_size=10, font_weight="bold")

    # Draw edges and labels
    nx.draw_networkx_edges(G, pos, edge_color=edge_colors, width=2.5, arrowsize=20)
    edge_labels = nx.get_edge_attributes(G, 'label')
    nx.draw_networkx_edge_labels(G, pos, edge_labels=edge_labels, font_color="black")

    # Add legend
    green_patch = mpatches.Patch(color="green", label="Grounded (Entailment)")
    red_patch = mpatches.Patch(color="red", label="Hallucination (Neutral/Contradiction)")
    plt.legend(handles=[green_patch, red_patch], loc="lower right")

    plt.title("GraphEval Hallucination Map", fontsize=14, fontweight="bold")
    plt.axis('off')
    plt.tight_layout()
    plt.show()

# Render the knowledge graph
visualize_grapheval(extracted_triples, evaluation_results)