Large-scale language models (LLMs) work well for common tasks, but struggle for specialized tasks that require understanding proprietary data, internal processes, and industry-specific terminology. Supervised fine-tuning (SFT) adapts LLM to these organizational contexts. SFT can be implemented through two different methodologies. One is parameter-efficient fine-tuning (PEFT), which updates only a subset of model parameters and provides faster training and lower computational cost while maintaining reasonable performance gains. Full rank SFT. It updates all model parameters instead of a subset and incorporates more domain knowledge than PEFT.
Full-rank SFTs often face the challenge of catastrophic forgetting. As models learn domain-specific patterns, they lose general abilities such as following instructions, reasoning, and broad knowledge. Organizations must choose between domain expertise and general intelligence, limiting the model’s usefulness across enterprise use cases.
Amazon Nova Forge addresses this issue. Nova Forge is a new service that you can use to build your own Frontier models using Nova. Nova Forge customers can start development with an initial model checkpoint, blend their own data with Amazon Nova-curated training data, and securely host their custom models on AWS.
In this post, the AWS China Applied Science team shares the results of a comprehensive evaluation of Nova Forge using a challenging Voice of the Customer (VOC) classification task, benchmarked against open source models. Using a sample of over 16,000 customer comments across a complex four-level label hierarchy with 1,420 leaf categories, we demonstrate how Nova Forge’s data-mixing approach provides two benefits.
- Improving performance for intra-domain tasks: Achieved 17% improvement in F1 score
- Saved general features: Maintain MMLU (Massive Multitask Language Understanding) score close to baseline and ability to follow instructions after fine-tuning
Challenge: Classifying real-world customer feedback
Consider a typical scenario in a large e-commerce company. Our Customer Experience team receives thousands of customer comments every day, including detailed feedback ranging from product quality, shipping experiences, payment issues, website usability, and customer service interactions. To operate effectively, you need an LLM that can automatically classify each comment into actionable categories with high accuracy. Each classification should be specific enough to route the issue to the appropriate team (logistics, finance, development, customer service) and trigger the appropriate workflow. This requires Domain specialization.
However, this same LLM does not work on its own. Across organizations, teams need models to:
- Generate customer-facing responses Requires general communication skills
- Perform data analysis Requires mathematical and logical reasoning
- draft document Follow specific formatting guidelines
This requires wide range of general abilities—Following instructions, reasoning, cross-domain knowledge, and conversational fluency.
Evaluation method
Test overview
To test whether Nova Forge can provide both domain specialization and general functionality, we designed a dual evaluation framework to measure performance across two dimensions.
For domain-specific performance, see Real Voice of the Customer (VOC) Dataset Derived from real customer reviews. The dataset contains 14,511 training samples and 861 test samples that reflect production-scale enterprise data. The dataset employs a four-level taxonomy, where level 4 represents leaf categories (final classification targets). Each category includes a description of its scope. Examples of categories:
| level 1 | level 2 | level 3 | Level 4 (leaf category) |
| Installation – App settings | Initial setup guidance | Setup process | Easy setup experience: Characteristics and level of complexity of the installation process |
| Usage – Hardware Experience | night vision performance | Low light image quality | night vision clarity: Night vision mode produces images even in low light or low light conditions. |
| Usage – Hardware Experience | Pan tilt zoom function | Rotation ability | 360 degree rotation: The camera can rotate 360 degrees, covering a complete panorama. |
| After-sales policy and costs | Return and Exchange Policy | Execute recovery process | Product return completed: A customer initiated and completed a product return due to a functionality issue. |
This dataset exhibits extreme class imbalance typical of real-world customer feedback environments. The following figure shows the distribution of classes.
As a result, this dataset poses significant challenges to classification accuracy.
For evaluation of general-purpose functions, publicly available Splitting the test set of MMLU (Massive Multitask Language Understanding) benchmark (all subsets). The test spans subjects in the humanities, social sciences, hard sciences, and other areas where learning is important to some people. In this post, MMLU will act as a proxy. Preservation of general abilities. We use this to measure whether supervised fine-tuning improves domain performance at the cost of degrading the underlying model’s behavior and to assess the effectiveness of Nova data mixing in mitigating catastrophic forgetting.
| item | explanation |
| Total number of samples | 15,372 customer reviews |
| label hierarchy | 4-level classification, total 1,420 classifications |
| training set | 14,511 samples |
| test set | 861 samples |
| MMLU All Benchmarks (Test Split) | 14,000 samples |
Evaluating in-domain tasks: Classifying customer testimonials
To understand how Nova Forge performs in a real-world enterprise scenario, we first evaluate the model’s accuracy on a VOC classification task before and after supervised fine-tuning. This approach allows us to quantify domain adaptation gains while establishing a baseline for subsequent robustness analyses.
Base model evaluation
First, Base model evaluation You can evaluate the out-of-the-box performance of VOC classification tasks without any task-specific fine-tuning. This setting establishes each model’s unique ability to handle very fine-grained classification under strict output format constraints. The following prompts are used for VOC classification tasks.
# Role Definition
You are a rigorous customer experience classification system. Your sole responsibility is to map user feedback to the existing label taxonomy at Level 1 through Level 4 (L1–L4). You must strictly follow the predefined taxonomy structure and must not create, modify, or infer any new labels.
## Operating Principles
### 1. Strict taxonomy alignment
All classifications must be fully grounded in the provided label taxonomy and strictly adhere to its hierarchical structure.
### 2. Feedback decomposition using MECE principles
A single piece of user feedback may contain one or multiple issues. You must carefully analyze all issues described and decompose the feedback into multiple non-overlapping segments, following the MECE (Mutually Exclusive, Collectively Exhaustive) principle:
- **Semantic singularity**: Each segment describes only one issue, function, service, or touchpoint (for example, pricing, performance, or UI).
- **Independence**: Segments must not overlap in meaning.
- **Complete coverage**: All information in the original feedback must be preserved without omission.
### 3. No taxonomy expansion
You must not invent, infer, or modify any labels or taxonomy levels.
## Label Taxonomy
The following section provides the label taxonomy: {tag category}. Use this taxonomy to perform L1–L4 classification for the original VOC feedback. No taxonomy expansion is allowed.
## Task Instructions
You will be given a piece of user feedback: {user comment}. Users may come from different regions and use different languages. You must accurately understand the user's language and intent before assigning labels.
Refer to the provided examples for the expected labeling format.
## Output Format
Return the classification results in JSON format only. For each feedback segment, output the original text along with the corresponding L1–L4 labels and sentiment. Do not generate or rewrite content.
```json
[
{
"content": "
"L1": "
"L2": "
"L3": "
"L4": "
"emotion": "
}
]
```
For the evaluation of the basic model, we selected:
| model | accuracy | recollection | F1 score |
| nova 2 light | 0.4596 | 0.3627 | 0.387 |
| Quen 3-30B-A3B | 0.4567 | 0.3864 | 0.394 |
F1 scores reveal that Nova 2 Lite and Qwen3-30B-A3B show comparable performance For this domain-specific task, both models achieved F1 scores close to 0.39. These results also highlight the inherent difficulty of this task. Even powerful underlying models struggle with fine-grained label classification when domain-specific data is not provided.
Supervised fine-tuning
Apply next Full parameter supervised fine tuning (SFT) Use customer VOC data. All models were fine-tuned using the same dataset and comparable training configurations to make a fair comparison.
Training infrastructure:
Performance comparison of intra-domain tasks
| model | training data | accuracy | recollection | F1 score |
| nova 2 light | None (baseline) | 0.4596 | 0.3627 | 0.387 |
| nova 2 light | Customer data only | 0.6048 | 0.5266 | 0.5537 |
| Quen 3-30B | Customer data only | 0.5933 | 0.5333 | 0.5552 |
After fine-tuning only the customer data, Nova 2 Lite delivers significant performance improvementsF1 increased from 0.387 to 0.5537, an absolute increase of 17 points. This result puts the Nova model at the top of the list for this task, and its performance is comparable to the fine-tuned Qwen3-30B open-source model. These results are Nova Full parameter SFT For complex enterprise classification workloads.
General functional evaluation: MMLU benchmark
Models fine-tuned for VOC classification are often deployed beyond single tasks and integrated into broader enterprise workflows. It is important to maintain generic functionality. Industry standard benchmarks such as MMLU provide an effective mechanism for evaluating general-purpose features and detecting catastrophic forgetting in fine-tuned models.
For fine-tuned Nova models, Amazon SageMaker HyperPod provides ready-to-use evaluation recipes that streamline MMLU evaluation with minimal configuration.
| model | training data | VOC F1 score | MMLU accuracy |
| nova 2 light | None (baseline) | 0.38 | 0.75 |
| nova 2 light | Customer data only | 0.55 | 0.47 |
| nova 2 light | 75% Customer + 25% Nova Data | 0.5 | 0.74 |
| Quen 3-30B | Customer data only | 0.55 | 0.0038 |
Nova 2 Lite is fine-tuned using only customer data. MMLU accuracy decreased significantly from 0.75 to 0.47indicating loss of general purpose function. The degradation is even more pronounced in the Qwen model, which loses a significant amount of its ability to follow instructions after fine-tuning. Example of degraded output of Qwen model:
{
"prediction": "[\n {\n \"content\": \"x^5 + 3x^3 + x^2 + 2x in Z_5\",\n \"A\": \"0\",\n \"B\": \"1\",\n \"C\": \"0,1\",\n \"D\": \"0,4\",\n \"emotion\": \"neutral\"\n }\n]"
}This behavior is also relevant for VOC prompt design, where category knowledge is internalized through supervised fine-tuning, a common approach in large-scale classification systems.
especially, Nova data mixing When applied during tweaks, Nova 2 Lite maintains general performance close to baseline. MMLU accuracy is maintained 0.74is only 0.01 lower than the original baseline, but VOC F1 is still improved by 0.01. 12 points (0.38→0.50). this validates it Nova data mixing is a practical and effective mechanism Reduce deadly forgetfulness while maintaining domain performance.
Key findings and practical recommendations
This evaluation shows that the basic model provides a strong foundation; Amazon Nova Forge’s fully parameter-supervised fine-tuning can deliver significant benefits for complex enterprise classification tasks. At the same time, the results confirm that catastrophic forgetting is a major concern when fine-tuning production workflows. Fine-tuning only customer data reduces general-purpose capabilities such as following instructions and reasoning, which can limit the model’s usefulness across a broader range of business scenarios.
of Nova Forge’s data mixing capabilities provide an effective mitigation strategy. By blending customer data with Nova’s curated datasets during fine-tuning, teams can continue to achieve strong domain-specific performance while maintaining common functionality close to the baseline.
Based on these findings, we recommend the following methods when using Nova Forge:
- Use supervised fine-tuning to maximize performance within the domain of complex or highly customized tasks.
- If your model is expected to support multiple generic workflows in production, apply Nova data mixing to reduce the risk of catastrophic forgetting.
Together, these practices balance model customization and operational robustness, allowing for more reliable deployment of fine-tuned models in enterprise environments.
conclusion
In this post, we demonstrated how organizations can use Nova Forge’s data mixing capabilities to build specialized AI models without sacrificing general intelligence. Depending on your use case and business goals, Nova Forge can also provide other benefits, such as access to checkpoints across all phases of model development and the ability to perform reinforcement learning with reward functions in your environment. To start experimenting, see the Nova Forge Developer Guide for detailed documentation.
About the author
Yuan Wei He is an applied scientist at Amazon Web Services, working with enterprise customers on proofs of concept and technical advice. She specializes in large-scale language models and vision language models, with an emphasis on evaluating new technologies under real-world data, cost, and system constraints.
Xin Hao He is a senior AI/ML Go-to-Market specialist at AWS, helping customers achieve success with Amazon Nova models and related generative AI solutions. He has extensive hands-on experience in cloud computing, AI/ML, and generative AI. Prior to joining AWS, Xin worked in the industrial manufacturing sector for over 10 years, including industrial automation and CNC machining.
Sharon Lee I’m an AI/ML Specialist Solutions Architect at Amazon Web Services (AWS) based in Boston, Massachusetts. With a passion for leveraging cutting-edge technology, Sharon is at the forefront of developing and deploying innovative generative AI solutions on the AWS cloud platform.
