How ANOMALO solves unstructured data quality issues Provide reliable assets for AI using AWS

This post is co-authored with Anomalo's Vicky Andonova and Jonathan Karon.

Generated AI has rapidly evolved from novelty to a powerful driver of innovation. From summarizing complex legal documents to turning on advanced chat-based assistants, AI capabilities are expanding at an increasing pace. Large-scale language models (LLMS) continue to push new boundaries, but quality data remains a determinant for achieving real-world impacts.

A year ago, it seemed that the main differentiator of generating AI applications was those who could afford to build or use the biggest models. However, recent breakthroughs in base model training costs (such as DeepSeek-R1) and continuous pricing improvements have made strong models more product-friendly. Success in Generation AI is less about building the right model, and less about finding the right use case. As a result, competitiveness has shifted towards data access and data quality.

In this environment, businesses are positioned to be excellent. They have hidden goldmine of unstructured texts that span decades. Everything from calling transcripts and scan reports to supporting tickets and social media logs. The challenge is how you use that data. Converting unstructured files, maintain compliance and mitigating data quality issues all become important hurdles as organizations move from AI pilots to production deployments.

In this post, we explore how to use Anomalo with Amazon Web Services (AWS) AI and Machine Learning (AI/ML) to profile, validate, and cleanse data lakes into trusted sources of production-enabled AI initiatives, as shown in the diagram below.

Issues: Analyze unstructured enterprise documents of scale

Despite the widespread adoption of AI, many corporate AI projects fail due to poor data quality and poor control. Gartner predicts that 30% of its generation AI projects will be abandoned in 2025. Even most data-driven organizations focus primarily on the use of structured data, with unstructured content remaining underutilized in data lakes or file systems. But more than 80% of enterprise data is unstructured (according to MIT Sloan School Research), ranging from legal contracts and financial applications to social media posts.

For Chief Information Officers, Chief Technology Officers (CTOs), and Chief Information Security Officers (CISOs), unstructured data represents both risk and opportunity. Before using unstructured content in a generated AI application, the following important hurdles must be addressed:

• extraction – Optical character recognition (OCR), analysis, and metadata generation may not be reliable if not automated. Furthermore, inconsistent or incomplete extracts can lead to incorrect data.
• Compliance and security – Processing of Personally Identifiable Information (PII) or your own Intellectual Property (IP) requires strict governance, particularly in the EU AI Act, Colorado AI Act, General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), and similar regulations. Confidential information is difficult to identify in unstructured text, which can lead to careless errors in that information.
• Data Quality – Incomplete, deprecated, duplicated, topical, or unwritten data can contaminate the generated AI model and the search-extended generation (RAG) context and produce hallucinated, outdated, inappropriate, or misleading output. Ensuring that your data is of high quality can reduce these risks.
• Scalability and cost – Training or fine-tuning models on loud data increase computational costs by unnecessarily expanding the training dataset (the computational costs of training tend to grow linearly in dataset size).

In short, generative AI initiatives often shaking, not because of insufficient underlying models, but because existing data pipelines are not designed to process unstructured data and not to meet large amounts of high quality intake and compliance requirements. Many companies are in the early stages of addressing these hurdles and face these issues in their existing processes.

• It takes manual and time – Analysis of a huge collection of unstructured documents relies on manual reviews by employees, creating a time-consuming process that delays projects.
• Error-prone – Human reviews are susceptible to mistakes and inconsistencies, leading to careless exclusion of important data and inclusion of incorrect data.
• Resource Intensive – The manual document review process requires important staff time that is more suitable for valued business activities. Budgets cannot support the staffing level required to review an enterprise document collection.

While existing document analysis processes provide valuable insights, they are not efficient or accurate enough to meet the latest business needs for timely decision-making. Organizations need solutions that help them process large amounts of unstructured data and maintain regulatory compliance while protecting sensitive information.

Solution: An enterprise-grade approach to unstructured data quality

Anomalo can be used to detect, isolate and address data quality issues in unstructured data in minutes rather than weeks using the extremely secure and scalable stack provided by AWS. This helps data teams deliver high-value AI applications faster and less risk. The architecture of the anomalo solution is shown in the following diagram.

1. Automated Intake and Metadata Extraction – ANOMALO uses Amazon Elastic Cloud Compute (Amazon EC2) instances (Amazon EKS), Amazon Elastic Container Registry (Amazon ECR) to automate the OCR and text analysis of PDF files, PowerPoint presentations, and word documents stored in Amazon Simple Storage Service (Amazon S3).
2. Continuous Data Observability – Anomalo inspects each batch of extracted data and detects anomalies such as text, empty fields, and duplications that have been truncated before the data reaches the model. In the process, you will monitor for flagging unstructured pipeline health, failed documents or abnormal data drifts (for example, new file formats, unexpected numbers added or removed, or document size changes). This information reviewed and reported by Anomalo allows engineers to spend less time manually combing via logs and more time optimizing AI capabilities, but CISO gains visibility into data-related risks.
3. Governance and Compliance – Built-in problem detection and policy enforcement helps mask or remove PII and abusive language. If a batch of scanned documents contains a personal address or your own design, you can flag it for legal or security reviews. Minimizes regulatory and reputation risks. With Anomalo, you can define and extract custom issues and metadata from your documents to solve a wide range of governance and business needs.
4. Scalable AI on AWS – Anomalo uses Amazon Bedrock to provide businesses with a flexible and scalable LLM choice to analyze document quality. Anomalo's latest architecture can be deployed via Software as a Service (SAAS) or Amazon Virtual Private Cloud (Amazon VPC) connections to meet your security and operational needs.
5. Trusted data for AI business applications – The verified data layers provided by Anomalo and AWS Glue help ensure that only clean and approved content flows to your application.
6. Supports generation AI architecture – Whether you use fine-tuning or continuous training in LLM to create subject experts, save content in RAG's vector databases, or experiment with other generated AI architectures, improve application output, maintain brand trust and reduce business risks by ensuring that your data is clean and validated.

Impact

Using Anomalo and AWS AI/ML services for unstructured data provides these benefits.

• The operational burden has been reduced – Saves development time and months of continuous maintenance for anomalo's ready-made rules and evaluation engines, saving time for freeing up new features instead of developing data quality rules.
• Optimized Costs – Training LLMS and ML models on low-quality data wastes valuable GPU capacity, but vectorizing and storing that data for RAG increases overall operational costs and both reduce application performance. Initial data filtering reduces these hidden costs.
• Fast time to insight – Anomalo automatically classifies and labels unstructured text, providing data scientists with rich data, spins up new generated prototypes or dashboards without slacking off Prework labels.
• Enhanced compliance and security – Identifying PIIs and adhering to data retention rules is built into the pipeline, supporting security policies and reducing the preparation required for external audits.
• Create durable values – Generated AI landscapes continue to evolve rapidly. LLM and application architecture investments can depreciate quickly, but reliable curated data is a sure bet that won't be wasted.

Conclusion

Generating AI can bring value at scale. Gartner estimates there are 15-20% revenue growth, 15% cost savings, and 22% productivity gains. To achieve these results, your application must be built on a reliable, complete, timely data foundation. Anomalo helps to deliver more AI projects to production faster, while meeting both user and governance requirements.

Are you interested in learning more? Check out Anomalo's unstructured data quality solutions and request a demo or contact us for a detailed discussion on how to start or expand your generation AI journey.

---

About the author

Vicki Andnova Anomalo Generate AI GM and reinvents the quality of enterprise data. As a member of the founding team, Vicki has spent the past six years pioneering Anomalo's machine learning initiatives, transforming advanced AI models into actionable insights that will help businesses trust their data. Today, she leads a team that not only offers innovative generation AI products to the market, but also builds a class of data quality monitoring solutions designed specifically for unstructured data. Previously, with Instacart, Vicky built the company's experimental platform, leading company-wide initiatives to grocery delivery quality. She graduated from Columbia University.

Jonathan Carron He leads his partner innovation at Anomalo. He works closely with businesses across the data ecosystem to integrate data quality monitoring into key tools and workflows, allowing businesses to achieve high-performance data practices and leverage new technologies faster. Before Anomalo, Jonathan created mobile app observability, data intelligence, and DevSecops products at New Relic and was the head of the product at a startup of generative AI sales and customer success. He holds a bachelor's degree in cognitive science from Hampshire University and has worked with AI and data exploration techniques throughout his career.

Mahesh Biradar He is AWS Senior Solutions Architect with a history in the IT and services industry. He helps US SMBs achieve their business goals with cloud technology. He holds a bachelor's degree in engineering from VJTI and is based in New York City (USA).

Emad Tawfik He is a veteran senior solutions architect at Amazon Web Services and boasts over 10 years of experience. His speciality lies in the realm of storage and cloud solutions, where he excels at creating cost-effective and scalable architectures for his customers.