In the digital age, government agencies are grappling with unprecedented amounts of data, challenging the effective management, access, and declassification of information.
The State Department is no exception. According to Eric Stein, deputy assistant secretary in the Office of Global Information Services, the agency's eRecords archive system currently contains more than 4 billion of his artifacts, including email and cable traffic. “The latter is a way to communicate with embassies overseas,” Stein said.
However, over time, authorities will have to declare what is available to the public and what remains confidential, a process that takes time and effort.
The State Department is turning to cutting-edge technologies such as artificial intelligence (AI) and machine learning (ML) to find more efficient solutions. Through his three pilot projects, the agency successfully streamlined the document review process for declassification and improved the customer experience with FOIA (Freedom of Information Act) requests.
Declassification efforts using ML
At the root of this issue is Executive Order 13526, which requires that classified records of permanent historical value be automatically declassified after 25 years unless a review determines an exemption. . For the State Department, cables are among the most historically important records produced by the department. However, current processes and resource levels do not work to review electronic records, including classified emails, created since the early 2000s, jeopardizing declassification reviews starting in 2025.
Recognizing the need for a more efficient process, the Department launched a pilot for declassification review using ML in October 2022. Stein came up with the idea for this pilot after he participated in the AI Federal Leadership Program, which is supported by major cloud providers including Microsoft.
In the pilot, the department used cables from 1997 and created a review model based on human judgments in 2020 and 2021 on cables marked classified in 1995 and 1996. This model uses discriminative AI to score cables and classify them into three categories: What we believe should be declassified, what we believe should not be declassified, and what requires manual review.
According to Stein, for a 1997 pilot group of more than 78,000 cables, the model performed as well as human reviewers 97% to 99% of the time and saved staff at least 60% of their time. Reduced.
“We create a project [this technology] Instead of requiring more money for human resources and different tools to support this, we can use this technology, which will lead to millions of dollars in cost savings over the next few years,” Stein explained. Did. “That way we can focus our human resources on higher-level analytical thinking and making some of the more difficult decisions, as opposed to processes that were very manual.”
Pay attention to FOIA
Following the success of the declassification efforts, the State Department has launched two other pilots to enhance the Freedom of Information Act (FOIA) process from June 2023 to February 2024.
Similar to cable declassification efforts, processing FOIA requests is a highly manual process. According to Stein, these requests he makes can be as little as one sentence. Others are multiple pages. However, regardless of length, a staff member must approve the request, advise whether the department should move forward with it, and then manually search terms within those requests in separate databases to find relevant information. there is.
Stein said that by leveraging the lessons learned from the declassification exam, State Department staff has the opportunity to streamline certain parts of the FOIA process by simultaneously searching for what is already in the State Department's reading rooms and archives holdings. He said he noticed something.
“If that information is already publicly available, we can immediately inform the requester,” Stein said. “Even if this is not the case, if there are similar studies or reviews already conducted by the agency, you can leverage those existing studies, resulting in significant savings in staff time and response time. It will connect.”
Beyond internal operations, the State Department also sought to improve the customer experience for FOIA applicants by modernizing its public website and search capabilities. The department uses AI-driven search algorithms and automated request processing to “find and direct customers to existing released documents” and “automate customer engagement early in the request process. ” is what we aim to do.
lessons learned
Since launching the first pilot in 2022, team members have learned a few things. The first is to start small and give them the space and time to get used to the technology. “There are always demands and things to do, but it’s important to have time to focus and learn,” Stein says.
Another lesson is the importance of collaboration. “It was helpful to have conversations among different communities to not only understand how this technology could be beneficial, but also understand what concerns were being raised, and to address those early on. “We can talk,” he said. “The sooner everyone starts spending time thinking critically about AI and machine learning, the better.”
Another lesson: “Recognize that you need to continually train your model, because you can't just do this once and leave it alone.” You always have to (consider) how you are training your models,” he said.
These pilots also demonstrated how the technology can help State Department employees address other needs, such as FOIA requests. For example, someone might request something a certain way, but that's not the way to discuss it internally.
“This technology allows us to say, 'Oh, they asked for this, but maybe that's what they meant,'” Stein says. “So you can make connections that may have been missing before.”
The State Department’s strategic adoption of AI and ML technologies in its records management and transparency efforts highlights the transformative potential of these tools. By starting small, fostering collaboration, and prioritizing user-centered design, the department paved the way for broader applications of AI and ML to support more efficient and transparent government operations.
This report was produced by FedScoop's Scoop News Group as part of an investigation. Series on innovation in governmentunderwriter microsoft federal government. Want to learn more about Microsoft's AI for government? Sign up here Receive news and updates about how advanced AI can power your organization.