What OpenAI is asking is Third-party contractors can upload actual assignments and tasks from their current or former workplaces and use that data to evaluate the performance of next-generation AI models, according to records obtained by OpenAI and training data company Handshake AI.
This project appears to be part of OpenAI's efforts to establish human baselines for various tasks that AI models can be compared to. In September, the company launched a new evaluation process to measure the performance of its AI models against human experts in various industries. OpenAI says this is an important indicator of progress toward AGI, or AI systems that outperform humans at the most economically valuable tasks.
One of OpenAI's confidential documents reads, “We hired people from a variety of professions to help us collect real-world tasks modeled after tasks they performed in their full-time jobs. This allowed us to measure how well our AI models performed on those tasks.” “Take existing pieces of long-term or complex work (more than a few hours or days) that you have done in your profession and convert each one into a task.”
According to an OpenAI presentation on the project seen by WIRED, OpenAI asks contractors to describe their current work or tasks they have completed in the past and upload examples of their work. Each example should be a “specific output (the actual file, not a file summary), such as a Word doc, PDF, Powerpoint, Excel, image, repository, etc.,” notes the presentation notes. OpenAI says people can also share fabricated working examples created to demonstrate how they would realistically respond in a particular scenario.
OpenAI and Handshake AI declined to comment.
According to OpenAI's presentation, a real-world task has two components. There are task demands (what the person's manager or co-workers tell the person to do) and task products (the actual work the person produces in response to those requests). The company emphasizes multiple times in its instructions that the examples contractors share should reflect that person's “actual field work.”actually end. “
One example of an OpenAI presentation outlines a task from a “senior lifestyle manager at a luxury concierge company for ultra-high-net-worth individuals.” The goal: “Create a short two-page PDF draft outlining a 7-day yacht trip to the Bahamas for a family traveling to the Bahamas for the first time.” This includes additional details about your family's interests and what your itinerary should be. Next, “Experienced Human Deliverables” shows what the contractor in this case uploads: the actual Bahamas itinerary created for the client.
OpenAI instructs contractors to remove company intellectual property and personally identifiable information from the work files they upload. In a section titled “Important Notes,” OpenAI instructs employees to “delete or anonymize any personal information, proprietary or confidential data, and material non-public information (such as internal strategies, unpublished product details, etc.).”
One of the files viewed by WIRED mentions a ChatGPT tool called “Superstar Scrubing” that provides advice on how to remove sensitive information.
Evan Brown, an intellectual property attorney at Neal & McDevitt, told WIRED that an AI lab on this scale that receives confidential information from contractors could be subject to trade secret misappropriation claims. Contractors who provide documents, even scrubbed ones, from their previous workplaces to AI companies may be at risk of violating non-disclosure agreements with their former employers or compromising trade secrets.
“AI Labs puts a lot of trust in contractors to determine what is and is not sensitive,” Brown says. “If something is leaked, are AI labs really taking the time to determine what is and isn't a trade secret? It seems to me that AI labs are putting themselves at great risk.”
