OpenAI Urges Contractors to Share Previous Work for AI Evaluation

OpenAI is seeking the assistance of third-party contractors by requesting them to submit authentic tasks and assignments sourced from their previous roles. This effort is part of their process to gauge the effectiveness of upcoming AI models, as reported by records procured by WIRED from OpenAI and Handshake AI.

This initiative falls under OpenAI’s strategy to establish a foundational human performance baseline across various job functions, serving as a comparison point with AI outcomes. As of September, they initiated a methodology designed to assess how AI systems perform relative to human experts across diverse sectors. OpenAI regards this comparison as integral to its progress towards its ultimate goal of creating AI systems that surpass human capabilities in economically significant tasks.

According to a confidential document from OpenAI, professionals have been recruited to assist in curating real-world tasks similar to those performed in their full-time roles, which will help determine the proficiency of AI systems. The directive asks contractors to identify long-term or intricate tasks (those requiring hours or days) from their professional experience and convert these into testable tasks.

Contractors are instructed to provide detailed descriptions of work they have done currently or in previous positions, along with actual samples of their output, as seen in a presentation from OpenAI reviewed by WIRED. The requirement specifies that examples should be actual files such as documents, spreadsheets, presentations, images, or code repositories rather than summaries. There’s also an option to produce fabricated examples designed to demonstrate realistic personal responses to hypothetical scenarios.

In terms of security, OpenAI stipulates that submissions must be devoid of corporate intellectual property and personally sensitive data. Contractors are repeatedly reminded to ensure that the work examples reflect authentic job tasks they have completed. The instructions highlight the importance of purging all private, proprietary, and non-public information from these examples.

An illustrative task from the OpenAI presentation involves a Senior Lifestyle Manager from an elite concierge service. Their task was to draft a concise, two-page itinerary outlining a seven-day yacht excursion to the Bahamas for a family, with specific preferences and trip details included. Contractors, in this example, would use a genuine itinerary they crafted for a customer as their submission.

Legal experts express caution about the handling of proprietary information within these AI labs. Evan Brown, a lawyer specializing in intellectual property, highlights potential risks associated with inadvertent trade secret disclosure. Contractors may unknowingly breach prior non-disclosure agreements or expose sensitive company information, placing the AI firms at significant legal risk.

To meet the needs of advanced AI models, labs like OpenAI have engaged a significant number of contractors. These contractors are tasked with generating high-quality training data necessary to refine AI tools intended for automating professional tasks. This practice has given rise to a competitive sub-industry focusing on producing premier training datasets for AI development.

Historically, companies like Surge, Mercor, and Scale AI have facilitated the hiring and coordination of data contractors for AI labs. However, the demand for superior data has escalated, compelling these labs to invest more in talented individuals capable of meeting this requirement. This burgeoning demand has enriched the sector associated with AI data preparation.

Furthermore, OpenAI has explored additional methods for acquiring genuine company data. An individual involved in managing asset sales for defunct businesses reported receiving an inquiry from an OpenAI representative regarding the potential acquisition of such data sets. While assurances were made concerning the removal of any identifiable information, concerns about the thoroughness of this process discouraged any further action. This individual opted out of proceeding due to doubts over completely safeguarding personal data.