Building powerful AI models requires lots of good data, and Amazon goes to great lengths to collect this valuable information.
Amazon recently asked employees to sign up for and share their accounts with Microsoft's software development platform GitHub, a move seen by Business Insider, that will allow Amazon to collect data from the platform more quickly.
This marks an important step in Amazon's efforts to train future AI models it develops in-house.
In an internal memo shared with employees last month, Amazon's artificial intelligence group said it needed “quantitative and qualitative metadata from GitHub” for AI training.
But there's a problem: Each GitHub account can only perform 5,000 data collection requests per hour. According to the memo, GitHub is home to over 150 million public data repositories, so scraping all this information would take too long due to the account limit.
To get around this issue, the Amazon AGI team is asking employees to create new GitHub accounts and share them with the company, so Amazon can run them all simultaneously, cutting the time it takes to collect data to “a few weeks,” according to the memo.
“It would take years to capture all of this in a single account,” the memo explained. “To speed up the process of collecting metadata, we are asking team members to create GitHub accounts.[s] Share your API key.
Rohit Prasad, chief scientist at Amazon Noor Photo
Amazon management has openly asked employees for help with this workaround.
In an internal email titled “Helping with Data” in late May, Rohit Prasad, Amazon's chief scientist and SVP of the AGI group, said the company encouraged employees to share their GitHub accounts to “collect higher quality code data to help train our underlying models.”
Another email from Amazon AGI director urged employees, “It will only take 5 minutes!”
The episode highlights a ferocious hunger for data among technology companies developing their own AI models. To become more intelligent and human-like, these models require vast amounts of high-quality information. This information is in finite supply, which has sparked a “data war” among technology companies.
In Amazon's case, it needs more data to train a new, yet-to-be-released AI model that it's internally calling its “most ambitious” AI project. Launching new, more powerful AI models is important for Amazon as it tries to catch up with rivals Microsoft, Google and Meta in the field of generative AI.
Suspected license violation
GitHub's workaround will likely speed up Amazon's AI training process, but it could raise ethical concerns about accessing data without proper authorization.
Microsoft would likely be unhappy to learn that its biggest rival, Amazon, relies heavily on GitHub for its AI training data.
Microsoft itself is facing a lawsuit alleging it violated a licensing agreement when it used GitHub data to train its Copilot AI service.
“Amazon supports established legal frameworks that protect rights holders and content creators, and foster the development of innovative and beneficial services,” Amazon said in a statement. “Our law master's students are trained on data from a variety of sources, including licensed data, proprietary data, open-source datasets, and publicly available data where appropriate. This is an evolving field, but we follow industry best practices for data collection to train our models.”
The company also explained that it has created a system to “properly credit open source developers when generated code suggestions are similar to their projects.”
Spokespeople for GitHub and Microsoft did not respond to requests for comment.
“Show your hand”
Amazon wrote in an internal memo that GitHub's workaround was approved by both the company's legal and security teams.
Amazon said that by following the guidelines it was able to comply with GitHub's rate limits and avoid having its accounts blocked.
In terms of “revealing its hand”, Amazon's move “should not surprise anyone”, the memo said, adding that the company has multiple products in development simultaneously.
The memo said employees interested in collaborating should use their Amazon work emails, rather than personal accounts, when signing up for GitHub.
The company also said that Amazon employees must create a “classic personal token” when they sign up, rather than a “fine-grained personal token.” GitHub's classic personal tokens provide access to a broader range of code repositories, but may be less secure, according to the GitHub website.
Amazon's instructions also state that the expiration time for these tokens should be set to one year and that the “scope” should not be selected to give the token “read-only” access to public information.
Once signed up, Amazon employees must copy and paste their personal access token into a shared company file, the memo adds.
The “Most Scalable” Model
For Amazon, more data is essential for new AI models. Last year, Amazon CEO Andy Jassy wrote in an internal email that Prasad would lead a newly formed AGI team with the goal of building the company's “most scalable” large-scale language models. Prasad now reports directly to Jassy.
Amazon may be falling behind some AI competitors that have been engaged in big land grabs for years to gather more training data.
For example, OpenAI has signed a series of licensing agreements with a number of companies, including Reddit, Shutterstock, and News Corp, to use their content to train its AI models. Tech companies seeking more training data are also giving themselves new permission to use more of consumer information.
Meanwhile, Amazon's AGI team has already undergone a major reorganization, firing some employees who worked on Alexa-related projects in November, BI reported. Prasad also outlined six new focus areas for the AGI group at the time, including foundational models and conversational assistant services, BI previously reported.
Is it a difficult position?
While Amazon's legal team has approved GitHub's data scraping workaround, the move could put Amazon in a tricky position.
In 2022, programmer Matthew Butterick and law firm Joseph Saberi filed a class action lawsuit against GitHub's owner, Microsoft, for open source license violations. According to Saberi's website, Microsoft trained its Copilot AI service with publicly available code from GitHub without complying with “underlying open source licenses and other legal requirements.”
Open source code on GitHub is generally free to use but comes with certain obligations, such as maintaining accurate attribution to the source code, Butterick wrote on the lawsuit's website. Because Copilot is built on billions of lines of code from GitHub, it's nearly impossible to credit the original source, while Microsoft can sell it without giving anything back to the open source community, he wrote.
“Like Neo plugged into the Matrix or cows on a farm, Copilot seeks to turn us into mere producers of resources to be extracted (well, until we are culled altogether),” Butterick wrote. “And to what end? Even the cows get their food and shelter from this trade. Copilot has contributed nothing to our individual projects, and nothing to open source as a whole.”
Do you work at Amazon? Have any tips?
Encrypted messaging apps Signal or Telegram (+1-650-942-3061) or email (ekim@businessinsider.comStay in touch using non-work devices. Source Guide For more tips on sharing information safely, click here.
