Tokyo Institute of Technology, Tohoku University, Fujitsu Limited, RIKEN
Tokyo, May 22, 2023
Tokyo Institute of Technology (Tokyo Tech), Tohoku University, Fujitsu Limited, and RIKEN today announced that they will embark on research and development for distributed training of large-scale language models (LLMs). (1) It will be put into practical use in the supercomputer “Fugaku” in May 2023 within the scope of “Fugaku” utilization efforts stipulated by national policy.
LLM is an AI model for deep learning that is the core of generative AI such as ChatGPT. (2). The four organizations will develop an environment for creating LLMs that can be widely used by academia and companies, and by disclosing the results, contribute to improving Japan’s AI research capabilities and increase the utility value of Fugaku in the academic and industrial fields. Aim for This research and development in the future.
Background
While many expect LLM and generative AI to play a fundamental role in the research and development of technologies for security, the economy, and society at large, progress and refinement of these models will require large-scale You need high-performance computing resources that can process your data efficiently. amount of data.
Tokyo Institute of Technology, Tohoku University, Fujitsu, and RIKEN are working to focus on R&D towards distributed training for LLM.
Implementation period
From May 24, 2023 to March 31, 2024 *Effort period to utilize Fugaku in Japanese policy
Role of each organization/company
The technology used in this project will enable efficient training of large-scale language models on the large-scale parallel computing environment of the supercomputer Fugaku. The roles of each organization and company are as follows.
Tokyo Institute of Technology: Monitoring of the entire process, parallelization and acceleration of LLM
Tohoku University: Collection of learning data, selection of models
Fujitsu: Accelerating LLM
RIKEN: Distributed parallelization and communication speedup of LLM, LLM speedup
Future Plans
In the future, in order to support LLM development by Japanese researchers and engineers, the four organizations plan to publish research results obtained within the scope of efforts to use Fugaku as stipulated by Japanese policy on GitHub. (3) hugging face (Four) In addition, many researchers and engineers will participate in the improvement of the basic model and new applied research, and it is expected that efficient methods will be created that will lead to next-generation innovative research and business results.
The four organizations are also collaborating with Nagoya University, which develops data generation and learning methods for multimodal applications in industrial fields such as manufacturing, and CyberAgent, Inc., which provides data and technology for building LLMs. We will consider.
Comment from Toshio Endo, Professor, Global Science Information Center, Tokyo Institute of Technology:
“In this collaboration, Tokyo Institute of Technology and RIKEN will parallelize and accelerate a large-scale language model using the supercomputer Fugaku, while Fujitsu will develop high-performance computing platform software for Fugaku and perform AI model performance tuning. , and integrated natural language processing by Tohoku University. technology. In collaboration with Fujitsu, we will also utilize the small-scale laboratory established in 202X under the name of the “Fujitsu Next-Generation Computing Infrastructure Collaborative Research Center.” We would like to contribute to the improvement of Japan’s AI research capabilities together with everyone by utilizing the large-scale distributed deep learning function provided by “Fugaku”. ”
Comment from Professor Kentaro Inui, Graduate School of Information Science and Technology, Tohoku University:
“We aim to build large-scale language models that are open-source, commercially available, and primarily based on Japanese data, with transparency of training data. Traceability of training data. By enabling , we hope to promote research robust enough to scientifically examine issues related to common AI black-box problems, bias, misinformation, and the so-called ‘hallucination’ phenomenon. will be We will build a large-scale model using the knowledge obtained from deep learning of Japanese natural language processing developed at Tohoku University. We look forward to sharing the research results obtained through this effort with researchers and developers and contributing to the improvement of our domestic and international AI research capabilities. ”
Comment from Seiji Okamoto, Executive Officer, General Manager of Fujitsu Research Division, Fujitsu Limited:
“We are excited about the opportunity to leverage the powerful parallel computing resources of the supercomputer Fugaku to enhance our research in AI and advance the research and development of LLMS. We aim to provide applications that cause a paradigm shift that contributes to the realization of a sustainable society. ”
Comment from Satoshi Matsuoka, Director of the RIKEN Center for Computational Science:
“A64FX” (Five) The CPU is equipped with an AI acceleration function called SVE.
However, software development and optimization are essential to maximize its capabilities and exploit them for AI applications. This collaboration brings together his LLM and computer science experts in Japan, including researchers and engineers from RIKEN R-CCS, to develop the technology to build his LLM on the supercomputer Fugaku. I believe that it will play an important role in Together with our collaborators, we will contribute to the realization of Society 5.0. ”
Project name
Distributed Training of Large Language Models in Fugaku (Project number: hp230254)
About Tokyo Institute of Technology
Tokyo Institute of Technology is at the forefront of research and higher education as Japan’s leading science and technology university. Tokyo Tech researchers excel in fields ranging from materials science to biology to computer science to physics. Founded in 1881, Tokyo Tech enrolls more than 10,000 undergraduate and graduate students each year who grow to become scientific leaders and the industry’s most sought-after engineers. Embodying the Japanese philosophy of monozukuri, which means “technical ingenuity and innovation,” the Tokyo Tech community strives to contribute to society through impactful research.
https://www.titech.ac.jp/english/
About Tohoku University
Tohoku University has 10 faculties, 15 graduate schools, 6 research institutes, and 18,000 students. About 10% of our students come from abroad, contributing to one of the most international academic environments in Japan. Tohoku University was granted Designated National University status by the Japanese government in June 2017 for its excellent learning environment, international outlook, and influence on research. It has also held the #1 spot in Times Higher magazine for the past four years. Japanese university rankings published annually by Education. A list focused on institutional resources, educational quality and the overall student experience.
About Fujitsu
Fujitsu’s purpose is to build trust in society through innovation and make the world more sustainable. As the digital transformation partner of choice for customers in over 100 countries, our 124,000 employees are committed to solving some of the greatest challenges facing humanity. Our wide range of services and solutions leverages his five key technologies: Computing, Networks, AI, Data & Security, and Converging Technologies, which combine to deliver a sustainable transformation. Fujitsu Limited (TSE: 6702) reports consolidated sales of ¥3.7 trillion (US$28 billion) for the fiscal year ending March 31, 2023, still trailing Japan in market share. At the top of the digital services company. For more information, please visit www.Fujitsu.com.
About RIKEN
RIKEN Center for Computational Science (R-CCS) is a leadership center for high-performance computing that explores “the science of computing, by computing, for computing’s sake.” As a result of its exploration, technologies such as open source software are its core competencies. R-CCS strives to strengthen its core competencies and spread its technology around the world. Together with Fujitsu, R-CCS has developed the world’s most powerful supercomputer, Fugaku. Fugaku started full-scale operation in March 2021, with an order of magnitude improvement in computing power and synergies with other IT ecosystems such as big data and artificial intelligence. In the past, R-CCS operated the K computer (2012-2019), and brought many world-leading scientific and technological achievements not only to academia but also to industry.
press contact
Tokyo Institute of Technology
Public Relations Division, Tokyo Institute of Technology
Email: media@jim.titech.ac.jp
Phone number: 03-5734-2975
Fujitsu Limited
Public Relations/Investor Relations Department
inquiry
All company or product names mentioned herein are trademarks or registered trademarks of their respective owners. The information contained in this press release is correct at the time of publication and is subject to change without notice.
date: May 22, 2023
City: Tokyo Japan
company: Tokyo Institute of Technology, Tohoku University, Fujitsu Limited, RIKEN
