First there was ChatGPT, an artificial intelligence model with a seemingly uncanny ability to mimic human language. Introducing BloombergGPT, the first large-scale language model built specifically for the financial industry, created by Bloomberg.
Similar to ChatGPT and other popular language models recently introduced, this new AI system can create human-level text, answer questions, and complete various tasks, making it unique to the financial industry. It will be able to support various natural language processing tasks.
Mark Dreze, Associate Professor of Computer Science at Johns Hopkins University’s Whiting School of Engineering and Bloomberg Visiting Scholar, was part of the team that created the program. Dreze is also the founding Director of Research (Foundations of AI) for his new AI-X foundry at Johns Hopkins University.
The hub spoke with Doreze about the Bloomberg GPT and its far-reaching implications for AI research at Johns Hopkins University.
Image Caption: Mark Drezet
What were the goals of the BloombergGPT project?
Many of you have seen ChatGPT and other large language models. These are impressive new artificial intelligence technologies with great capabilities for processing language and responding to people’s requests. It is clear that these models have the potential to change society. To date, most models focus on general purpose use cases. But we also need domain-specific models that understand the complexities and nuances of specific domains. ChatGPT is great for many uses, but we need specialized models for medicine, science, and many other areas. It is not clear what the optimal strategy is for building these models.
In collaboration with Bloomberg, we explored this issue by building an English language model for the financial sector. We took a new approach, building a large dataset of financial-related texts and combining it with an equally large dataset of general-purpose texts. The resulting dataset is about 700 billion tokens, which is about 30 times the size of the entire Wikipedia text.
We trained a new model on this combined dataset and tested it across various linguistic tasks on financial documents. BloombergGPT was found to significantly outperform existing models of similar scale for financial tasks. Surprisingly, even though we aimed to build a domain-specific model, the model still performed on par with the general-purpose benchmark.
Why does finance need its own language model?
Recent advances in AI models are demonstrating exciting new applications in many fields, but given the complexity and unique terminology of the financial sector, sector-specific models are needed. This is similar to other disciplines such as medicine, which contain vocabulary not found in general-purpose texts. Finance-specific models can improve existing financial NLP tasks such as sentiment analysis, named entity recognition, news classification, and question answering. However, we also expect domain-specific models to create new opportunities.
For example, we envision BloombergGPT transforming natural language queries from financial professionals into Bloomberg Query Language (BQL), which is enabled. This BQL is a very powerful tool that allows financial professionals to quickly identify and manipulate data on various types of securities. So when the user asks “What is Apple’s last price and market cap?”, the system will get(px_last,cur_mkt_cap) for([‘AAPL US Equity’]). This set of code makes it quick and easy to import the resulting data into data science and portfolio management tools.
What did you learn while building the new model?
Building these models is not easy. A large number of details must be understood correctly in order for the model to work. We learned a lot from reading papers from other research groups that built language models. To serve the community, we wrote a 70+ page paper detailing how the dataset was constructed, the choices made to the model architecture, how the model was trained, and an extensive evaluation of the resulting model. I was. We have also released a detailed “Training Chronicle” that narratively describes the model training process. Our goal is to be as open as possible about how we built our models to support other research groups looking to build their own models.
what was your role?
This research was a collaboration between Bloomberg’s AI engineering team and the ML Products and Research Group in the company’s Chief Technology Office, where I am a visiting researcher. This was an intensive effort during which we regularly discussed data and model decisions and conducted detailed model evaluations. We read together every paper we could find on the subject to gain insights from other groups and often made decisions together.
The experience of observing the training of the model over several weeks was intense as we looked at multiple metrics of the model to best understand if it was working. It was a great team effort to put together the extensive evaluation and the paper itself. I am honored to be part of this wonderful group.
Was Johns Hopkins otherwise involved in this effort?
The team has a strong bond with John Hopkins. One of his engineers to lead this project is his Shijie Wu, who will be completing his PhD in 2021 at Johns Hopkins University. Additionally, Gideon Mann, who received his Ph.D. from Johns Hopkins University in 2006, was the leader of the team. I think this shows the tremendous value of a Johns Hopkins education that alumni continue to advance in their scientific fields long after they graduate.
How will Johns Hopkins University benefit from this initiative?
There is a great demand from students to learn how large language models work and how they can contribute to building language models. In the past year alone, the Computer Science Department at the Whiting School of Engineering has introduced three new courses that cover some of large-scale language models.
The latest advances in this area come from industry. Through my role on this industry team, I have gained important insight into how these models are built and evaluated. I incorporate these insights into my research and classrooms, giving my students a front row seat to researching these exciting models. I think the involvement of our faculty in these efforts speaks volumes about his AI leadership at Johns Hopkins.
How does this job relate to your role as research director at the new AI-X Foundry?
AI-X Foundry’s goal is to transform the way Johns Hopkins University conducts research through AI. Johns Hopkins researchers are among the world leaders in harnessing artificial intelligence to understand and improve the human condition. We recognize that a key part of this goal is strong collaboration between our faculty and AI industry leaders like Bloomberg. Building these relationships with AI-X Foundry will enable researchers to conduct truly innovative and cross-cutting AI research, while providing students with the best AI education possible. .
