Introducing PRODIGY: a pre-trained AI framework that enables in-context learning on graphs

The GPT model is the transformer architecture behind the famous chatbot developed by OpenAI called ChatGPT, which works on the concept of learning a task using just a few examples. This approach, called in-context learning, allows the model to learn how to perform well on different tasks without having to fine-tune the model with thousands of input texts, using only task-specific examples as input. will do so. Since GPT is a “large” language model with billions of parameters and every model parameter needs to be updated during fine-tuning, fine-tuning the model for a specific task can be relatively costly. You can see that it takes

Although in-context learning has been effectively used for code generation, question answering, machine translation, etc., it is still lacking and faces challenges for use in graph machine learning tasks. Some of the Graph machine learning tasks include identifying spreaders that spread semi-truths and false news on social networks and recommending products across e-commerce websites. In-context learning faces limitations in formulating and modeling these tasks on graphs with a unified task representation that allows models to tackle different tasks without retraining or parameter tuning. To do.

Recently, a team of researchers introduced PRODIGY, a pre-training framework that enables in-context learning on graphs, in a research paper. PRODIGY (Pretraining Over Diverse In-Context Graph Systems) uses a prompted graph representation to formulate in-context learning for graphs. A prompt graph serves as an in-context graph task representation that integrates the modeling of node, edge, and graph-level machine learning tasks. A prompt network connects input nodes or edges to additional label nodes to contextualize prompt examples and queries. This interconnected representation allows different graph machine learning tasks to be specified on the same model, regardless of graph size.

Proposed by researchers from Stanford University and the University of Ljubljana, the team designed a graph neural network architecture that is specifically tailored to handle prompted graphs and that effectively models and learns from graph-structured data. The proposed design makes use of GNNs to learn representations of the nodes and edges of the prompt graph. It also introduces a family of in-context pre-training targets to guide the learning process. This provides supervisory signals that allow the model to capture relevant graph patterns and generalize across different tasks.

To evaluate the performance and effectiveness of PRODIGY, the authors conducted experiments on tasks involving citation networks and knowledge graphs. A citation network represents the relationships between scientific papers, and a knowledge graph captures structured information about different disciplines. The pre-trained model was tested on these tasks using in-context learning, and the result was a contrasting pre-training base with hard-coded adaptation and standard fine-tuning with limited data. compared to the line. PRODIGY outperformed the contrasting pre-trained baseline with hard-coded adaptation by an average of 18% in terms of accuracy. When applying in-context learning, we achieved an average of 33% improvement compared to standard fine-tuning on limited data.

In conclusion, PRODIGY seems promising in graph-based scenarios such as in-context learning in graph machine learning applications. It can also perform downstream classification tasks on previously invisible graphs, making it even more effective and informative.

please check out paper. don’t forget to join 23,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email us. Asif@marktechpost.com

🚀 Check out 100’s of AI Tools at the AI ​​Tools Club