Publications

2023

1. 

   GEO: Generative Engine Optimization

   Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, and 3 more authors

   In , 2023

   Abstract Cite Paper Code Website

   The advent of large language models (LLMs) has ushered in a new paradigm of search engines that use generative models to gather and summarize information to answer user queries. This emerging technology, which we formalize under the unified framework of Generative Engines (GEs), has the potential to generate accurate and personalized responses, and is rapidly replacing traditional search engines like Google and Bing. Generative Engines typically satisfy queries by synthesizing information from multiple sources and summarizing them with the help of LLMs. While this shift significantly improves user utility and generative search engine traffic, it results in a huge challenge for the third stakeholder – website and content creators. Given the black-box and fast-moving nature of Generative Engines, content creators have little to no control over when and how their content is displayed. With generative engines here to stay, the right tools should be provided to ensure that creator economy is not severely disadvantaged. To address this, we introduce Generative Engine Optimization (GEO), a novel paradigm to aid content creators in improving the visibility of their content in Generative Engine responses through a black-box optimization framework for optimizing and defining visibility metrics. We facilitate systematic evaluation in this new paradigm by introducing GEO-bench, a benchmark of diverse user queries across multiple domains, coupled with sources required to answer these queries. Through rigorous evaluation, we show that GEO can boost visibility by up to 40% in generative engine responses. Moreover, we show the efficacy of these strategies varies across domains, underscoring the need for domain-specific methods. Our work opens a new frontier in the field of information discovery systems, with profound implications for generative engines and content creators.

   @inproceedings{Aggarwal2023GEOGE,
     title = {GEO: Generative Engine Optimization},
     author = {Aggarwal, Pranjal and Murahari, Vishvak and Rajpurohit, Tanmay and Kalyan, A. and Narasimhan, Karthik R and Deshpande, A.},
     year = {2023},
     url = {https://api.semanticscholar.org/CorpusID:264306222},
   }

2. 

   AutoMix: Automatically Mixing Language Models

   Aman Madaan*, Pranjal Aggarwal*, Ankit Anand, and 10 more authors

   ArXiv, 2023

   Abstract Cite Paper Code Website

   Large language models (LLMs) are now available in various sizes and configurations from cloud API providers. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present AutoMix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to AutoMix is a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring training. Given that verifications can be noisy, we employ a meta verifier in AutoMix to refine the accuracy of these assessments. Our experiments using LLAMA2-13/70B, on five context-grounded reasoning datasets demonstrate that AutoMix surpasses established baselines, improving the incremental benefit per cost by up to 89%.

   @article{Madaan2023AutoMixAM,
     title = {AutoMix: Automatically Mixing Language Models},
     author = {Madaan*, Aman and Aggarwal*, Pranjal and Anand, Ankit and Potharaju, Srividya Pranavi and Mishra, Swaroop and Zhou, Pei and Gupta, Aditya and Rajagopal, Dheeraj and Kappaganthu, Karthik and Yang, Yiming and Upadhyay, Shyam and Mausam and Faruqui, Manaal},
     journal = {ArXiv},
     year = {2023},
     volume = {abs/2310.12963},
     url = {https://api.semanticscholar.org/CorpusID:264306222},
   }

3. 

   Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs

   Pranjal Aggarwal, Aman Madaan, Yiming Yang, and 1 more author

   Emphirical Methods in Natural Language Processing, 2023

   Abstract Paper Code Website

   A popular approach for improving the correctness of output from large language models (LLMs) is Self-Consistency - poll the LLM multiple times and output the most frequent solution. Existing Self-Consistency techniques always generate a constant number of samples per question, where a better approach will be to non-uniformly distribute the available budget based on the amount of agreement in the samples generated so far. In response, we introduce Adaptive-Consistency, a cost-efficient, model-agnostic technique that dynamically adjusts the number of samples per question using a lightweight stopping criterion. Our experiments over 17 reasoning and code generation datasets and three LLMs demonstrate that Adaptive-Consistency reduces sample budget by up to 7.9 times with an average accuracy drop of less than 0.1%.

4. 

   SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification

   Pranjal Aggarwal, Ameet Deshpande, and Karthik Narasimhan

   International Conference on Machine Learning, 2023

   Abstract Cite Paper Code Website

   Extreme classification (XC) involves predicting over large numbers of classes (thousands to millions), with real-world applications like news article classification and e-commerce product tagging. The zero-shot version of this task requires generalization to novel classes without additional supervision. In this paper, we develop SemSup-XC, a model that achieves state-of-the-art zero-shot and few-shot performance on three XC datasets derived from legal, e-commerce, and Wikipedia data. To develop SemSup-XC, we use automatically collected semantic class descriptions to represent classes and facilitate generalization through a novel hybrid matching module that matches input instances to class descriptions using a combination of semantic and lexical similarity. Trained with contrastive learning, SemSup-XC significantly outperforms baselines and establishes state-of-the-art performance on all three datasets considered, gaining up to 12 precision points on zero-shot and more than 10 precision points on one-shot tests, with similar gains for recall@10. Our ablation studies highlight the relative importance of our hybrid matching module and automatically collected class descriptions.

   @article{aggarwal2023semsup,
     title = {SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification},
     author = {Aggarwal, Pranjal and Deshpande, Ameet and Narasimhan, Karthik},
     year = {2023},
     journal = {International Conference on Machine Learning}
   }

5. Computer Vision

   Deep learning for detection of iso-dense, obscure masses in mammographically dense breasts.

   Krithika Rangarajan*, Pranjal Aggarwal*, Dhruv Kumar Gupta, and 7 more authors

   European radiology, 2023

   Cite

   @article{Rangarajan2023DeepLF,
     title = {Deep learning for detection of iso-dense, obscure masses in mammographically dense breasts.},
     author = {Rangarajan*, Krithika and Aggarwal*, Pranjal and Gupta, Dhruv Kumar and Dhanakshirur, Rohan Raju and Baby, Akhil and Pal, Chandan and Gupta, Arunkumar and Hari, Smriti and Banerjee, Subhashis and Arora, Chetan},
     journal = {European radiology},
     year = {2023},
     paper = {https://link.springer.com/article/10.1007/s00330-023-09717-7},
     doi = {10.1007/s00330-023-09717-7},
   }