The COVID-19 pandemic spurred a massive surge in scientific publications related to the virus and the disease. This rapid growth of information created a challenge for healthcare workers, researchers, and policymakers who needed to quickly access relevant and accurate information. Existing search engines were often inadequate for this task, highlighting the need for a specialized search engine designed to handle the complexities of the COVID-19 literature. This paper presents CO-Search, a novel semantic search engine designed to address this critical need. CO-Search aims to provide a reliable and efficient way to find scientific answers to complex questions related to COVID-19, thus potentially reducing the spread of misinformation during a time of crisis. The system indexes content from the COVID-19 Open Research Dataset (CORD-19), which contains over 400,000 scientific publications. The core goal is to offer a dedicated scientific search engine that minimizes the impact of misinformation.

The paper reviews existing COVID-19 search engines such as Neural Covidex, SLEDGE, and CovidQA, highlighting their architectures and limitations. It notes that existing systems often utilize deep learning models like BERT and SciBERT, fine-tuned on datasets such as MS MARCO, for query-document relevance prediction. The paper also mentions other related work, such as multi-document summarization systems and named entity recognition efforts focusing on the COVID-19 corpus. Existing systems often faced challenges due to the limited and rapidly evolving nature of the COVID-19 literature corpus. The authors emphasize the unique challenges posed by the COVID-19 literature compared to typical web-based search, such as the specialized terminology and rapid growth of the document collection.

CO-Search employs a cascaded retriever-ranker architecture. The retriever uses a hybrid approach combining semantic embeddings from Siamese-BERT (SBERT) with keyword-based methods (BM25 and TF-IDF). To address the limited size of the CORD-19 dataset, a text augmentation technique splits documents into paragraphs and their citations, generating millions of training tuples for SBERT. The retrieval step fuses the results of SBERT, TF-IDF, and BM25 using linear fusion and reciprocal rank fusion (RRF). The re-ranker further refines the ranking by incorporating outputs from a question-answering (QA) module and an abstractive summarization module. The QA module uses multi-hop reasoning to extract answer candidates from relevant paragraphs, while the summarizer generates concise summaries. The final ranking integrates the scores from the retriever, QA, and summarizer. SBERT is trained to learn correspondences between short text strings and longer documents, significantly improving efficiency compared to standard BERT. The TF-IDF and BM25 methods leverage keyword matching for enhanced retrieval. The re-ranking step uses a weighted combination of retrieval scores, QA output, and summarizer output to refine the final ranking.

CO-Search was evaluated on the TREC-COVID dataset, using metrics such as nDCG, P@N, MAP, and Bpref. The results demonstrate strong performance across all rounds of the competition. In the context of all submissions (including manual and feedback systems), CO-Search consistently ranked within the top 21. When compared strictly against automatic systems using only judged pairs, CO-Search achieved top-6 rankings across all metrics and rounds, and ranked first in half of the evaluations. An ablation study showed that the combination of SBERT, TF-IDF, and BM25 in the retriever was crucial for performance. The re-ranker, with its QA and summarization modules, further improved results. Comparison with top-performing automatic systems from Round 5 revealed that while CO-Search didn't outperform others across all metrics, it demonstrated competitive performance, suggesting the potential for hybrid systems combining the strengths of different approaches. The analysis of CO-Search's performance on specific topics revealed that its strength lies in queries with specific keywords and semantic nuances, while it performs less well on more general or less keyword-rich queries. The sparse nature of relevance judgments in the TREC-COVID dataset highlights the importance of using robust metrics like Bpref for evaluation.

CO-Search's strong performance on the TREC-COVID challenge demonstrates the effectiveness of its multi-stage architecture and the benefits of combining semantic and keyword-based approaches. The system's ability to handle complex queries and leverage both semantic understanding and keyword matching is key to its success. The study highlights the importance of data augmentation techniques for addressing limitations in training data size. The integration of QA and abstractive summarization modules significantly enhances the ranking quality. The findings suggest that hybrid systems combining various techniques could further improve retrieval performance. The limitations of the TREC-COVID evaluation dataset, such as sparse annotations, are acknowledged. Future work could involve further exploration of hybrid methods and potentially leveraging synthetic data generation for improved model training.

CO-Search offers a valuable tool for accessing and interpreting the complex and ever-growing body of research on COVID-19. The system's effectiveness in handling complex queries and mitigating misinformation makes it particularly relevant during public health crises. Future research directions could explore the integration of additional knowledge sources, improved methods for handling ambiguous queries, and more sophisticated ranking algorithms. The development of more comprehensive evaluation datasets would also be beneficial.

The study acknowledges limitations inherent in the TREC-COVID dataset, particularly the sparse availability of relevance judgments. This sparsity could affect the accuracy of certain evaluation metrics. The system's reliance on pre-trained models introduces potential biases from the training data. Future improvements could involve exploring methods to mitigate these biases and address the challenges posed by the ever-evolving nature of the COVID-19 literature.