Jivi.AI, a Gurugram-based startup leveraging generative AI to transform primary healthcare for people in India, has seen its medical LLM JiviMedX-8B rise to the top of Hugging Face’ Open Medical-LLM Leaderboard, outperforming OpenAI and Google’s Med-PaLM 2, with 91.65%. The leaderboard offers a comprehensive assessment of each model’s medical knowledge and question answering capabilities.

JiviMed-8B builds upon the frameworks of the Meta-Llama-3-8B models, integrating a specially curated and diverse medical dataset. Over 100,000 data points were curated to ensure the model is well-versed in the nuances of biomedical language.

The Open Medical LLM Leaderboard aims to track, rank and evaluate the performance of LLMs on medical question answering tasks. It evaluates LLMs across a diverse array of medical datasets, including MedQA (USMLE), PubMedQA, MedMCQA, and subsets of MMLU related to medicine and biology.

The datasets cover various aspects of medicine such as general medical knowledge, clinical knowledge, anatomy, genetics, and more. They contain multiple-choice and open-ended questions that require medical reasoning and understanding.