logo
ResearchBunny Logo
Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data

Computer Science

Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data

C. H. Martin, T. (. Peng, et al.

Discover groundbreaking insights from Charles H. Martin, Tongsu (Serena) Peng, and Michael W. Mahoney as they tackle the daunting challenge of evaluating pre-trained neural network models without any access to training data. Their research reveals that power law-based metrics significantly outperform traditional measures in distinguishing model quality and uncovering hidden issues.

00:00
00:00
Playback language: English
Abstract
This paper addresses the challenge of evaluating the quality of pre-trained neural network models without access to training or testing data. The authors conduct a meta-analysis of hundreds of publicly available pre-trained models from computer vision and natural language processing, examining norm-based and power law-based metrics. They find that power law-based metrics are superior in discriminating well-trained from poorly trained models and in identifying model problems not detectable through training/test accuracies alone.
Publisher
Nature Communications
Published On
Jul 05, 2021
Authors
Charles H. Martin, Tongsu (Serena) Peng, Michael W. Mahoney
Tags
neural networks
model evaluation
pre-trained models
power law metrics
computer vision
natural language processing

Related Publications

Explore these studies to deepen your understanding of the subject.

Evaluating the effectiveness of the Kidogo model in empowering women and strengthening their capacities to engage in paid labor opportunities through the provision of quality childcare: a study protocol for an exploratory study in Nakuru County, Kenya
Social Work

Evaluating the effectiveness of the Kidogo model in empowering women and strengthening their capacities to engage in paid labor opportunities through the provision of quality childcare: a study protocol for an exploratory study in Nakuru County, Kenya

K. Okelo, M. Nampijja, et al.

The impact of COVID-19 on digital data practices in museums and art galleries in the UK and the US
The Arts

The impact of COVID-19 on digital data practices in museums and art galleries in the UK and the US

L. Noehrer, A. Gilmore, et al.

Analysing inter-state communication dynamics and roles in the networks of the International Institute of Intellectual Cooperation
Humanities

Analysing inter-state communication dynamics and roles in the networks of the International Institute of Intellectual Cooperation

R. Rodríguez-casañ, E. Carbó-catalan, et al.

The ability of different compositions of calcium silicate and epoxy sealers to withstand gutta percha removal via in vitro pull-out testing
Medicine and Health

The ability of different compositions of calcium silicate and epoxy sealers to withstand gutta percha removal via in vitro pull-out testing

I. Stiklaru, E. Lalum, et al.

Listen, Learn & Level Up
Over 10,000 hours of research content in 25+ fields, available in 12+ languages.
No more digging through PDFs, just hit play and absorb the world's latest research in your language, on your time.
listen to research audio papers with researchbunny