0

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Introduces BIG-bench, a 200+ task collaborative benchmark spanning logic, social bias, code, and creative reasoning, contributed by 450+ authors.

Open
Publisher
Google Research
Year
2022
Venue
TMLR
ArXiv
arxiv.org/abs/2206.04615
URL
arxiv.org/abs/2206.04615
Code
github.com/google/BIG-bench
Authors
451
Hosting
External sourcelicense unknown

Cite

Notes

Only stored in your browser.

Attribution

Abstract & full text
arxiv.org/abs/2206.04615
Attribution policy →

Introduces 2 artifacts - 2 evals

TL;DR

Semantic Scholar

Evaluation of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters finds that model performance and calibration both improve with scale, but are poor in absolute terms.

Artifacts

2

Authors

451
Aarohi SrivastavaAbhinav RastogiAlethea PowerAlex RayAmanda AskellAndy ZouBarret ZophBill Yuchen LinDan HendrycksDanny HernandezDanqi ChenDavid DohanDeep GanguliElizabeth Barnes (Beth Barnes)Eric TangEthan DyerFrançois CholletGuy Gur-AriHannaneh HajishirziJackson KernionJacob HiltonJaehoon LeeJared KaplanJason WeiLudwig SchmidtMaarten BosmaMantas MazeikaMirac SuzgunNiklas MuennighoffOwain EvansPercy LiangRuslan SalakhutdinovSebastian GehrmannStella BidermanStephanie LinTao YuVedant MisraWilliam FedusWilliam SaundersXiang RenYejin ChoiYuntao BaiRui ZhangSam ShleiferLianhui QinKarl KrauthPeter ChangZirui WangAbubakar AbidJames ZouShubham ToshniwalElad SegalZijie J. WangChenlin MengColin RaffelNoah FiedelSwaroop MishraDerek ChenLeo GaoShixiang Shane GuAngela JiangNitish Shirish KeskarDaniel LevyLuke MetzGiambattista ParascandoloAndrea MadottoAdrià Garriga-AlonsoTushar KhotKyle RichardsonJiaming SongPaul Pu LiangJan KoconJesse EngelPascale FungJaime Fernández FisacChris Callison-BurchWilliam ZhangNayeon LeeHongming ZhangMohit BansalDiyi YangSamuel R. BowmanGaurav MishraAitor LewkowyczDan GarretteAbhishek RaoAbu Awal Md ShoebAdam FischAdam R. BrownAdam SantoroAditya GuptaAgnieszka KluskaAkshat AgarwalAlex WarstadtAlexander W. KocurekAli SafayaAli TazarvAlice XiangAlicia ParrishAllen NieAman HussainAmanda DsouzaAmbrose SloneAmeet RahaneAnantharaman S. IyerAnders AndreassenAndrea SantilliAndreas StuhlmüllerAndrew DaiAndrew LaAndrew LampinenAngelica ChenAnh VuongAnimesh GuptaAnna GottardiAntonio NorelliAnu VenkateshArash GholamidavoodiArfa TabassumArul MenezesArun KirubarajanAsher MullokandovAshish SabharwalAustin HerrickAvia EfratAykut ErdemAyla KarakaşB. Ryan RobertsBao Sheng LoeBartłomiej BojanowskiBatuhan ÖzyurtBehnam HedayatniaBehnam NeyshaburBenjamin IndenBenno SteinBerk EkmekciBlake HowaldBryan OrinionCameron DiaoCameron DourCatherine StinsonCedrick ArguetaCésar Ferri RamírezChandan SinghCharles RathkopfChitta BaralChiyu WuChris WaitesChristian VoigtChristopher D. ManningChristopher PottsCindy RamirezClara E. RiveraClemencia SiroCourtney AshcraftCristina GarbaceaDamien SileoDan KilmanDan RothDaniel FreemanDaniel KhashabiDaniel Moseguí GonzálezDanielle PerszykDaphne IppolitoDar GilboaDavid DrakardDavid JurgensDebajyoti DattaDenis EmelinDenis KleykoDeniz YuretDerek TamDieuwke HupkesDiganta MisraDilyar BuzanDimitri Coelho MolloDong-Ho LeeDylan SchraderEkaterina ShutovaEkin Dogus CubukEleanor HagermanElizabeth DonowayEllie PavlickEmanuele RodolaEmma LamEric ChuErkut ErdemErnie ChangEthan A. ChiEthan JerzakEthan KimEunice Engefu ManyasiEvgenii ZheltonozhskiiFanyue XiaFatemeh SiarFernando Martínez-PlumedFrancesca HappéFrieda RongGenta Indra WinataGerard de MeloGermán KruszewskiGiorgio MarianiGloria WangGonzalo Jaimovitch-LópezGregor BetzHana GalijasevicHannah KimHannah RashkinHarsh MehtaHayden BogarHenry ShevlinHinrich SchützeHiromu YakuraHugh Mee WongIan NgIsaac NobleJaap JumeletJack GeissingerJames B. SimonJames KoppelJames ZhengJana ThompsonJanelle WingfieldJarema RadomJascha Sohl-DicksteinJason PhangJason YosinskiJekaterina NovikovaJelle BosscherJennifer MarshJeremy KimJeroen TaalJesujoba AlabiJiacheng XuJillian TangJoan WaweruJohn BurdenJohn MillerJohn U. BalisJonathan BatchelderJonathan BerantJörg FrohbergJos RozenJose Hernandez-OralloJoseph BoudemanJoseph GuerrJoseph JonesJoshua B. TenenbaumJoshua S. RuleJoyce ChuaKamil KanclerzKaren LivescuKarthik GopalakrishnanKaterina IgnatyevaKatja MarkertKaustubh D. DholeKevin GimpelKevin OmondiKory MathewsonKristen ChiafulloKsenia ShkarutaKumar ShridharKyle McDonellLaria ReynoldsLi ZhangLiam DuganLidia Contreras-OchandoLouis-Philippe MorencyLuca MoschellaLucas LamLucy NobleLuheng HeLuis Oliveros ColónLütfi Kerem ŞenelMaarten SapMaartje ter HoeveMaheen FarooqiManaal FaruquiMarco BaturanMarco MarelliMarco MaruMaria Jose Ramírez QuintanaMarie TolkiehnMario GiulianelliMartha LewisMartin PotthastMatthew L. LeavittMatthias HagenMátyás SchubertMedina Orduna BaitemirovaMelody ArnaudMelvin McElrathMichael A. YeeMichael CohenMichael GuMichael IvanitskiyMichael StarrittMichael StrubeMichał SwędrowskiMichele BevilacquaMichihiro YasunagaMihir KaleMike CainMimee XuMitch WalkerMo TiwariMoin AminnaseriMor GevaMozhdeh GheiniMukund Varma TNanyun PengNathan A. ChiNeta Gur-Ari KrakoverNicholas CameronNicholas RobertsNick DoironNicole MartinezNikita NangiaNiklas DeckersNiveditha S. IyerNoah ConstantNuan WenOliver ZhangOmar AghaOmar ElbaghdadiOmer LevyPablo Antonio Moreno CasaresParth DoshiPaul VicolPegah AlipoormolabashiPeiyuan LiaoPeter EckersleyPhu Mon HtutPinyu HwangPiotr MiłkowskiPiyush PatilPouya PezeshkpourPriti OliQiaozhu MeiQing LyuQinlang ChenRabin BanjadeRachel Etta RudolphRaefer GabrielRahel HabackerRamon RiscoRaphaël MillièreRhythm GargRichard BarnesRif A. SaurousRiku ArakawaRobbe RaymaekersRobert FrankRohan SikandRoman NovakRoman SitelewRonan LeBrasRosanne LiuRowan JacobsRyan ChiRyan LeeRyan StovallRyan TeehanRylan YangSahib SinghSaif M. MohammadSajant AnandSam DillavouSam WisemanSamuel GruetterSamuel S. SchoenholzSanghyun HanSanjeev KwatraSarah A. RousSarik GhazarianSayan GhoshSean CaseySebastian BischoffSebastian SchusterSepideh SadeghiShadi HamdanSharon ZhouShashank SrivastavaSherry ShiShikhar SinghShima AsaadiShubh PachchigarShyam UpadhyayShyamolimaDebnathSiamak ShakeriSimon ThormeyerSimone MelziSiva ReddySneha Priscilla MakiniSoo-Hwan LeeSpencer ToreneSriharsha HatwarStanislas DehaeneStefan DivicStefano ErmonStephen PrasadSteven T. PiantadosiStuart M. ShieberSummer MisherghiSvetlana KiritchenkoTal LinzenTal SchusterTao LiTariq AliTatsu HashimotoTe-Lin WuThéo DesbordesTheodore RothschildThomas PhanTianle WangTiberius NkinyiliTimo SchickTimofei KornevTitus TundunyTobias GerstenbergTrenton ChangTrishala NeerajTyler ShultzUri ShahamVera DembergVictoria NyamaiVikas RaunakVinay RamaseshVinay Uday PrabhuVishakh PadmakumarVivek SrikumarWout VossenXiaoyu TongXinran ZhaoXinyi WuXudong ShenYadollah YaghoobzadehYair LakretzYangqiu SongYasaman BahriYichi YangYiding HaoYifu ChenYonatan BelinkovYu HouYufang HouZachary SeidZhuoye ZhaoZijian WangZiyi Wu