학술논문
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Document Type
Working Paper
Author
Reid, Machel; Savinov, Nikolay; Teplyashin, Denis; Lepikhin, Dmitry; Lillicrap, Timothy; Alayrac, Jean-baptiste; Soricut, Radu; Lazaridou, Angeliki; Firat, Orhan; Schrittwieser, Julian; Antonoglou, Ioannis; Anil, Rohan; Borgeaud, Sebastian; Dai, Andrew; Millican, Katie; Dyer, Ethan; Glaese, Mia; Sottiaux, Thibault; Lee, Benjamin; Viola, Fabio; Reynolds, Malcolm; Xu, Yuanzhong; Molloy, James; Chen, Jilin; Isard, Michael; Barham, Paul; Hennigan, Tom; McIlroy, Ross; Johnson, Melvin; Schalkwyk, Johan; Collins, Eli; Rutherford, Eliza; Moreira, Erica; Ayoub, Kareem; Goel, Megha; Meyer, Clemens; Thornton, Gregory; Yang, Zhen; Michalewski, Henryk; Abbas, Zaheer; Schucher, Nathan; Anand, Ankesh; Ives, Richard; Keeling, James; Lenc, Karel; Haykal, Salem; Shakeri, Siamak; Shyam, Pranav; Chowdhery, Aakanksha; Ring, Roman; Spencer, Stephen; Sezener, Eren; Vilnis, Luke; Chang, Oscar; Morioka, Nobuyuki; Tucker, George; Zheng, Ce; Woodman, Oliver; Attaluri, Nithya; Kocisky, Tomas; Eltyshev, Evgenii; Chen, Xi; Chung, Timothy; Selo, Vittorio; Brahma, Siddhartha; Georgiev, Petko; Slone, Ambrose; Zhu, Zhenkai; Lottes, James; Qiao, Siyuan; Caine, Ben; Riedel, Sebastian; Tomala, Alex; Chadwick, Martin; Love, Juliette; Choy, Peter; Mittal, Sid; Houlsby, Neil; Tang, Yunhao; Lamm, Matthew; Bai, Libin; Zhang, Qiao; He, Luheng; Cheng, Yong; Humphreys, Peter; Li, Yujia; Brin, Sergey; Cassirer, Albin; Miao, Yingjie; Zilka, Lukas; Tobin, Taylor; Xu, Kelvin; Proleev, Lev; Sohn, Daniel; Magni, Alberto; Hendricks, Lisa Anne; Gao, Isabel; Ontañón, Santiago; Bunyan, Oskar; Byrd, Nathan; Sharma, Abhanshu; Zhang, Biao; Pinto, Mario; Sinha, Rishika; Mehta, Harsh; Jia, Dawei; Caelles, Sergi; Webson, Albert; Morris, Alex; Roelofs, Becca; Ding, Yifan; Strudel, Robin; Xiong, Xuehan; Ritter, Marvin; Dehghani, Mostafa; Chaabouni, Rahma; Karmarkar, Abhijit; Lai, Guangda; Mentzer, Fabian; Xu, Bibo; Li, YaGuang; Zhang, Yujing; Paine, Tom Le; Goldin, Alex; Neyshabur, Behnam; Baumli, Kate; Levskaya, Anselm; Laskin, Michael; Jia, Wenhao; Rae, Jack W.; Xiao, Kefan; He, Antoine; Giordano, Skye; Yagati, Lakshman; Lespiau, Jean-Baptiste; Natsev, Paul; Ganapathy, Sanjay; Liu, Fangyu; Martins, Danilo; Chen, Nanxin; Xu, Yunhan; Barnes, Megan; May, Rhys; Vezer, Arpi; Oh, Junhyuk; Franko, Ken; Bridgers, Sophie; Zhao, Ruizhe; Wu, Boxi; Mustafa, Basil; Sechrist, Sean; Parisotto, Emilio; Pillai, Thanumalayan Sankaranarayana; Larkin, Chris; Gu, Chenjie; Sorokin, Christina; Krikun, Maxim; Guseynov, Alexey; Landon, Jessica; Datta, Romina; Pritzel, Alexander; Thacker, Phoebe; Yang, Fan; Hui, Kevin; Hauth, Anja; Yeh, Chih-Kuan; Barker, David; Mao-Jones, Justin; Austin, Sophia; Sheahan, Hannah; Schuh, Parker; Svensson, James; Jain, Rohan; Ramasesh, Vinay; Briukhov, Anton; Chung, Da-Woon; von Glehn, Tamara; Butterfield, Christina; Jhakra, Priya; Wiethoff, Matthew; Frye, Justin; Grimstad, Jordan; Changpinyo, Beer; Lan, Charline Le; Bortsova, Anna; Wu, Yonghui; Voigtlaender, Paul; Sainath, Tara; Smith, Charlotte; Hawkins, Will; Cao, Kris; Besley, James; Srinivasan, Srivatsan; Omernick, Mark; Gaffney, Colin; Surita, Gabriela; Burnell, Ryan; Damoc, Bogdan; Ahn, Junwhan; Brock, Andrew; Pajarskas, Mantas; Petrushkina, Anastasia; Noury, Seb; Blanco, Lorenzo; Swersky, Kevin; Ahuja, Arun; Avrahami, Thi; Misra, Vedant; de Liedekerke, Raoul; Iinuma, Mariko; Polozov, Alex; York, Sarah; Driessche, George van den; Michel, Paul; Chiu, Justin; Blevins, Rory; Gleicher, Zach; Recasens, Adrià; Rrustemi, Alban; Gribovskaya, Elena; Roy, Aurko; Gworek, Wiktor; Arnold, Séb; Lee, Lisa; Lee-Thorp, James; Maggioni, Marcello; Piqueras, Enrique; Badola, Kartikeya; Vikram, Sharad; Gonzalez, Lucas; Baddepudi, Anirudh; Senter, Evan; Devlin, Jacob; Qin, James; Azzam, Michael; Trebacz, Maja; Polacek, Martin; Krishnakumar, Kashyap; Chang, Shuo-yiin; Tung, Matthew; Penchev, Ivo; Joshi, Rishabh; Olszewska, Kate; Muir, Carrie; Wirth, Mateo; Hartman, Ale Jakse; Newlan, Josh; Kashem, Sheleem; Bolina, Vijay; Dabir, Elahe; van Amersfoort, Joost; Ahmed, Zafarali; Cobon-Kerr, James; Kamath, Aishwarya; Hrafnkelsson, Arnar Mar; Hou, Le; Mackinnon, Ian; Frechette, Alexandre; Noland, Eric; Si, Xiance; Taropa, Emanuel; Li, Dong; Crone, Phil; Gulati, Anmol; Cevey, Sébastien; Adler, Jonas; Ma, Ada; Silver, David; Tokumine, Simon; Powell, Richard; Lee, Stephan; Chang, Michael; Hassan, Samer; Mincu, Diana; Yang, Antoine; Levine, Nir; Brennan, Jenny; Wang, Mingqiu; Hodkinson, Sarah; Zhao, Jeffrey; Lipschultz, Josh; Pope, Aedan; Chang, Michael B.; Li, Cheng; Shafey, Laurent El; Paganini, Michela; Douglas, Sholto; Bohnet, Bernd; Pardo, Fabio; Odoom, Seth; Rosca, Mihaela; Santos, Cicero Nogueira dos; Soparkar, Kedar; Guez, Arthur; Hudson, Tom; Hansen, Steven; Asawaroengchai, Chulayuth; Addanki, Ravi; Yu, Tianhe; Stokowiec, Wojciech; Khan, Mina; Gilmer, Justin; Lee, Jaehoon; Bostock, Carrie Grimes; Rong, Keran; Caton, Jonathan; Pejman, Pedram; Pavetic, Filip; Brown, Geoff; Sharma, Vivek; Lučić, Mario; Samuel, Rajkumar; Djolonga, Josip; Mandhane, Amol; Sjösund, Lars Lowe; Buchatskaya, Elena; White, Elspeth; Clay, Natalie; Jiang, Jiepu; Lim, Hyeontaek; Hemsley, Ross; Labanowski, Jane; De Cao, Nicola; Steiner, David; Hashemi, Sayed Hadi; Austin, Jacob; Gergely, Anita; Blyth, Tim; Stanton, Joe; Shivakumar, Kaushik; Siddhant, Aditya; Andreassen, Anders; Araya, Carlos; Sethi, Nikhil; Shivanna, Rakesh; Hand, Steven; Bapna, Ankur; Khodaei, Ali; Miech, Antoine; Tanzer, Garrett; Swing, Andy; Thakoor, Shantanu; Pan, Zhufeng; Nado, Zachary; Winkler, Stephanie; Yu, Dian; Saleh, Mohammad; Maggiore, Loren; Barr, Iain; Giang, Minh; Kagohara, Thais; Danihelka, Ivo; Marathe, Amit; Feinberg, Vladimir; Elhawaty, Mohamed; Ghelani, Nimesh; Horgan, Dan; Miller, Helen; Walker, Lexi; Tanburn, Richard; Tariq, Mukarram; Shrivastava, Disha; Xia, Fei; Chiu, Chung-Cheng; Ashwood, Zoe; Baatarsukh, Khuslen; Samangooei, Sina; Alcober, Fred; Stjerngren, Axel; Komarek, Paul; Tsihlas, Katerina; Boral, Anudhyan; Comanescu, Ramona; Chen, Jeremy; Liu, Ruibo; Bloxwich, Dawn; Chen, Charlie; Sun, Yanhua; Feng, Fangxiaoyu; Mauger, Matthew; Dotiwalla, Xerxes; Hellendoorn, Vincent; Sharman, Michael; Zheng, Ivy; Haridasan, Krishna; Barth-Maron, Gabe; Swanson, Craig; Rogozińska, Dominika; Andreev, Alek; Rubenstein, Paul Kishan; Sang, Ruoxin; Hurt, Dan; Elsayed, Gamaleldin; Wang, Renshen; Lacey, Dave; Ilić, Anastasija; Zhao, Yao; Aroyo, Lora; Iwuanyanwu, Chimezie; Nikolaev, Vitaly; Lakshminarayanan, Balaji; Jazayeri, Sadegh; Kaufman, Raphaël Lopez; Varadarajan, Mani; Tekur, Chetan; Fritz, Doug; Khalman, Misha; Reitter, David; Dasgupta, Kingshuk; Sarcar, Shourya; Ornduff, Tina; Snaider, Javier; Huot, Fantine; Jia, Johnson; Kemp, Rupert; Trdin, Nejc; Vijayakumar, Anitha; Kim, Lucy; Angermueller, Christof; Lao, Li; Liu, Tianqi; Zhang, Haibin; Engel, David; Greene, Somer; White, Anaïs; Austin, Jessica; Taylor, Lilly; Ashraf, Shereen; Liu, Dangyi; Georgaki, Maria; Cai, Irene; Kulizhskaya, Yana; Goenka, Sonam; Saeta, Brennan; Vodrahalli, Kiran; Frank, Christian; de Cesare, Dario; Robenek, Brona; Richardson, Harry; Alnahlawi, Mahmoud; Yew, Christopher; Ponnapalli, Priya; Tagliasacchi, Marco; Korchemniy, Alex; Kim, Yelin; Li, Dinghua; Rosgen, Bill; Levin, Kyle; Wiesner, Jeremy; Banzal, Praseem; Srinivasan, Praveen; Yu, Hongkun; Ünlü, Çağlar; Reid, David; Tung, Zora; Finchelstein, Daniel; Kumar, Ravin; Elisseeff, Andre; Huang, Jin; Zhang, Ming; Zhu, Rui; Aguilar, Ricardo; Giménez, Mai; Xia, Jiawei; Dousse, Olivier; Gierke, Willi; Yeganeh, Soheil Hassas; Yates, Damion; Jalan, Komal; Li, Lu; Latorre-Chimoto, Eri; Nguyen, Duc Dung; Durden, Ken; Kallakuri, Praveen; Liu, Yaxin; Johnson, Matthew; Tsai, Tomy; Talbert, Alice; Liu, Jasmine; Neitz, Alexander; Elkind, Chen; Selvi, Marco; Jasarevic, Mimi; Soares, Livio Baldini; Cui, Albert; Wang, Pidong; Wang, Alek Wenjiao; Ye, Xinyu; Kallarackal, Krystal; Loher, Lucia; Lam, Hoi; Broder, Josef; Holtmann-Rice, Dan; Martin, Nina; Ramadhana, Bramandia; Toyama, Daniel; Shukla, Mrinal; Basu, Sujoy; Mohan, Abhi; Fernando, Nick; Fiedel, Noah; Paterson, Kim; Li, Hui; Garg, Ankush; Park, Jane; Choi, DongHyun; Wu, Diane; Singh, Sankalp; Zhang, Zhishuai; Globerson, Amir; Yu, Lily; Carpenter, John; Quitry, Félix de Chaumont; Radebaugh, Carey; Lin, Chu-Cheng; Tudor, Alex; Shroff, Prakash; Garmon, Drew; Du, Dayou; Vats, Neera; Lu, Han; Iqbal, Shariq; Yakubovich, Alex; Tripuraneni, Nilesh; Manyika, James; Qureshi, Haroon; Hua, Nan; Ngani, Christel; Raad, Maria Abi; Forbes, Hannah; Bulanova, Anna; Stanway, Jeff; Sundararajan, Mukund; Ungureanu, Victor; Bishop, Colton; Li, Yunjie; Venkatraman, Balaji; Li, Bo; Thornton, Chloe; Scellato, Salvatore; Gupta, Nishesh; Wang, Yicheng; Tenney, Ian; Wu, Xihui; Shenoy, Ashish; Carvajal, Gabriel; Wright, Diana Gage; Bariach, Ben; Xiao, Zhuyun; Hawkins, Peter; Dalmia, Sid; Farabet, Clement; Valenzuela, Pedro; Yuan, Quan; Welty, Chris; Agarwal, Ananth; Chen, Mia; Kim, Wooyeol; Hulse, Brice; Dukkipati, Nandita; Paszke, Adam; Bolt, Andrew; Davoodi, Elnaz; Choo, Kiam; Beattie, Jennifer; Prendki, Jennifer; Vashisht, Harsha; Santamaria-Fernandez, Rebeca; Cobo, Luis C.; Wilkiewicz, Jarek; Madras, David; Elqursh, Ali; Uy, Grant; Ramirez, Kevin; Harvey, Matt; Liechty, Tyler; Zen, Heiga; Seibert, Jeff; Hu, Clara Huiyi; Khorlin, Andrey; Le, Maigo; Aharoni, Asaf; Li, Megan; Wang, Lily; Kumar, Sandeep; Lince, Alejandro; Casagrande, Norman; Hoover, Jay; Badawy, Dalia El; Soergel, David; Vnukov, Denis; Miecnikowski, Matt; Simsa, Jiri; Koop, Anna; Kumar, Praveen; Sellam, Thibault; Vlasic, Daniel; Daruki, Samira; Shabat, Nir; Zhang, John; Su, Guolong; Zhang, Jiageng; Liu, Jeremiah; Sun, Yi; Palmer, Evan; Ghaffarkhah, Alireza; Xiong, Xi; Cotruta, Victor; Fink, Michael; Dixon, Lucas; Sreevatsa, Ashwin; Goedeckemeyer, Adrian; Dimitriev, Alek; Jafari, Mohsen; Crocker, Remi; FitzGerald, Nicholas; Kumar, Aviral; Ghemawat, Sanjay; Philips, Ivan; Liu, Frederick; Liang, Yannie; Sterneck, Rachel; Repina, Alena; Wu, Marcus; Knight, Laura; Georgiev, Marin; Lee, Hyo; Askham, Harry; Chakladar, Abhishek; Louis, Annie; Crous, Carl; Cate, Hardie; Petrova, Dessie; Quinn, Michael; Owusu-Afriyie, Denese; Singhal, Achintya; Wei, Nan; Kim, Solomon; Vincent, Damien; Nasr, Milad; Choquette-Choo, Christopher A.; Tojo, Reiko; Lu, Shawn; Casas, Diego de Las; Cheng, Yuchung; Bolukbasi, Tolga; Lee, Katherine; Fatehi, Saaber; Ananthanarayanan, Rajagopal; Patel, Miteyan; Kaed, Charbel; Li, Jing; Sygnowski, Jakub; Belle, Shreyas Rammohan; Chen, Zhe; Konzelmann, Jaclyn; Põder, Siim; Garg, Roopal; Koverkathu, Vinod; Brown, Adam; Dyer, Chris; Liu, Rosanne; Nova, Azade; Xu, Jun; Petrov, Slav; Hassabis, Demis; Kavukcuoglu, Koray; Dean, Jeffrey; Vinyals, Oriol
Source
Subject
Language
Abstract
In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks across modalities, improves the state-of-the-art in long-document QA, long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.