Abacus.AI Explores Expanding Contexts for AI Models | Chatgpt Cos'è | Chatgpt Download | Llm Chatgpt | Turtles AI

  #AI #startup #Abacus.AI explores expanding #context length for large language models (LLM) without additional #training. New paper introduces #Giraffe models finetuned from #LLaMA and #LLaMA2. Aims to enable tasks needing more recall without scaling challenges. But evaluations show performance still degrades with longer contexts.  Researchers at AI startup Abacus.AI have released a new paper exploring techniques to expand the context length capabilities of large language models (LLMs). The paper introduces a new family of LLMs called Giraffe that are finetuned from existing models LLaMA and LLaMA2.  The work aims to enable LLMs to handle longer input contexts without additional training on long contexts. This could allow LLMs to complete tasks requiring recall across larger bodies of text, like conversing with long histories or coding assistance for large codebases.  A core challenge is that the self-attention mechanism in LLMs scales poorly as context length grows. Abacus.AI tested various proposed techniques for mitigating this, plus a new truncation method. Their evaluations suggest performance still degrades gradually with longer contexts.  The paper emphasizes new evaluation tasks focused on accuracy over just coherence. These include question answering over Wikipedia and extending a dialogue task to longer contexts. Abacus.AI believes metrics like perplexity are poor indicators of context length capability.  The research contributes open-sourced code, models and datasets. But substantially improving context length extrapolation remains an open problem. Abacus.AI plans to continue working in this area.   Highlights:

• - Abacus.AI paper introduces Giraffe models finetuned from LLaMA and LLaMA2
• - Aims to expand context length capabilities without scaling challenges
• - Evaluations show performance still degrades gradually with longer contexts
• - Emphasizes new evaluation tasks focused on accuracy over coherence

This research from Abacus.AI highlights the lingering challenges of expanding context length for large language models, despite introducing new techniques. Readers, what do you think are the most promising avenues for overcoming these scaling limitations? How much progress is still needed before we can build AI systems that reliably leverage longer contexts? I'm curious to hear your perspectives on the path forward.