Falcon 40B Takes Flight

ABU DHABI: Amazon Web Services, Inc. (AWS) — an Amazon.com, Inc. company — announced that Technology Innovation Institute (TII) — the applied research pillar of Abu Dhabi’s Advanced Technology Research Council — trained its top-performing, open source Falcon 40B model on AWS.

Falcon 40B is a 40-billion-parameter large language model (LLM) available under the Apache 2.0 license that ranked #1 in Hugging Face’s Open LLM Leaderboard, which tracks, ranks, and evaluates LLMs across multiple benchmarks to identify top performing models. It was trained on 1 trillion tokens on Amazon SageMaker, a fully managed service for developing, training, tuning, and hosting machine learning models, including LLMs.

The launch represents a significant milestone in the UAE’s commitment to fostering AI innovation and scientific contributions in line with the UAE National AI Strategy 2031.

Customers can now deploy Falcon 40B from Amazon SageMaker JumpStart, a machine learning (ML) hub that offers pre-trained models, giving customers access to Falcon 40B’s state-of-the-art accuracy and industry-leading performance without having to build their own model from scratch. To get started with Amazon SageMaker JumpStart, visit aws.amazon.com/sagemaker/jumpstart.

Dr. Ebtesam Almazrouei, Executive Director – Acting Chief AI Researcher of AI Cross Center Unit and Project Lead for LLM Projects at TII, highlighted that Falcon 40B in a new blogpost, stating: “We proudly announce the official open-source release of Falcon 40B, the world’s top-ranking open-source language model. Falcon 40B is an exceptional open-source model with

40B parameters, specifically designed as a causal decoder-only model. It was trained on a vast dataset of 1,000B tokens, including RefinedWeb enhanced with curated corpora. The model is available under the Apache 2.0 license, ensuring its accessibility and usability. The architecture of Falcon 40B is optimised for inference, incorporating FlashAttention and multiquery techniques. By releasing Falcon 40B as an open-source model, we provide researchers, entrepreneurs, and organisations with the opportunity to harness its exceptional capabilities and drive advancements in AI-driven solutions from healthcare to space, finance, advanced manufacturing to biotech; the possibilities for AI-driven solutions are boundless.

Stability AI, AI 21 Labs, Hugging Face, and LG AI use Amazon SageMaker, AWS’s end-to-end machine ML service, to build, train, and deploy their LLMs to improve performance and cost.

“In line with the UAE National AI Strategy 2031, the UAE’s participation in global technological advancements like Falcon LLM is a critical component in our journey towards a knowledge-based economy,” continues Dr. Almazrouei. “The UAE chooses to actively involve itself in the broader conversation by investing in and developing AI solutions that will help create new economic, social, and educational opportunities. As part of this commitment, the open-source release of Falcon LLM, showcasing the UAE’s dedication to fostering collaboration, promoting transparency, and supporting innovation and research in the field of AI. Next steps include contributing to further advancements in the field of AI and advanced technologies, with new models on the horizon, and promoting the utilisation of advanced AI tech within organizations and businesses.”

A new generation of LLMs

TII turned to Amazon SageMaker to build its Falcon 40B model, which allowed them to focus on custom training mechanisms and optimisations instead of managing its ML infrastructure because SageMaker is fully managed. TII optimised training speed with deep learning operators to reduce training costs and time to market. AWS collaborated with TII to use SageMaker to improve resiliency and reduce developer interruptions during training.

TII announced its Falcon 40B model in March 2023, and released it as an open source in May 2023 under the Apache 2.0 license. Falcon 40B has outperformed contemporary models in multiple benchmarks without fine-tuning. Amazon SageMaker Jumpstart makes Falcon 40B easier for customers to access. Now customers of any size and industry can quickly and easily deploy their own Falcon 40B model and customise it for translation, question answering, summarising information, or identifying images.

Wojciech Bajda, Managing Director – Public Sector Middle East and Africa at AWS, said: “We are proud to have collaborated with the Technology Innovation Institute on the development

of the Falcon LLM 40B model, which leverages Amazon SageMaker to train a model that is state-of the art in multiple dimensions: technological breakthroughs, science quality, training speed and also operational simplicity. The open source release of Falcon 40B enables organisations to leverage its exceptional capabilities and advance AI-driven solutions to create new opportunities for progress and advancements.”

With Falcon LLM 40B, organisations worldwide can access its advanced tech capabilities and empower researchers to drive innovation in AI. To learn more about the Falcon LLM 40B model and how it can revolutionise industries, please visit FalconLLM.tii.ae.

For more information and deeper insights into the Falcon LLM project, please visit Dr. Ebtesam Almazrouei blog: Technology Innovation Institute trains the state-of-the-art Falcon LLM 40B foundation model on Amazon SageMaker.