MosaicML launches MPT-7B-8K, a 7B-parameter open-source LLM

MosaicML has unveiled MPT-7B-8K, an open-source giant language mannequin (LLM) with 7 billion parameters and an 8k context size. 

In line with the corporate, the mannequin is skilled on the MosaicML platform and underwent a pretraining course of commencing from the MPT-7B checkpoint. The pretraining part was carried out utilizing Nvidia H100s, with a further three days of coaching on 256 H100s, incorporating a formidable 500 billion tokens of knowledge.

Beforehand, MosaicML had made waves within the AI group with its launch of MPT-30B, an open-source and commercially licensed decoder-based LLM. The corporate claimed it to be extra highly effective than GPT-3-175B, with solely 17% of GPT-3’s parameters, equal to 30 billion. 

MPT-30B surpassed GPT-3’s efficiency throughout varied duties and proved extra environment friendly to coach than fashions of comparable sizes. As an illustration, LLaMA-30B required roughly 1.44 occasions extra FLOPs price range than MPT-30B, whereas Falcon-40B had a 1.27 occasions greater FLOPs price range than MPT-30B.

MosaicML claims that the brand new mannequin MPT-7B-8K displays distinctive proficiency in doc summarization and question-answering duties in comparison with all beforehand launched fashions. 

The corporate mentioned the mannequin is particularly optimized for accelerated coaching and inference for faster outcomes. Furthermore, it permits fine-tuning of domain-specific information inside the MosaicML platform.

The corporate has additionally introduced the provision of commercial-use licensing for MPT-7B-8k, highlighting its distinctive coaching on an in depth dataset comprising 1.5 trillion tokens, surpassing comparable fashions like XGen, LLaMA, Pythia, OpenLLaMA and StableLM.

MosaicML claims that by way of the usage of FlashAttention and FasterTransformer, the mannequin excels in fast coaching and inference whereas benefiting from the open-source coaching code obtainable by way of the llm-foundry repository.

The corporate has launched the mannequin in three variations:

• MPT-7B-8k-Base: This decoder-style transformer is pretrained primarily based on MPT-7B and additional optimized with an prolonged sequence size of 8k. It undergoes extra coaching with 500 billion tokens, leading to a considerable corpus of 1.5 trillion tokens encompassing textual content and code.

• MPT-7B-8k-Instruct: This mannequin is designed for long-form instruction duties, together with summarization and question-answering. It’s crafted by fine-tuning MPT-7B-8k utilizing rigorously curated datasets.

• MPT-7B-8k-Chat: This variant capabilities as a chatbot-like mannequin, specializing in dialogue technology. It’s created by finetuning MPT-7B-8k with roughly 1.5 billion tokens of chat information.

Mosaic asserts that MPT-7B-8k fashions exhibit comparable or superior efficiency to different at the moment obtainable open-source fashions with an 8k context size, as confirmed by the corporate’s in-context studying analysis harness.

The announcement coincides with Meta’s unveiling of the LLaMA 2 mannequin, now obtainable on Microsoft Azure. Not like LLaMA 1, LLaMA 2 gives varied mannequin sizes, boasting 7, 13 and 70 billion parameters.

Meta asserts that these pre-trained fashions have been skilled on an unlimited dataset, 40% bigger than that of LLaMA 1, with an expanded context size of two trillion tokens, twice the dimensions of LLaMA 1. LLaMA 2 outperforms its predecessor based on Meta’s benchmarks.