GOAT AI releases a state-of-the-art open-source model

Last night, we released our first LLaMA-2 fine-tune, which achieved state-of-the-art results on the MMLU and BigBench Hard benchmarks. The model is open source and can be downloaded—and liked—directly from Hugging Face at the link below.

Link

This is an open version of an AI assistant similar to ChatGPT. It can be fine-tuned or run locally for research purposes. A demo is also available at the following link.

A few important details:

• Context length: 4096
• Model size: 7B parameters
• Dataset size: 72,000 dialogues

What makes our work unique? We focused on careful data cleaning, an essential part of a scientific approach to data analysis. Rather than cleaning the data in a single pass, we kept improving the process iteratively.

One key step toward better results was removing standard responses from larger AI models, such as “Sorry, as an AI assistant I cannot...”, from the dataset. After that, our model achieved the best results among all available 7-billion-parameter models.

The development team includes members of DSML KZ:

• Yerzat Dulat @rlprompt
• Aibek Bekbayev @qaszjo
• Sungbae Chun @Sungbae_Chun
• Jimmy Yamazaki @myxik

More technical details are available in the blog post about the model's release.