Benchmarking Language Models using the Together Research Computer — TOGETHER

together.xyz · 994 words · saved by 1 readers

Stanford Center for Research on Foundation Models (CRFM) announced Holistic Evaluation of Language Models (HELM), a comprehensive effort to benchmark 30 language models, including models with limited API access such as OpenAI’s GPT-3 as well as the new emerging ecosystem of open models (e.g., Meta’s

Together’s software network harnessed spare GPU cycles across thousands of servers to benchmark 10 prominent open language models and process 11 billion tokens. We have entered the era of foundation models — massive models trained on huge amounts of data — which can be adapted to a wide range of applications. Language models such as GPT-3 in particular have rich capabilities. They can improve the quality of existing applications (e.g., question answering) and introduce novel applications (e.g., brainstorming slogans or writing blog posts). The pace of innovation is rapid, with new models being

Benchmarking Language Models using the Together Research Computer — TOGETHER

related reading