swiss-ai

A continued pre-train of the fully open-source Apertus 1.0, using 2T more tokens.

A series of small models (0.5B-4B) distilled from their 8B model using logit distillation.

An open model by the Swiss EPFL, ETH Zürich, and Swiss National Supercomputing Centre, using 15T tokens from open datasets such as FineWeb, DCLM or The Stack. Importantly, they filtered this data by excluding domains which opted out of training. While this model itself is not pushing the open state of the art, it is an important first step towards a higher diversity, especially in the truly open model space.