Skip to content

Posts

Datahåndtering

For at kunne træne sprogmodeller (Large Language Models, LLM) skal der store mængder af data til. Fra vi modtager rådata til at de kan bruges til at træne sprogmodeller på, gennemgår de en transformationsprocess.

Følgende er en overordnet beskrivelse af denne processen. Vi udvikler og forbedre løbende processen, for at sikre at vi bruger state-pf-the-art metoder og praksis.

Datakilder

De data som sprogmodeller trænes på er afgørende for hvad de kan bruges til. I Danish Foundation Models (DFM) er tilgangen at vi skal have sikkerhed for at vi må benytte de data vi træner på fra data ejere, samt at vi har fokus på værdiskabende use-cases. Dette gør vi blandt andet gennem samarbejdet med Dansk Sprogmodel Konsortium.

Tutorial: Finetuning Language Models

This notebook will allow you to try out finetuning of the munin-7b-alpha model or, indeed, any other generative model out there.

We'll be finetuning the model on a Danish translated instruction tuning dataset, using the QLoRA method.

Tutorial: Merging Language Models

Model merging is a relatively new method that allows one to combine the weights of different language models into a single model.

In this notebook you'll get to try this out, as well as try to interact with the merged model to see the results!

Releasing Munin 7B Alpha - A Danish LLM

We are excited to announce the release of the first model from the Danish Foundation Models project, nicknamed Munin 7B Alpha. This model represents the beginning of our research into Danish Large Language Models (LLMs), employing continual pre-training based on the already pre-trained Mistral-7b-v0.1 model. It has been pre-trained on the Danish Gigaword dataset, which has been instrumental in training various Danish BERT-style models.