Norwegian Petroleum Corpus
52,865
MT pairs
73,866
RAG chunks
0
Split leakage
Tech Stack
Description
This project develops an open-source and reproducible pipeline for constructing Norwegian petroleum-domain research data from public English and Norwegian documents. The current implementation processes Equinor web pages and PDFs to support machine translation, language-model research, and information retrieval while preserving document-level provenance.
My Contributions
As the project lead, I designed and implemented the complete corpus-construction pipeline, from source collection and text extraction to bilingual document pairing, sentence alignment, quality filtering, and dataset splitting. I also created traceable research outputs for machine translation, language-model research, and information retrieval.
Project Highlights
Reproducible Corpus Pipeline
- 1
Collect and Extract
The pipeline collects public English and Norwegian web pages and PDFs from Equinor. It extracts the text and saves the title, language, URL, document ID, and other source information.
- 2
Pair Bilingual Documents
For the machine translation data, official language-switch links are used to identify matching English and Norwegian articles. Reciprocal links are checked to confirm that the two articles form a valid bilingual pair.
- 3
Align Sentences
The matched articles are divided into sentences. LaBSE is then used to find English and Norwegian sentences that are likely to be translations of each other.
- 4
Filter Sentence Pairs
The pipeline retains sentence pairs with similar meanings and reasonable lengths. Website noise and duplicate pairs are removed, and the Norwegian sentences are checked to ensure that they are written in Bokmal.
- 5
Create Dataset Splits
The final sentence pairs are divided into training, validation, and test sets by bilingual document pair. This prevents content from the same article from appearing in different sets.
PDFs and web pages without a matching translation are used only for the language-model and retrieval data.
Research Outputs
| Output | Intended use |
|---|---|
| English-Norwegian sentence pairs | Machine translation |
| Monolingual document records | Language-model training and adaptation |
| Traceable text chunks | Information retrieval and RAG |
Research Connection
This corpus provides the new-domain data for a subsequent continual-adaptation study of new-source learning, old-source forgetting, and replay-based retention in petroleum-domain machine translation.
View Continual Adaptation Study