Skip to main content Skip to page footer

Stage 3: Process & Analyse

Transform raw data into interpretable outputs using a meticulous processing pipeline.

  • Pipeline Steps: Include cleaning (filling missing values/removing outliers), normalisation (using calibration data to make data comparable), transformation, and aggregation (data fusion).

  • Track Changes: Use version control systems (like Git or GitLab) to manage dataset iterations. Maintain a CHANGELOG file to record what changed, when, by whom, and why. Use semantic versioning (e.g., v1.0, v1.1).

  • Data Quality Assurance (DQA): Apply quality metrics. The European Metrology Network for Mathematics and Statistics (Mathmet) provides "Quality Assurance Tools" that evaluate the "data integrity level" based on data complexity and criticality of usage.