Stage 3: Process & Analyse
Transform raw data into interpretable outputs using a meticulous processing pipeline.
-
Pipeline Steps: Include cleaning (filling missing values/removing outliers), normalisation (using calibration data to make data comparable), transformation, and aggregation (data fusion).
-
Track Changes: Use version control systems (like Git or GitLab) to manage dataset iterations. Maintain a CHANGELOG file to record what changed, when, by whom, and why. Use semantic versioning (e.g., v1.0, v1.1).
-
Data Quality Assurance (DQA): Apply quality metrics. The European Metrology Network for Mathematics and Statistics (Mathmet) provides "Quality Assurance Tools" that evaluate the "data integrity level" based on data complexity and criticality of usage.