Overview

Ideal Data Structure

A lot of the coding effort is devoted to dealing with very messy formatting. The simplest solution is to use rsync to create an incremental data repository of local data on the server. Every time this application is opened, new files or changes are automatically uploaded to the parquet database (process_device_v12.R). Note that if a file is changed and processed it will appear multiple times in the log file. For errors, only the latest error is rcorded in the log file.

Main Data Issues

• There are duplicate files across sites and within sites
• The file naming convention is not robust - perhaps the cause of above
• The heart rate data measurement interavl doesn't appear to be 15s
• Over 95% of all minute intervals have 4 identical heart rates
• There appear to be spurious inconsistent heart rate peaks in the remaining minutes
• So averaging may not be a good solution - take the mode?
• It is vital for the analysis to know what the measurement interval is. Taking it as 15s will give a huge understimate of the variablitiy and compromise inference
• Open pdf of major issues

R Code Description: process_watch_v3.R

The process_watch_v3.R script processes files by:

1. Reading CSV or XLSX files from site folders.
2. Ignores files from 'original data files - causes repeats'.
3. Standardizing datetime fields, column names, and formats.
4. Standardizing units - meta data helps to highlight issue such as temeprature problem.
5. Parquet catalogs are created with the file name where the data originated - important for error correction debugging.
6. Partitions Parquet catalog by source file - helps debugging.
7. Recording results in a structured log with status, notes, and timestamps.
8. Data is only updated if it has changed in the raw data reference.
9. Note only data in Basse and Brikama folders are downlaoded. All other OneDrive folder names are ignored