Differences

This shows you the differences between two versions of the page.

--- atmos:dsp:workspace:fall2026:justin_abold_labreche [2026/10/15 19:22] – justin.abold
+++ atmos:dsp:workspace:fall2026:justin_abold_labreche [2026/10/22 02:00] (current) – justin.abold
@@ Line 170: / Line 170: @@
 ===== What I completed this week: =====
+I continued investigating possible sources of independent ground-truth data for evaluating the Clifford Hall MiniOFS visibility sensor. Our goal is to distinguish genuine periods of reduced atmospheric visibility from potentially erroneous sensor readings to be able to calculate probability of detection stats for our proposed data marking methods.
-Continued investigating possible sources of independent ground-truth data for evaluating the Clifford Hall MiniOFS visibility sensor. Our goal is to distinguish genuine periods of reduced atmospheric visibility from potentially erroneous sensor readings.
+I investigated three possible reference sources:
-Investigated three possible reference sources:
+  * Gorman Field MiniOFS: Obtained and examined observations from a second MiniOFS sensor approximately 15–20 miles west of Clifford Hall. Compared visibility patterns on selected dates, including September 14, 21, and 25.
+  * Grand Forks International Airport (GFK) ASOS: Located and downloaded historical airport visibility observations through the Iowa Environmental Mesonet. Developed Python code to extract observations for selected dates and successfully examined March 16, when all 24 airport observations reported 10 statute miles of visibility.
+  * North Dakota Agricultural Weather Network (NDAWN): Identified Grand Forks 3S (Station 26), near UND, as a possible source of independent meteorological observations, including temperature, relative humidity, dew point, wind, and solar radiation.
+  * Still TBD: EFM on Clifford Hall and at Gorman Field
-Gorman Field MiniOFS: Obtained and examined observations from a second MiniOFS sensor approximately 15–20 miles west of Clifford Hall. Compared visibility patterns on selected dates, including September 14, 21, and 25.
+I also developed code to automate the creation of Python dictionaries for the Clifford Hall and Gorman Field datasets from all source files for 2026. Observations are now organized by their internal date stamps rather than by individual files, allowing multiple files to contribute observations to the same calendar date. This is an improvement over the past week.
-Grand Forks International Airport (GFK) ASOS: Located and downloaded historical airport visibility observations through the Iowa Environmental Mesonet. Developed Python code to extract observations for selected dates and successfully examined March 16, when all 24 airport observations reported 10 statute miles of visibility.
+I completed a preliminary inventory of the two datasets:
-North Dakota Agricultural Weather Network (NDAWN): Identified Grand Forks 3S (Station 26), near UND, as a possible source of independent meteorological observations, including temperature, relative humidity, dew point, wind, and solar radiation.
+  * Gorman Field: 203 unique filename dates, 195 dates with observations, and 541,471 observations through approximately October 15.
+  * Clifford Hall: 210 unique filename dates, 177 dates with observations, and 437,037 observations through September 29. The unique filename-date count slightly underestimates the dates represented because some early observations were stored in rolling files.
+  * Average observations per day with data: 2,776.77 at Gorman Field and 2,469.14 at Clifford Hall.
+  * Identified 128 dates with observations from both sensors.
-Developed Python dictionaries for the Clifford Hall and Gorman Field datasets. Observations are now organized by their internal date stamps rather than by individual files, allowing multiple files to contribute observations to the same calendar date.
+I also discussed with Shawn the possibility of examining Gorman Field electric field mill (EFM) measurements to determine whether changes in atmospheric electric fields correlate with or precede changes in visibility. This may be the best source of ground truth since the sensor is collocated geographically with the minOFS visibility sensor.
-Completed a preliminary inventory of the two datasets:
-Gorman Field: 203 unique filename dates, 195 dates with observations, and 541,471 observations through approximately October 15.
-Clifford Hall: 210 unique filename dates, 177 dates with observations, and 437,037 observations through September 29. The unique filename-date count slightly underestimates the dates represented because some early observations were stored in rolling files.
-Average observations per day with data: 2,776.77 at Gorman Field and 2,469.14 at Clifford Hall.
-Identified 128 dates with observations from both sensors.
-Discussed with Shawn the possibility of examining Gorman Field electric field mill (EFM) measurements to determine whether changes in atmospheric electric fields correlate with or precede changes in visibility.
 ===== What I learned: =====
@@ Line 200: / Line 195: @@
 Ground truth is more complicated than simply comparing two instruments. Gorman Field provides measurements from a similar sensor, but the stations are geographically separated, and genuine local differences in visibility are possible.
-The airport ASOS provides an independent visibility measurement, but its observations are generally hourly, compared with the approximately 30-second MiniOFS sampling interval. This limits its usefulness for validating individual short-duration visibility changes.
+The airport ASOS provides an independent visibility measurement, but its observations are generally hourly, compared with the approximately 30-second MinOFS sampling interval. This limits its usefulness for validating individual short-duration visibility changes. Also both MinOFS and ASOS visibility measurements have upper reporting limits. A MinOFS reading of 4,000 meters represents visibility at or above its reporting ceiling, while an ASOS reading of 10 miles may also represent an upper reporting limit. Neither necessarily represents the exact atmospheric visibility. Historical GFK ASOS visibility dataset obtained through the Iowa Environmental Mesonet.
-Both MiniOFS and ASOS visibility measurements have upper reporting limits. A MiniOFS reading of 4,000 meters represents visibility at or above its reporting ceiling, while an ASOS reading of 10 miles may also represent an upper reporting limit. Neither necessarily represents the exact atmospheric visibility.
+NDAWN does not directly measure visibility, but relative humidity, temperature, dew point, and wind measurements could help determine whether conditions were consistent with fog or other causes of reduced visibility. This makes using NDAWN more complicated.
-NDAWN does not directly measure visibility, but relative humidity, temperature, dew point, and wind measurements could help determine whether conditions were consistent with fog or other causes of reduced visibility.
 Comparing selected Clifford Hall and Gorman Field plots showed both agreements and disagreements. For example, September 21 showed generally good agreement at maximum visibility, while September 14 showed more substantial reductions at Clifford Hall than at Gorman Field. On September 25, both stations recorded reduced visibility, but their patterns differed.
@@ Line 210: / Line 203: @@
 These comparisons suggest that isolated reductions, abrupt changes, and differences between stations may be useful indicators of questionable observations, but none independently proves that a reading is erroneous.
-Learned how to use Python dictionaries to organize large datasets by date, including .setdefault(), .append(), .keys(), .values(), and .items(). Also learned how to count observations, identify overlapping dates, and handle malformed data records.
+I also earned improved my ability to use Python dictionaries to organize large datasets by date, including .setdefault(), .append(), .keys(), .values(), and .items(). Also learned how to handle malformed data records.
 Recognized the importance of distinguishing dates represented by filenames from dates containing actual observations, as well as checking data completeness before comparing daily statistics.
@@ Line 219: / Line 212: @@
 Create Dictionaries.py — Python script for reading and aggregating Clifford Hall and Gorman Field MiniOFS observations.
-Consolidated dictionaries containing 541,471 Gorman Field observations and 437,037 Clifford Hall observations.
+{{ :atmos:dsp:workspace:fall2026:pythoncode20261015.docx |}}
-Preliminary inventory of the two datasets, including 128 overlapping observation dates.
-Historical GFK ASOS visibility dataset obtained through the Iowa Environmental Mesonet.
 Comparative visibility plots for selected Clifford Hall and Gorman Field dates.
@@ Line 232: / Line 221: @@
-How can we establish sufficiently reliable ground-truth periods when the available instruments have different sampling frequencies, reporting limits, and locations?
+  * How can we establish sufficiently reliable ground-truth periods when the available instruments have different sampling frequencies, reporting limits, and locations?
+  * How much simultaneous coverage exists within the 128 dates when both MiniOFS sensors recorded observations?
+  * Earlier daily descriptive statistics may be incomplete because observations from multiple files associated with the same date were not always combined.
+  * Can Gorman Field EFM measurements provide useful information about visibility changes, either through correlation or by providing an advance indication of changes?
-How much simultaneous coverage exists within the 128 dates when both MiniOFS sensors recorded observations?
+===== What I tried: =====
-Earlier daily descriptive statistics may be incomplete because observations from multiple files associated with the same date were not always combined.
+  * Compared MiniOFS visibility plots from both stations on selected dates to identify agreement, disagreement, and potentially anomalous behavior.
+  * Retrieved and parsed GFK ASOS observations, including a test of March 16 data, to explore their potential as an independent visibility reference.
+  * Investigated NDAWN station availability and the meteorological variables that could support interpretation of MiniOFS measurements.
+  * Developed a Python approach for grouping all observations by the actual dates recorded inside the data files, rather than relying on filenames.
+  * Added checks to exclude malformed records with fewer than four fields.
+  * Used dictionary operations to calculate observation counts, identify overlapping dates, and evaluate the completeness of the datasets.
-Can Gorman Field EFM measurements provide useful information about visibility changes, either through correlation or by providing an advance indication of changes?
+===== Next step: =====
-===== What I tried: =====
+  * Finalize the Clifford Hall and Gorman Field dictionaries, including chronological sorting and checks for duplicate timestamps.
+  * Recalculate daily descriptive statistics using all observations for each date, then compare the results for both stations, particularly on the 128 overlapping dates.
+  * Begin identifying candidate ground-truth periods using agreement between MiniOFS observations and GFK ASOS reports, with NDAWN meteorological observations as additional supporting evidence.
+  * Access the Gorman Field EFM dataset, determine its structure and sampling frequency, and produce initial time-aligned plots of electric field and visibility measurements.
+  * Investigate whether changes in electric field measurements correlate with or precede changes in reported visibility.
-Compared MiniOFS visibility plots from both stations on selected dates to identify agreement, disagreement, and potentially anomalous behavior.
+====== 2026-10-21 ======
-Retrieved and parsed GFK ASOS observations, including a test of March 16 data, to explore their potential as an independent visibility reference.
+===== What I completed this week =====
-Investigated NDAWN station availability and the meteorological variables that could support interpretation of MiniOFS measurements.
+  * Successfully completed dictionaries of daily aggregated data and exported descriptive statistics for each day to excel files.
+  * Reviewed the daily statistics and comparisons between the two stations, including the frequency of observations below 4,000 meters and the number of observations flagged as potentially erroneous.
+  * Met with Shawn to discuss the results of the 2−/2+ screening rule, which identifies isolated observations below 4,000 meters when the two preceding and two following observations are all 4,000 meters.
+  * Examined how applying the rule changes the number of days containing only 4,000-meter observations. The rule identifies an additional 23 such days at Clifford Hall and 34 at Gorman Field.
+  * Investigated published research and technical documentation concerning MiniOFS measurement artifacts and methods for processing visibility data.
-Developed a Python approach for grouping all observations by the actual dates recorded inside the data files, rather than relying on filenames.
+===== What I learned =====
-Added checks to exclude malformed records with fewer than four fields.
+  * The 2−/2+ rule identifies potentially erroneous observations, but the pattern alone does not establish that an observation is incorrect.
+  * Although Clifford Hall experienced a known wiring problem, no comparable physical cause has been identified for the suspicious observations at Gorman Field.
+  * The frequency of observations flagged by the rule may depend on weather conditions. During sustained reduced visibility, isolated erroneous measurements may not satisfy the rule because the surrounding observations are also below 4,000 meters.
+  * Rolling median filters are another established method for identifying short-duration anomalies. Depending on the window size, they can identify sequences of suspicious observations that the 2−/2+ rule would not detect.
+  * Published MiniOFS studies use various approaches to data processing, including temporal averaging and removal of artifacts. However, I have not found evidence that the specific isolated downward spikes observed in this dataset are a documented characteristic of the MiniOFS.
-Used dictionary operations to calculate observation counts, identify overlapping dates, and evaluate the completeness of the datasets.
+====== Paper, figure, dataset, tool, script, or output shared ======
-===== Next step: =====
+  * Daily statistical summaries for Clifford Hall and Gorman Field, exported to Excel.
+  * Comparison of the 128 dates with observations from both stations.
+  *  Results of applying the 2−/2+ screening rule, including the number of additional days classified as containing only 4,000-meter observations.
+  * Reviewed published MiniOFS studies and documentation describing visibility measurement and data-processing methods.
+====== Question or blocker ======
-  * Finalize the Clifford Hall and Gorman Field dictionaries, including chronological sorting and checks for duplicate timestamps.
+  * The underlying cause of isolated sub-4,000-meter observations remains uncertain, particularly at Gorman Field, where no physical sensor problem has been identified. Publicly available materials on the sensor as well as visibility sensors in general describe common problems, but there is a lack of specific discussions of short term spikes for MiniOFS. This may be because industry / government implementations of MiniOFS are using a longer sampling timeframe (5 min or 10 min) and this is smoothing out the spikes we are seeing in 30 second sample timeframe.
-  * Recalculate daily descriptive statistics using all observations for each date, then compare the results for both stations, particularly on the 128 overlapping dates.
+  * It is also unclear how reliably the 2−/2+ rule distinguishes erroneous observations from genuine short-duration reductions in visibility. This is an area I am working on putting more rigor around with probability of detection methods.
-  * Begin identifying candidate ground-truth periods using agreement between MiniOFS observations and GFK ASOS reports, with NDAWN meteorological observations as additional supporting evidence.
-  * Access the Gorman Field EFM dataset, determine its structure and sampling frequency, and produce initial time-aligned plots of electric field and visibility measurements.
+===== What I tried =====
-  * Investigate whether changes in electric field measurements correlate with or precede changes in reported visibility.
+  * Investigated whether other MiniOFS users have reported similar observations or published methods for filtering them.
+  * Considered the potential advantages and limitations of rolling median filtering compared with the existing 2−/2+ rule based on literature review. I will likely test this experimentally with the data from Clifford Hall and Gorman Field.
+===== Next step =====
+  * Unordered List ItemBegin comparing the remaining sub-4,000-meter observations with independent meteorological measurements, including precipitation and electric field mill data. Specifically, for the days that have sub 4K observations after the 2+/2- rule has been applied and those observations meeting the rule criteria are "corrected" to 4K, are there independent sources that confirm a reduction in visibility for those days.
+  * Examine whether observations removed by the 2−/2+ rule correspond to genuine weather events.
+  * Investigate the distribution of flagged observations over time, accounting for differences in the opportunity to detect isolated anomalies during clear and reduced-visibility conditions.
+  * Consider comparing the results of the 2−/2+ rule with a rolling median filter.
+  * Investigate if the aggregation of data into daily files has left "gaps" (due to lack of sensor observations) that may impact the application of the 2-/2+ rule.
+  * Contact researchers at the University of California, Santa Cruz (UCSC), and potentially the University of Connecticut (UConn), to request samples of their raw MiniOFS observations. Apply the same 2−/2+ screening rule to these independent datasets to determine whether similar isolated downward spikes occur at other installations.