Differences

This shows you the differences between two versions of the page.

--- atmos:dsp:workspace:fall2026:justin_abold_labreche [2026/10/01 22:50] – delene
+++ atmos:dsp:workspace:fall2026:justin_abold_labreche [2026/10/22 02:00] (current) – justin.abold
@@ Line 45: / Line 45: @@
-__Week 1 Update Date: 2026-09-23__
+====== 2026-09-23 ======
 **What I completed this week:**
@@ Line 65: / Line 66: @@
 Write and test Python code implementing the temporal cleansing rule, compare its behavior on the July 1 and September 14 datasets, and begin obtaining additional historical files so the approach can be tested on more than two days. Longer term, explore whether the cleansing process can be automated across the historical dataset rather than processing files individually.
-====== __2026-09-29__ ======
+====== 2026-09-29 ======
 **What I completed this week:**
@@ Line 128: / Line 127: @@
   - Begin thinking through how to automate the cleansing process - read a list of file names, iterate through and open each file - read the data, cleanse it, write the cleansed data to a few file.
+====== 2026-10-06 ======
+**What I completed:**
+  * Expanded the Python analysis from individual MiniOFS files to batch processing of the historical data files.
+  * Created functions to extract and separate sensor data and calculate descriptive statistics for each file.
+  * Applied the 2-before/2-after rule across the historical dataset to identify potentially erroneous sub-4000 m observations.
+  * Added statistics describing total observations, sub-4000 observations, jump frequency, jumps from 4000 m, transitions occurring entirely below 4000 m, jumps ≥500 m, and the proportion of sub-4000 observations identified by the 2/2 rule.
+  * Created a pandas DataFrame and exported the results to Excel for easier review.
+  * Identified substantial variation in the number of observations per file, including partial files, header-only files with zero observations, and files with substantially more observations than a typical day.
+**What I learned:**
+  * The suspected erroneous behavior appears to have a distinctive temporal pattern: isolated or short excursions from the 4000 m ceiling, often involving large jumps, while legitimate reduced-visibility periods tend to contain sustained sequences of changing values below 4000 m.
+  * The 2/2 rule identifies a high percentage of sub-4000 observations on many days during the suspected problem period, but a much smaller percentage during sustained low-visibility events. For example, July 1 was 88.46%, while September 14 and September 26 were 3.15% and 2.47%, respectively.    Pasted markdown    Pasted markdown
+  * Data completeness and file structure need to be understood before calculating formal probability-of-detection or false-alarm statistics.
+  * Gained additional Python experience with functions, nested dictionaries, exception handling, pandas DataFrames, batch file processing, and exporting results to Excel.
+**Paper, figure, dataset, tool, script, or output shared:**
+  * Python script for batch processing and descriptive analysis of historical MiniOFS files.
+  * Excel table containing file-level descriptive statistics and results of the 2/2 potential-error rule.
+  * Historical MiniOFS dataset reviewed through September 2026.
+**Question or blocker:**
+  * What accounts for the large variation in observations per file? A typical complete file contains approximately 2,800 observations, but some files contain only a few hundred, some contain zero observations, and some contain substantially more than a normal day (e.g., 19,751 observations on February 16).    Pasted markdown
+  * What do the header-only/zero-observation files represent?
+  * What are the exact dates of the wiring problem and repair?
+  * What should be used as ground truth for evaluating probability of detection and false alarms?
+**What I tried:**
+  * Tested the 2/2 rule on multiple known or presumed good and bad days and then automated it across the historical files.
+  * Compared several characteristics of the visibility data, including the proportion of observations below 4000 m, frequency of jumps, jump magnitude, and whether transitions occurred from the 4000 m ceiling or between two sub-4000 values.
+  * Used exception handling to accommodate files with no observations or no jumps without stopping batch processing.
+  * Examined several zero-observation files directly and confirmed that at least some contain only header information and no sensor observations.
+**Next step:**
+  * Review the statistical/outlier-filter Python code provided by Prof. Delene and compare its behavior with the current rule-based approach.
+  * Discuss the newly identified file-structure/data-completeness questions and the historical timing of the wiring problem with Shawn.
+  * Determine an appropriate ground-truth strategy and statistics for evaluating probability of detection and false alarms.
+  * Continue developing the automated workflow toward producing cleansed output files while preserving the original data.
+====== 2026-10-15 ======
+===== What I completed this week: =====
+I continued investigating possible sources of independent ground-truth data for evaluating the Clifford Hall MiniOFS visibility sensor. Our goal is to distinguish genuine periods of reduced atmospheric visibility from potentially erroneous sensor readings to be able to calculate probability of detection stats for our proposed data marking methods.
+I investigated three possible reference sources:
+  * Gorman Field MiniOFS: Obtained and examined observations from a second MiniOFS sensor approximately 15–20 miles west of Clifford Hall. Compared visibility patterns on selected dates, including September 14, 21, and 25.
+  * Grand Forks International Airport (GFK) ASOS: Located and downloaded historical airport visibility observations through the Iowa Environmental Mesonet. Developed Python code to extract observations for selected dates and successfully examined March 16, when all 24 airport observations reported 10 statute miles of visibility.
+  * North Dakota Agricultural Weather Network (NDAWN): Identified Grand Forks 3S (Station 26), near UND, as a possible source of independent meteorological observations, including temperature, relative humidity, dew point, wind, and solar radiation.
+  * Still TBD: EFM on Clifford Hall and at Gorman Field
+I also developed code to automate the creation of Python dictionaries for the Clifford Hall and Gorman Field datasets from all source files for 2026. Observations are now organized by their internal date stamps rather than by individual files, allowing multiple files to contribute observations to the same calendar date. This is an improvement over the past week.
+I completed a preliminary inventory of the two datasets:
+  * Gorman Field: 203 unique filename dates, 195 dates with observations, and 541,471 observations through approximately October 15.
+  * Clifford Hall: 210 unique filename dates, 177 dates with observations, and 437,037 observations through September 29. The unique filename-date count slightly underestimates the dates represented because some early observations were stored in rolling files.
+  * Average observations per day with data: 2,776.77 at Gorman Field and 2,469.14 at Clifford Hall.
+  * Identified 128 dates with observations from both sensors.
+I also discussed with Shawn the possibility of examining Gorman Field electric field mill (EFM) measurements to determine whether changes in atmospheric electric fields correlate with or precede changes in visibility. This may be the best source of ground truth since the sensor is collocated geographically with the minOFS visibility sensor.
+===== What I learned: =====
+Ground truth is more complicated than simply comparing two instruments. Gorman Field provides measurements from a similar sensor, but the stations are geographically separated, and genuine local differences in visibility are possible.
+The airport ASOS provides an independent visibility measurement, but its observations are generally hourly, compared with the approximately 30-second MinOFS sampling interval. This limits its usefulness for validating individual short-duration visibility changes. Also both MinOFS and ASOS visibility measurements have upper reporting limits. A MinOFS reading of 4,000 meters represents visibility at or above its reporting ceiling, while an ASOS reading of 10 miles may also represent an upper reporting limit. Neither necessarily represents the exact atmospheric visibility. Historical GFK ASOS visibility dataset obtained through the Iowa Environmental Mesonet.
+NDAWN does not directly measure visibility, but relative humidity, temperature, dew point, and wind measurements could help determine whether conditions were consistent with fog or other causes of reduced visibility. This makes using NDAWN more complicated.
+Comparing selected Clifford Hall and Gorman Field plots showed both agreements and disagreements. For example, September 21 showed generally good agreement at maximum visibility, while September 14 showed more substantial reductions at Clifford Hall than at Gorman Field. On September 25, both stations recorded reduced visibility, but their patterns differed.
+These comparisons suggest that isolated reductions, abrupt changes, and differences between stations may be useful indicators of questionable observations, but none independently proves that a reading is erroneous.
+I also earned improved my ability to use Python dictionaries to organize large datasets by date, including .setdefault(), .append(), .keys(), .values(), and .items(). Also learned how to handle malformed data records.
+Recognized the importance of distinguishing dates represented by filenames from dates containing actual observations, as well as checking data completeness before comparing daily statistics.
+===== Paper, figure, dataset, tool, script, or output shared: =====
+Create Dictionaries.py — Python script for reading and aggregating Clifford Hall and Gorman Field MiniOFS observations.
+{{ :atmos:dsp:workspace:fall2026:pythoncode20261015.docx |}}
+Comparative visibility plots for selected Clifford Hall and Gorman Field dates.
+Previously generated daily descriptive-statistics spreadsheets for both stations, which will be recalculated using the aggregated observations.
+===== Question or blocker: =====
+  * How can we establish sufficiently reliable ground-truth periods when the available instruments have different sampling frequencies, reporting limits, and locations?
+  * How much simultaneous coverage exists within the 128 dates when both MiniOFS sensors recorded observations?
+  * Earlier daily descriptive statistics may be incomplete because observations from multiple files associated with the same date were not always combined.
+  * Can Gorman Field EFM measurements provide useful information about visibility changes, either through correlation or by providing an advance indication of changes?
+===== What I tried: =====
+  * Compared MiniOFS visibility plots from both stations on selected dates to identify agreement, disagreement, and potentially anomalous behavior.
+  * Retrieved and parsed GFK ASOS observations, including a test of March 16 data, to explore their potential as an independent visibility reference.
+  * Investigated NDAWN station availability and the meteorological variables that could support interpretation of MiniOFS measurements.
+  * Developed a Python approach for grouping all observations by the actual dates recorded inside the data files, rather than relying on filenames.
+  * Added checks to exclude malformed records with fewer than four fields.
+  * Used dictionary operations to calculate observation counts, identify overlapping dates, and evaluate the completeness of the datasets.
+===== Next step: =====
+  * Finalize the Clifford Hall and Gorman Field dictionaries, including chronological sorting and checks for duplicate timestamps.
+  * Recalculate daily descriptive statistics using all observations for each date, then compare the results for both stations, particularly on the 128 overlapping dates.
+  * Begin identifying candidate ground-truth periods using agreement between MiniOFS observations and GFK ASOS reports, with NDAWN meteorological observations as additional supporting evidence.
+  * Access the Gorman Field EFM dataset, determine its structure and sampling frequency, and produce initial time-aligned plots of electric field and visibility measurements.
+  * Investigate whether changes in electric field measurements correlate with or precede changes in reported visibility.
+====== 2026-10-21 ======
+===== What I completed this week =====
+  * Successfully completed dictionaries of daily aggregated data and exported descriptive statistics for each day to excel files.
+  * Reviewed the daily statistics and comparisons between the two stations, including the frequency of observations below 4,000 meters and the number of observations flagged as potentially erroneous.
+  * Met with Shawn to discuss the results of the 2−/2+ screening rule, which identifies isolated observations below 4,000 meters when the two preceding and two following observations are all 4,000 meters.
+  * Examined how applying the rule changes the number of days containing only 4,000-meter observations. The rule identifies an additional 23 such days at Clifford Hall and 34 at Gorman Field.
+  * Investigated published research and technical documentation concerning MiniOFS measurement artifacts and methods for processing visibility data.
+===== What I learned =====
+  * The 2−/2+ rule identifies potentially erroneous observations, but the pattern alone does not establish that an observation is incorrect.
+  * Although Clifford Hall experienced a known wiring problem, no comparable physical cause has been identified for the suspicious observations at Gorman Field.
+  * The frequency of observations flagged by the rule may depend on weather conditions. During sustained reduced visibility, isolated erroneous measurements may not satisfy the rule because the surrounding observations are also below 4,000 meters.
+  * Rolling median filters are another established method for identifying short-duration anomalies. Depending on the window size, they can identify sequences of suspicious observations that the 2−/2+ rule would not detect.
+  * Published MiniOFS studies use various approaches to data processing, including temporal averaging and removal of artifacts. However, I have not found evidence that the specific isolated downward spikes observed in this dataset are a documented characteristic of the MiniOFS.
+====== Paper, figure, dataset, tool, script, or output shared ======
+  * Daily statistical summaries for Clifford Hall and Gorman Field, exported to Excel.
+  * Comparison of the 128 dates with observations from both stations.
+  *  Results of applying the 2−/2+ screening rule, including the number of additional days classified as containing only 4,000-meter observations.
+  * Reviewed published MiniOFS studies and documentation describing visibility measurement and data-processing methods.
+====== Question or blocker ======
+  * The underlying cause of isolated sub-4,000-meter observations remains uncertain, particularly at Gorman Field, where no physical sensor problem has been identified. Publicly available materials on the sensor as well as visibility sensors in general describe common problems, but there is a lack of specific discussions of short term spikes for MiniOFS. This may be because industry / government implementations of MiniOFS are using a longer sampling timeframe (5 min or 10 min) and this is smoothing out the spikes we are seeing in 30 second sample timeframe.
+  * It is also unclear how reliably the 2−/2+ rule distinguishes erroneous observations from genuine short-duration reductions in visibility. This is an area I am working on putting more rigor around with probability of detection methods.
+===== What I tried =====
+  * Investigated whether other MiniOFS users have reported similar observations or published methods for filtering them.
+  * Considered the potential advantages and limitations of rolling median filtering compared with the existing 2−/2+ rule based on literature review. I will likely test this experimentally with the data from Clifford Hall and Gorman Field.
+===== Next step =====
+  * Unordered List ItemBegin comparing the remaining sub-4,000-meter observations with independent meteorological measurements, including precipitation and electric field mill data. Specifically, for the days that have sub 4K observations after the 2+/2- rule has been applied and those observations meeting the rule criteria are "corrected" to 4K, are there independent sources that confirm a reduction in visibility for those days.
+  * Examine whether observations removed by the 2−/2+ rule correspond to genuine weather events.
+  * Investigate the distribution of flagged observations over time, accounting for differences in the opportunity to detect isolated anomalies during clear and reduced-visibility conditions.
+  * Consider comparing the results of the 2−/2+ rule with a rolling median filter.
+  * Investigate if the aggregation of data into daily files has left "gaps" (due to lack of sensor observations) that may impact the application of the 2-/2+ rule.
+  * Contact researchers at the University of California, Santa Cruz (UCSC), and potentially the University of Connecticut (UConn), to request samples of their raw MiniOFS observations. Apply the same 2−/2+ screening rule to these independent datasets to determine whether similar isolated downward spikes occur at other installations.