Differences

This shows you the differences between two versions of the page.

--- atmos:dsp:workspace:fall2026:justin_abold_labreche [2026/10/15 19:29] – justin.abold
+++ atmos:dsp:workspace:fall2026:justin_abold_labreche [2026/10/22 02:00] (current) – justin.abold
@@ Line 212: / Line 212: @@
 Create Dictionaries.py — Python script for reading and aggregating Clifford Hall and Gorman Field MiniOFS observations.
-FileNames = []
+{{ :atmos:dsp:workspace:fall2026:pythoncode20261015.docx |}}
-with open("Gorman Field/FileListGorman.txt", "r", encoding="utf-16") as FileList:
-    for line in FileList:
-        Fields = line.rstrip()
-        FileNames.append(Fields)
-ListofDatesGF = []
-for item in FileNames:
-    FileDate = item.split("_")
-    if FileDate[1] not in ListofDatesGF:
-        ListofDatesGF.append(FileDate[1])
-GormanField = {}
-for item in FileNames:
-    with open("Gorman Field/" + item) as SourceFile:
-        for x in range(4):
-            SourceFile.readline()
-        for line in SourceFile:
-            SourceData = line.rstrip().split()
-            if len(SourceData) >= 4:
-                DateTime = SourceData[0].split("_")
-                GormanField.setdefault(DateTime[0], [])
-                GormanField[DateTime[0]].append((DateTime[1], int(SourceData[3])))
 Comparative visibility plots for selected Clifford Hall and Gorman Field dates.
@@ Line 245: / Line 221: @@
-How can we establish sufficiently reliable ground-truth periods when the available instruments have different sampling frequencies, reporting limits, and locations?
+  * How can we establish sufficiently reliable ground-truth periods when the available instruments have different sampling frequencies, reporting limits, and locations?
+  * How much simultaneous coverage exists within the 128 dates when both MiniOFS sensors recorded observations?
-How much simultaneous coverage exists within the 128 dates when both MiniOFS sensors recorded observations?
+  * Earlier daily descriptive statistics may be incomplete because observations from multiple files associated with the same date were not always combined.
+  * Can Gorman Field EFM measurements provide useful information about visibility changes, either through correlation or by providing an advance indication of changes?
-Earlier daily descriptive statistics may be incomplete because observations from multiple files associated with the same date were not always combined.
-Can Gorman Field EFM measurements provide useful information about visibility changes, either through correlation or by providing an advance indication of changes?
 ===== What I tried: =====
@@ Line 269: / Line 242: @@
   * Access the Gorman Field EFM dataset, determine its structure and sampling frequency, and produce initial time-aligned plots of electric field and visibility measurements.
   * Investigate whether changes in electric field measurements correlate with or precede changes in reported visibility.
+====== 2026-10-21 ======
+===== What I completed this week =====
+  * Successfully completed dictionaries of daily aggregated data and exported descriptive statistics for each day to excel files.
+  * Reviewed the daily statistics and comparisons between the two stations, including the frequency of observations below 4,000 meters and the number of observations flagged as potentially erroneous.
+  * Met with Shawn to discuss the results of the 2−/2+ screening rule, which identifies isolated observations below 4,000 meters when the two preceding and two following observations are all 4,000 meters.
+  * Examined how applying the rule changes the number of days containing only 4,000-meter observations. The rule identifies an additional 23 such days at Clifford Hall and 34 at Gorman Field.
+  * Investigated published research and technical documentation concerning MiniOFS measurement artifacts and methods for processing visibility data.
+===== What I learned =====
+  * The 2−/2+ rule identifies potentially erroneous observations, but the pattern alone does not establish that an observation is incorrect.
+  * Although Clifford Hall experienced a known wiring problem, no comparable physical cause has been identified for the suspicious observations at Gorman Field.
+  * The frequency of observations flagged by the rule may depend on weather conditions. During sustained reduced visibility, isolated erroneous measurements may not satisfy the rule because the surrounding observations are also below 4,000 meters.
+  * Rolling median filters are another established method for identifying short-duration anomalies. Depending on the window size, they can identify sequences of suspicious observations that the 2−/2+ rule would not detect.
+  * Published MiniOFS studies use various approaches to data processing, including temporal averaging and removal of artifacts. However, I have not found evidence that the specific isolated downward spikes observed in this dataset are a documented characteristic of the MiniOFS.
+====== Paper, figure, dataset, tool, script, or output shared ======
+  * Daily statistical summaries for Clifford Hall and Gorman Field, exported to Excel.
+  * Comparison of the 128 dates with observations from both stations.
+  *  Results of applying the 2−/2+ screening rule, including the number of additional days classified as containing only 4,000-meter observations.
+  * Reviewed published MiniOFS studies and documentation describing visibility measurement and data-processing methods.
+====== Question or blocker ======
+  * The underlying cause of isolated sub-4,000-meter observations remains uncertain, particularly at Gorman Field, where no physical sensor problem has been identified. Publicly available materials on the sensor as well as visibility sensors in general describe common problems, but there is a lack of specific discussions of short term spikes for MiniOFS. This may be because industry / government implementations of MiniOFS are using a longer sampling timeframe (5 min or 10 min) and this is smoothing out the spikes we are seeing in 30 second sample timeframe.
+  * It is also unclear how reliably the 2−/2+ rule distinguishes erroneous observations from genuine short-duration reductions in visibility. This is an area I am working on putting more rigor around with probability of detection methods.
+===== What I tried =====
+  * Investigated whether other MiniOFS users have reported similar observations or published methods for filtering them.
+  * Considered the potential advantages and limitations of rolling median filtering compared with the existing 2−/2+ rule based on literature review. I will likely test this experimentally with the data from Clifford Hall and Gorman Field.
+===== Next step =====
+  * Unordered List ItemBegin comparing the remaining sub-4,000-meter observations with independent meteorological measurements, including precipitation and electric field mill data. Specifically, for the days that have sub 4K observations after the 2+/2- rule has been applied and those observations meeting the rule criteria are "corrected" to 4K, are there independent sources that confirm a reduction in visibility for those days.
+  * Examine whether observations removed by the 2−/2+ rule correspond to genuine weather events.
+  * Investigate the distribution of flagged observations over time, accounting for differences in the opportunity to detect isolated anomalies during clear and reduced-visibility conditions.
+  * Consider comparing the results of the 2−/2+ rule with a rolling median filter.
+  * Investigate if the aggregation of data into daily files has left "gaps" (due to lack of sensor observations) that may impact the application of the 2-/2+ rule.
+  * Contact researchers at the University of California, Santa Cruz (UCSC), and potentially the University of Connecticut (UConn), to request samples of their raw MiniOFS observations. Apply the same 2−/2+ screening rule to these independent datasets to determine whether similar isolated downward spikes occur at other installations.