What I completed this week: I became familiar with the MiniOFS visibility sensor dataset and began exploring differences between a known problematic historical file (July 1) and a good comparison file (September 14). I plotted visibility over time and examined the relationship among the sensor's solar irradiance, extinction coefficient, and visibility fields. Based on the pattern in the visibility data, I began developing a simple rule-based approach for identifying potentially erroneous observations: when a sub-4000 visibility reading is surrounded by readings of 4000, mark the questionable observation as 9999 rather than deleting it.

What I learned: The visibility field is capped at 4000 m, and the problematic July 1 data contain repeated brief drops below 4000 that appear different from sustained changes in visibility. I also identified unexpected negative extinction-coefficient values in the September 14 data. Discussion with Shawn clarified that negative extinction values are not physically meaningful but can result from conditions such as dew on the sensor. The absence of those values on July 1 therefore does not explain the erroneous visibility readings, but examining the additional sensor fields provided useful insight into how the instrument behaves.

Paper, figure, dataset, tool, script, or output shared: Python scripts and Matplotlib plots comparing the July 1 problematic dataset with the September 14 good dataset. I produced plots of visibility over time and explored visibility in relation to solar irradiance and extinction coefficient. I also documented the negative extinction-coefficient finding for discussion with the project team.

Question or blocker: The primary question is how to distinguish erroneous historical visibility readings from legitimate atmospheric changes without removing valid observations. At this stage, only a small number of example days have been examined, so a rule that appears effective on July 1 needs to be tested against additional good and problematic days.

What I tried: I visually compared good and problematic datasets and explored whether the other sensor fields could help explain the erroneous visibility readings. I began testing the idea that an isolated sub-4000 observation surrounded by 4000-m readings could provide a simple temporal signature for questionable data. Rather than deleting questionable observations, the proposed approach is to preserve the record while recoding identified values as 9999.

Next step: Write and test Python code implementing the temporal cleansing rule, compare its behavior on the July 1 and September 14 datasets, and begin obtaining additional historical files so the approach can be tested on more than two days. Longer term, explore whether the cleansing process can be automated across the historical dataset rather than processing files individually.

What I completed this week: I developed and tested three candidate temporal filtering rules for identifying questionable visibility observations: 2-before/2-after (2/2), 2-before/1-after (2/1), and 1-before/1-after (1/1). I tested the rules against July 1 (problematic data) and September 14 (good data). The 2/2 rule identified 92 observations on July 1 versus 10 on September 14; relaxing the rule to 2/1 produced 94 versus 13, and 1/1 produced 96 versus 24. I also conducted preliminary analysis of the magnitude of changes between consecutive visibility readings and gained access through WinSCP/VPN to the historical sensor server and MiniOFS directory, although I do not yet have permission to access the data files.

What I learned: The preliminary results suggest that the conservative 2/2 rule provides better separation between the two test days than relaxing the temporal requirement. July 1 contained 104 sub-4000 observations, of which 92 (88.5%) met the 2/2 criterion, compared with only 10 of 317 (3.2%) on September 14.

The jump analysis also revealed substantial structural differences. On July 1, 108 of 204 nonzero changes (52.9%) were ≥500 m, compared with 28 of 385 (7.3%) on September 14. In addition, only 4 of July's 204 nonzero changes (2.0%) occurred between two sub-4000 readings, compared with 247 of 385 (64.2%) on September 14. The problematic day therefore appears dominated by abrupt excursions away from and back to the 4000-m ceiling, while the good day shows much more continuous variation below 4000.

I also examined what the 2/2 rule misses. The 12 remaining sub-4000 observations on July 1 occur in short clusters that prevent them from meeting the 2/2 requirement. This suggests that jump magnitude could potentially complement the temporal rule.

Paper, figure, dataset, tool, script, or output shared: Python scripts implementing the 2/2, 2/1, and 1/1 rules; comparative plots and histograms for the July 1 and September 14 datasets; descriptive output comparing nonzero jumps, jump magnitude, and sub-4000 behavior. I also established a WinSCP connection to the littlestorm server through the UND VPN.

Python Code for 2/2 Rule Counter = 0 Counter9999 = 0 for item in VisibilityField3:

  if item <4000 and Counter >=2 and Counter <= len(VisibilityField3)-3:
      if (VisibilityField3[Counter-2] == 4000 and VisibilityField3[Counter-1] == 4000 
          and VisibilityField3[Counter+1] == 4000 and VisibilityField3[Counter+2] == 4000):
          CleanedVisibilityField3[Counter] = 9999
          Counter9999 += 1
  Counter += 1

Python Code for Count of Observations Less than 4K count4 = 0 for item in VisibilityField3:

  if item < 4000:
      count4 += 1

Python Code for Jump Size and Jump Characterization JumpSize = [] JumpEnd = 0 JumpStart = 0 Non4000Jump = 0 for x in range(1,len(JumpsVisibilityField3)):

  JumpDifference = abs(JumpsVisibilityField3[x]-JumpsVisibilityField3[x-1])
  if JumpDifference != 0:
      JumpSize.append(JumpDifference)
      if JumpsVisibilityField3[x] == 4000:
          JumpEnd += 1
      elif JumpsVisibilityField3[x-1] == 4000:
          JumpStart += 1
      else:
          Non4000Jump += 1

Question or blocker: I currently can connect to the historical sensor server and reach the MiniOFS directory, but I receive a permissions error when attempting to access the sensor data directory. I therefore cannot yet obtain additional historical files needed to determine whether the patterns observed in July 1 and September 14 generalize to other days. *Update: I have access to the MiniOFS director and sensor files thanks to the help of Atmospheric Sciences team! (Thank you!) I successfully downloaded ten additional files and was able to read them in.*

What I tried: I compared increasingly permissive temporal rules, examined the observations captured and missed by the 2/2 rule, and explored whether the size and structure of consecutive visibility changes provide additional information. I created histograms of jump magnitude and calculated the frequency of ≥500-m changes and whether transitions involved the 4000-m ceiling. I also connected to the server using WinSCP after determining that access requires the UND VPN.

Next steps:

1. Continue with access to additional historical MiniOFS files and apply the same rules and descriptive measures without further tuning to multiple good and potentially problematic days. This will test whether the differences found between July 1 and September 14 are reproducible. If they are, I will explore a possible multi-stage approach in which day-level characteristics identify potentially problematic files, the conservative 2/2 rule identifies isolated questionable observations, and jump magnitude helps identify short clusters that 2/2 misses.
2. Explore the statistical outliers python code to identify how this might help with identifying erroneous data.
3. Begin thinking through how to automate the cleansing process - read a list of file names, iterate through and open each file - read the data, cleanse it, write the cleansed data to a few file.

What I completed:

• Expanded the Python analysis from individual MiniOFS files to batch processing of the historical data files.
• Created functions to extract and separate sensor data and calculate descriptive statistics for each file.
• Applied the 2-before/2-after rule across the historical dataset to identify potentially erroneous sub-4000 m observations.
• Added statistics describing total observations, sub-4000 observations, jump frequency, jumps from 4000 m, transitions occurring entirely below 4000 m, jumps ≥500 m, and the proportion of sub-4000 observations identified by the 2/2 rule.
• Created a pandas DataFrame and exported the results to Excel for easier review.
• Identified substantial variation in the number of observations per file, including partial files, header-only files with zero observations, and files with substantially more observations than a typical day.

What I learned:

• The suspected erroneous behavior appears to have a distinctive temporal pattern: isolated or short excursions from the 4000 m ceiling, often involving large jumps, while legitimate reduced-visibility periods tend to contain sustained sequences of changing values below 4000 m.
• The 2/2 rule identifies a high percentage of sub-4000 observations on many days during the suspected problem period, but a much smaller percentage during sustained low-visibility events. For example, July 1 was 88.46%, while September 14 and September 26 were 3.15% and 2.47%, respectively. Pasted markdown Pasted markdown
• Data completeness and file structure need to be understood before calculating formal probability-of-detection or false-alarm statistics.
• Gained additional Python experience with functions, nested dictionaries, exception handling, pandas DataFrames, batch file processing, and exporting results to Excel.

Paper, figure, dataset, tool, script, or output shared:

• Python script for batch processing and descriptive analysis of historical MiniOFS files.
• Excel table containing file-level descriptive statistics and results of the 2/2 potential-error rule.
• Historical MiniOFS dataset reviewed through September 2026.

Question or blocker:

• What accounts for the large variation in observations per file? A typical complete file contains approximately 2,800 observations, but some files contain only a few hundred, some contain zero observations, and some contain substantially more than a normal day (e.g., 19,751 observations on February 16). Pasted markdown
• What do the header-only/zero-observation files represent?
• What are the exact dates of the wiring problem and repair?
• What should be used as ground truth for evaluating probability of detection and false alarms?

What I tried:

• Tested the 2/2 rule on multiple known or presumed good and bad days and then automated it across the historical files.
• Compared several characteristics of the visibility data, including the proportion of observations below 4000 m, frequency of jumps, jump magnitude, and whether transitions occurred from the 4000 m ceiling or between two sub-4000 values.
• Used exception handling to accommodate files with no observations or no jumps without stopping batch processing.
• Examined several zero-observation files directly and confirmed that at least some contain only header information and no sensor observations.

Next step:

• Review the statistical/outlier-filter Python code provided by Prof. Delene and compare its behavior with the current rule-based approach.
• Discuss the newly identified file-structure/data-completeness questions and the historical timing of the wiring problem with Shawn.
• Determine an appropriate ground-truth strategy and statistics for evaluating probability of detection and false alarms.
• Continue developing the automated workflow toward producing cleansed output files while preserving the original data.