What I completed this week: I became familiar with the MiniOFS visibility sensor dataset and began exploring differences between a known problematic historical file (July 1) and a good comparison file (September 14). I plotted visibility over time and examined the relationship among the sensor's solar irradiance, extinction coefficient, and visibility fields. Based on the pattern in the visibility data, I began developing a simple rule-based approach for identifying potentially erroneous observations: when a sub-4000 visibility reading is surrounded by readings of 4000, mark the questionable observation as 9999 rather than deleting it.

What I learned: The visibility field is capped at 4000 m, and the problematic July 1 data contain repeated brief drops below 4000 that appear different from sustained changes in visibility. I also identified unexpected negative extinction-coefficient values in the September 14 data. Discussion with Shawn clarified that negative extinction values are not physically meaningful but can result from conditions such as dew on the sensor. The absence of those values on July 1 therefore does not explain the erroneous visibility readings, but examining the additional sensor fields provided useful insight into how the instrument behaves.

Paper, figure, dataset, tool, script, or output shared: Python scripts and Matplotlib plots comparing the July 1 problematic dataset with the September 14 good dataset. I produced plots of visibility over time and explored visibility in relation to solar irradiance and extinction coefficient. I also documented the negative extinction-coefficient finding for discussion with the project team.

Question or blocker: The primary question is how to distinguish erroneous historical visibility readings from legitimate atmospheric changes without removing valid observations. At this stage, only a small number of example days have been examined, so a rule that appears effective on July 1 needs to be tested against additional good and problematic days.

What I tried: I visually compared good and problematic datasets and explored whether the other sensor fields could help explain the erroneous visibility readings. I began testing the idea that an isolated sub-4000 observation surrounded by 4000-m readings could provide a simple temporal signature for questionable data. Rather than deleting questionable observations, the proposed approach is to preserve the record while recoding identified values as 9999.

Next step: Write and test Python code implementing the temporal cleansing rule, compare its behavior on the July 1 and September 14 datasets, and begin obtaining additional historical files so the approach can be tested on more than two days. Longer term, explore whether the cleansing process can be automated across the historical dataset rather than processing files individually.

What I completed this week: I developed and tested three candidate temporal filtering rules for identifying questionable visibility observations: 2-before/2-after (2/2), 2-before/1-after (2/1), and 1-before/1-after (1/1). I tested the rules against July 1 (problematic data) and September 14 (good data). The 2/2 rule identified 92 observations on July 1 versus 10 on September 14; relaxing the rule to 2/1 produced 94 versus 13, and 1/1 produced 96 versus 24. I also conducted preliminary analysis of the magnitude of changes between consecutive visibility readings and gained access through WinSCP/VPN to the historical sensor server and MiniOFS directory, although I do not yet have permission to access the data files.

What I learned: The preliminary results suggest that the conservative 2/2 rule provides better separation between the two test days than relaxing the temporal requirement. July 1 contained 104 sub-4000 observations, of which 92 (88.5%) met the 2/2 criterion, compared with only 10 of 317 (3.2%) on September 14.

The jump analysis also revealed substantial structural differences. On July 1, 108 of 204 nonzero changes (52.9%) were ≥500 m, compared with 28 of 385 (7.3%) on September 14. In addition, only 4 of July's 204 nonzero changes (2.0%) occurred between two sub-4000 readings, compared with 247 of 385 (64.2%) on September 14. The problematic day therefore appears dominated by abrupt excursions away from and back to the 4000-m ceiling, while the good day shows much more continuous variation below 4000.

I also examined what the 2/2 rule misses. The 12 remaining sub-4000 observations on July 1 occur in short clusters that prevent them from meeting the 2/2 requirement. This suggests that jump magnitude could potentially complement the temporal rule.

Paper, figure, dataset, tool, script, or output shared: Python scripts implementing the 2/2, 2/1, and 1/1 rules; comparative plots and histograms for the July 1 and September 14 datasets; descriptive output comparing nonzero jumps, jump magnitude, and sub-4000 behavior. I also established a WinSCP connection to the littlestorm server through the UND VPN.

Python Code for 2/2 Rule Counter = 0 Counter9999 = 0 for item in VisibilityField3:

  if item <4000 and Counter >=2 and Counter <= len(VisibilityField3)-3:
      if (VisibilityField3[Counter-2] == 4000 and VisibilityField3[Counter-1] == 4000 
          and VisibilityField3[Counter+1] == 4000 and VisibilityField3[Counter+2] == 4000):
          CleanedVisibilityField3[Counter] = 9999
          Counter9999 += 1
  Counter += 1

Python Code for Count of Observations Less than 4K count4 = 0 for item in VisibilityField3:

  if item < 4000:
      count4 += 1

Python Code for Jump Size and Jump Characterization JumpSize = [] JumpEnd = 0 JumpStart = 0 Non4000Jump = 0 for x in range(1,len(JumpsVisibilityField3)):

  JumpDifference = abs(JumpsVisibilityField3[x]-JumpsVisibilityField3[x-1])
  if JumpDifference != 0:
      JumpSize.append(JumpDifference)
      if JumpsVisibilityField3[x] == 4000:
          JumpEnd += 1
      elif JumpsVisibilityField3[x-1] == 4000:
          JumpStart += 1
      else:
          Non4000Jump += 1

Question or blocker: I currently can connect to the historical sensor server and reach the MiniOFS directory, but I receive a permissions error when attempting to access the sensor data directory. I therefore cannot yet obtain additional historical files needed to determine whether the patterns observed in July 1 and September 14 generalize to other days. *Update: I have access to the MiniOFS director and sensor files thanks to the help of Atmospheric Sciences team! (Thank you!) I successfully downloaded ten additional files and was able to read them in.*

What I tried: I compared increasingly permissive temporal rules, examined the observations captured and missed by the 2/2 rule, and explored whether the size and structure of consecutive visibility changes provide additional information. I created histograms of jump magnitude and calculated the frequency of ≥500-m changes and whether transitions involved the 4000-m ceiling. I also connected to the server using WinSCP after determining that access requires the UND VPN.

Next steps:

1. Continue with access to additional historical MiniOFS files and apply the same rules and descriptive measures without further tuning to multiple good and potentially problematic days. This will test whether the differences found between July 1 and September 14 are reproducible. If they are, I will explore a possible multi-stage approach in which day-level characteristics identify potentially problematic files, the conservative 2/2 rule identifies isolated questionable observations, and jump magnitude helps identify short clusters that 2/2 misses.
2. Explore the statistical outliers python code to identify how this might help with identifying erroneous data.
3. Begin thinking through how to automate the cleansing process - read a list of file names, iterate through and open each file - read the data, cleanse it, write the cleansed data to a few file.

What I completed:

• Expanded the Python analysis from individual MiniOFS files to batch processing of the historical data files.
• Created functions to extract and separate sensor data and calculate descriptive statistics for each file.
• Applied the 2-before/2-after rule across the historical dataset to identify potentially erroneous sub-4000 m observations.
• Added statistics describing total observations, sub-4000 observations, jump frequency, jumps from 4000 m, transitions occurring entirely below 4000 m, jumps ≥500 m, and the proportion of sub-4000 observations identified by the 2/2 rule.
• Created a pandas DataFrame and exported the results to Excel for easier review.
• Identified substantial variation in the number of observations per file, including partial files, header-only files with zero observations, and files with substantially more observations than a typical day.

What I learned:

• The suspected erroneous behavior appears to have a distinctive temporal pattern: isolated or short excursions from the 4000 m ceiling, often involving large jumps, while legitimate reduced-visibility periods tend to contain sustained sequences of changing values below 4000 m.
• The 2/2 rule identifies a high percentage of sub-4000 observations on many days during the suspected problem period, but a much smaller percentage during sustained low-visibility events. For example, July 1 was 88.46%, while September 14 and September 26 were 3.15% and 2.47%, respectively. Pasted markdown Pasted markdown
• Data completeness and file structure need to be understood before calculating formal probability-of-detection or false-alarm statistics.
• Gained additional Python experience with functions, nested dictionaries, exception handling, pandas DataFrames, batch file processing, and exporting results to Excel.

Paper, figure, dataset, tool, script, or output shared:

• Python script for batch processing and descriptive analysis of historical MiniOFS files.
• Excel table containing file-level descriptive statistics and results of the 2/2 potential-error rule.
• Historical MiniOFS dataset reviewed through September 2026.

Question or blocker:

• What accounts for the large variation in observations per file? A typical complete file contains approximately 2,800 observations, but some files contain only a few hundred, some contain zero observations, and some contain substantially more than a normal day (e.g., 19,751 observations on February 16). Pasted markdown
• What do the header-only/zero-observation files represent?
• What are the exact dates of the wiring problem and repair?
• What should be used as ground truth for evaluating probability of detection and false alarms?

What I tried:

• Tested the 2/2 rule on multiple known or presumed good and bad days and then automated it across the historical files.
• Compared several characteristics of the visibility data, including the proportion of observations below 4000 m, frequency of jumps, jump magnitude, and whether transitions occurred from the 4000 m ceiling or between two sub-4000 values.
• Used exception handling to accommodate files with no observations or no jumps without stopping batch processing.
• Examined several zero-observation files directly and confirmed that at least some contain only header information and no sensor observations.

Next step:

• Review the statistical/outlier-filter Python code provided by Prof. Delene and compare its behavior with the current rule-based approach.
• Discuss the newly identified file-structure/data-completeness questions and the historical timing of the wiring problem with Shawn.
• Determine an appropriate ground-truth strategy and statistics for evaluating probability of detection and false alarms.
• Continue developing the automated workflow toward producing cleansed output files while preserving the original data.

I continued investigating possible sources of independent ground-truth data for evaluating the Clifford Hall MiniOFS visibility sensor. Our goal is to distinguish genuine periods of reduced atmospheric visibility from potentially erroneous sensor readings to be able to calculate probability of detection stats for our proposed data marking methods.

I investigated three possible reference sources:

• Gorman Field MiniOFS: Obtained and examined observations from a second MiniOFS sensor approximately 15–20 miles west of Clifford Hall. Compared visibility patterns on selected dates, including September 14, 21, and 25.
• Grand Forks International Airport (GFK) ASOS: Located and downloaded historical airport visibility observations through the Iowa Environmental Mesonet. Developed Python code to extract observations for selected dates and successfully examined March 16, when all 24 airport observations reported 10 statute miles of visibility.
• North Dakota Agricultural Weather Network (NDAWN): Identified Grand Forks 3S (Station 26), near UND, as a possible source of independent meteorological observations, including temperature, relative humidity, dew point, wind, and solar radiation.
• Still TBD: EFM on Clifford Hall and at Gorman Field

I also developed code to automate the creation of Python dictionaries for the Clifford Hall and Gorman Field datasets from all source files for 2026. Observations are now organized by their internal date stamps rather than by individual files, allowing multiple files to contribute observations to the same calendar date. This is an improvement over the past week.

I completed a preliminary inventory of the two datasets:

• Gorman Field: 203 unique filename dates, 195 dates with observations, and 541,471 observations through approximately October 15.
• Clifford Hall: 210 unique filename dates, 177 dates with observations, and 437,037 observations through September 29. The unique filename-date count slightly underestimates the dates represented because some early observations were stored in rolling files.
• Average observations per day with data: 2,776.77 at Gorman Field and 2,469.14 at Clifford Hall.
• Identified 128 dates with observations from both sensors.

I also discussed with Shawn the possibility of examining Gorman Field electric field mill (EFM) measurements to determine whether changes in atmospheric electric fields correlate with or precede changes in visibility. This may be the best source of ground truth since the sensor is collocated geographically with the minOFS visibility sensor.

Ground truth is more complicated than simply comparing two instruments. Gorman Field provides measurements from a similar sensor, but the stations are geographically separated, and genuine local differences in visibility are possible.

The airport ASOS provides an independent visibility measurement, but its observations are generally hourly, compared with the approximately 30-second MinOFS sampling interval. This limits its usefulness for validating individual short-duration visibility changes. Also both MinOFS and ASOS visibility measurements have upper reporting limits. A MinOFS reading of 4,000 meters represents visibility at or above its reporting ceiling, while an ASOS reading of 10 miles may also represent an upper reporting limit. Neither necessarily represents the exact atmospheric visibility. Historical GFK ASOS visibility dataset obtained through the Iowa Environmental Mesonet.

NDAWN does not directly measure visibility, but relative humidity, temperature, dew point, and wind measurements could help determine whether conditions were consistent with fog or other causes of reduced visibility. This makes using NDAWN more complicated.

Comparing selected Clifford Hall and Gorman Field plots showed both agreements and disagreements. For example, September 21 showed generally good agreement at maximum visibility, while September 14 showed more substantial reductions at Clifford Hall than at Gorman Field. On September 25, both stations recorded reduced visibility, but their patterns differed.

These comparisons suggest that isolated reductions, abrupt changes, and differences between stations may be useful indicators of questionable observations, but none independently proves that a reading is erroneous.

I also earned improved my ability to use Python dictionaries to organize large datasets by date, including .setdefault(), .append(), .keys(), .values(), and .items(). Also learned how to handle malformed data records.

Recognized the importance of distinguishing dates represented by filenames from dates containing actual observations, as well as checking data completeness before comparing daily statistics.

Create Dictionaries.py — Python script for reading and aggregating Clifford Hall and Gorman Field MiniOFS observations.

pythoncode20261015.docx

Comparative visibility plots for selected Clifford Hall and Gorman Field dates.

Previously generated daily descriptive-statistics spreadsheets for both stations, which will be recalculated using the aggregated observations.

• How can we establish sufficiently reliable ground-truth periods when the available instruments have different sampling frequencies, reporting limits, and locations?
• How much simultaneous coverage exists within the 128 dates when both MiniOFS sensors recorded observations?
• Earlier daily descriptive statistics may be incomplete because observations from multiple files associated with the same date were not always combined.
• Can Gorman Field EFM measurements provide useful information about visibility changes, either through correlation or by providing an advance indication of changes?

• Compared MiniOFS visibility plots from both stations on selected dates to identify agreement, disagreement, and potentially anomalous behavior.
• Retrieved and parsed GFK ASOS observations, including a test of March 16 data, to explore their potential as an independent visibility reference.
• Investigated NDAWN station availability and the meteorological variables that could support interpretation of MiniOFS measurements.
• Developed a Python approach for grouping all observations by the actual dates recorded inside the data files, rather than relying on filenames.
• Added checks to exclude malformed records with fewer than four fields.
• Used dictionary operations to calculate observation counts, identify overlapping dates, and evaluate the completeness of the datasets.

• Finalize the Clifford Hall and Gorman Field dictionaries, including chronological sorting and checks for duplicate timestamps.
• Recalculate daily descriptive statistics using all observations for each date, then compare the results for both stations, particularly on the 128 overlapping dates.
• Begin identifying candidate ground-truth periods using agreement between MiniOFS observations and GFK ASOS reports, with NDAWN meteorological observations as additional supporting evidence.
• Access the Gorman Field EFM dataset, determine its structure and sampling frequency, and produce initial time-aligned plots of electric field and visibility measurements.
• Investigate whether changes in electric field measurements correlate with or precede changes in reported visibility.

• Successfully completed dictionaries of daily aggregated data and exported descriptive statistics for each day to excel files.
• Reviewed the daily statistics and comparisons between the two stations, including the frequency of observations below 4,000 meters and the number of observations flagged as potentially erroneous.
• Met with Shawn to discuss the results of the 2−/2+ screening rule, which identifies isolated observations below 4,000 meters when the two preceding and two following observations are all 4,000 meters.
• Examined how applying the rule changes the number of days containing only 4,000-meter observations. The rule identifies an additional 23 such days at Clifford Hall and 34 at Gorman Field.
• Investigated published research and technical documentation concerning MiniOFS measurement artifacts and methods for processing visibility data.

• The 2−/2+ rule identifies potentially erroneous observations, but the pattern alone does not establish that an observation is incorrect.
• Although Clifford Hall experienced a known wiring problem, no comparable physical cause has been identified for the suspicious observations at Gorman Field.
• The frequency of observations flagged by the rule may depend on weather conditions. During sustained reduced visibility, isolated erroneous measurements may not satisfy the rule because the surrounding observations are also below 4,000 meters.
• Rolling median filters are another established method for identifying short-duration anomalies. Depending on the window size, they can identify sequences of suspicious observations that the 2−/2+ rule would not detect.
• Published MiniOFS studies use various approaches to data processing, including temporal averaging and removal of artifacts. However, I have not found evidence that the specific isolated downward spikes observed in this dataset are a documented characteristic of the MiniOFS.

• Daily statistical summaries for Clifford Hall and Gorman Field, exported to Excel.
• Comparison of the 128 dates with observations from both stations.
• Results of applying the 2−/2+ screening rule, including the number of additional days classified as containing only 4,000-meter observations.
• Reviewed published MiniOFS studies and documentation describing visibility measurement and data-processing methods.

• The underlying cause of isolated sub-4,000-meter observations remains uncertain, particularly at Gorman Field, where no physical sensor problem has been identified. Publicly available materials on the sensor as well as visibility sensors in general describe common problems, but there is a lack of specific discussions of short term spikes for MiniOFS. This may be because industry / government implementations of MiniOFS are using a longer sampling timeframe (5 min or 10 min) and this is smoothing out the spikes we are seeing in 30 second sample timeframe.
• It is also unclear how reliably the 2−/2+ rule distinguishes erroneous observations from genuine short-duration reductions in visibility. This is an area I am working on putting more rigor around with probability of detection methods.

• Investigated whether other MiniOFS users have reported similar observations or published methods for filtering them.
• Considered the potential advantages and limitations of rolling median filtering compared with the existing 2−/2+ rule based on literature review. I will likely test this experimentally with the data from Clifford Hall and Gorman Field.

• Unordered List ItemBegin comparing the remaining sub-4,000-meter observations with independent meteorological measurements, including precipitation and electric field mill data. Specifically, for the days that have sub 4K observations after the 2+/2- rule has been applied and those observations meeting the rule criteria are “corrected” to 4K, are there independent sources that confirm a reduction in visibility for those days.
• Examine whether observations removed by the 2−/2+ rule correspond to genuine weather events.
• Investigate the distribution of flagged observations over time, accounting for differences in the opportunity to detect isolated anomalies during clear and reduced-visibility conditions.
• Consider comparing the results of the 2−/2+ rule with a rolling median filter.
• Investigate if the aggregation of data into daily files has left “gaps” (due to lack of sensor observations) that may impact the application of the 2-/2+ rule.
• Contact researchers at the University of California, Santa Cruz (UCSC), and potentially the University of Connecticut (UConn), to request samples of their raw MiniOFS observations. Apply the same 2−/2+ screening rule to these independent datasets to determine whether similar isolated downward spikes occur at other installations.

This week I continued developing and evaluating a method for identifying potentially erroneous observations from the MiniOFS visibility sensor on the roof of Clifford Hall. The work focused on three related activities: evaluating the effectiveness of the 2-before/2-after screening rule, comparing MiniOFS observations with independent meteorological datasets, and investigating whether the Clifford Hall Electric Field Mill (EFM) might provide additional evidence about the reliability of the MiniOFS measurements.

I also developed additional Python code to import and organize EFM data, refined the selection of representative observation days, addressed several programming and data-completeness problems, and prepared a four-page collection of comparative plots for discussion with Shawn Wagner.

### A. Further evaluation of the 2-before/2-after screening rule

I continued evaluating the screening rule I developed to identify potentially erroneous MiniOFS observations.

The MiniOFS reports visibility up to a maximum of 4,000 meters, with observations recorded approximately every 30 seconds.

The rule identifies an observation below 4,000 meters as a candidate error when the two immediately preceding observations and the two immediately following observations all report 4,000 meters.

This means that each candidate is surrounded by approximately one minute of maximum-visibility observations on either side.

The underlying hypothesis is that an isolated reduction in reported visibility, surrounded by observations at the instrument's maximum, may be the result of an instrument malfunction rather than a genuine atmospheric change.

However, the rule does not establish that an individual observation is erroneous. It identifies observations that warrant further investigation.

I used the Python program developed previously to calculate daily statistics, including the number of observations below 4,000 meters, the number identified by the screening rule, and the number remaining after screening.

These statistics were organized in Excel and used both to evaluate the rule's effectiveness and to identify representative dates for additional analysis.

### B. Statistical comparison with airport METAR visibility observations

I compared daily MiniOFS visibility classifications with METAR visibility observations from the Grand Forks airport.

For this analysis, I classified each day as having high or low visibility depending on whether observations fell below the MiniOFS maximum of 4,000 meters. The corresponding METAR threshold is approximately 2.49 miles.

I used daily classifications rather than attempting to match individual observations because the instruments are geographically separated and may experience different atmospheric conditions. This approach provides a preliminary measure of agreement but does not establish whether individual MiniOFS readings are correct.

There were 177 days with overlapping observations.

Before applying the screening rule, the instruments agreed on 102 of 177 days:

102 / 177 = 57.6%

After applying the screening rule, agreement increased to 113 of 177 days:

113 / 177 = 63.8%

The screening procedure therefore increased daily agreement by 11 days, or approximately 6.2 percentage points.

The improvement was not uniform, however.

Seventeen days previously classified as disagreements became agreements, while six days previously classified as agreements became disagreements.

The largest change occurred in the category where METAR indicated high visibility but MiniOFS indicated low visibility. This category decreased from 64 to 47 days.

At the same time, the number of days where METAR indicated low visibility but MiniOFS indicated high visibility increased from 11 to 17.

These results suggest that the screening rule may improve the overall consistency of the MiniOFS dataset with independent visibility observations, but it may also flag some genuine low-visibility measurements.

The comparison therefore provides evidence that the rule is useful, but it does not establish its accuracy in identifying individual erroneous observations.

An additional limitation is the geographic separation between Clifford Hall and the airport. Localized fog, precipitation, or other atmospheric conditions could produce genuine differences in visibility at the two locations.

### C. Statistical comparison with NDAWN precipitation observations

I also compared MiniOFS visibility classifications with precipitation observations from the North Dakota Agricultural Weather Network (NDAWN).

The purpose was to investigate whether the screening rule improved consistency with another independent source of meteorological information.

The comparison included 177 days.

Before screening, the datasets agreed under this precipitation/visibility classification on 88 of 177 days:

88 / 177 = 49.7%

After screening, agreement increased to 91 of 177 days:

91 / 177 = 51.4%

This represents an improvement of only three days, or approximately 1.7 percentage points.

The improvement was considerably smaller than in the METAR comparison.

However, precipitation is not a direct measure of visibility. Fog and other visibility-reducing conditions can occur without precipitation, while precipitation does not necessarily reduce visibility below 4,000 meters.

Consequently, the relatively low agreement does not necessarily indicate that either dataset is inaccurate.

I concluded that NDAWN precipitation is useful as contextual meteorological information but is not sufficiently reliable to serve as ground truth for individual MiniOFS visibility observations.

### D. Investigation of remaining disagreements

I examined dates where the airport METAR observations indicated high visibility but the Clifford Hall MiniOFS continued to report visibility below 4,000 meters after application of the screening rule.

The remaining disagreements were not all similar.

Some dates contained relatively few low-visibility observations remaining after screening. Other dates contained hundreds or thousands of observations below 4,000 meters that were not identified by the rule.

For example, March 14 contained 2,847 observations below 4,000 meters, none of which were identified by the 2-before/2-after rule.

This is an important limitation of the current approach.

The rule is specifically designed to identify isolated low-visibility observations surrounded by readings at the instrument maximum. It cannot identify sustained periods of potentially erroneous measurements because those observations do not satisfy the surrounding-observation criterion.

Therefore, the absence of flagged observations on a particular day does not necessarily establish that the MiniOFS data are reliable.

This analysis helped distinguish two different problems: identifying isolated anomalous observations and identifying sustained periods of potentially erroneous sensor behavior.

It also reinforced the need for independent evidence before determining whether a sustained period of low reported visibility reflects genuine weather conditions or a sensor problem.

### E. Importing and organizing Clifford Hall EFM data

I began investigating whether observations from the Electric Field Mill (EFM) at Clifford Hall could provide additional information about the MiniOFS readings.

The EFM files contain two voltage measurements labeled Sensitive and Insensitive.

Because the EFM is located at Clifford Hall, it provides a local source of observations. However, I have not established whether the physical phenomena measured by the EFM should correspond to changes in optical visibility.

The initial objective was therefore exploratory: to examine the EFM time series and determine whether recognizable patterns occurred during different types of MiniOFS behavior.

I developed Python code to import the EFM data.

Unlike the MiniOFS files, the EFM files have a different header structure and represent time using seconds since midnight rather than a complete datetime value for each observation.

My program reads the header to identify the observation date, skips the remaining header information, and imports the data as numeric values.

I organized the EFM observations in a dictionary named `CliffordHallEFM`, using Python datetime objects representing each date as dictionary keys.

Each observation is stored as a tuple containing:

- Seconds since midnight UTC - Sensitive voltage - Insensitive voltage

I also revised the MiniOFS import code so that its dictionary, `CliffordHall`, uses the same type of datetime key.

This allows me to retrieve the MiniOFS and EFM observations for a particular date using a common date variable.

The MiniOFS observations retain their complete datetime timestamps and visibility values.

I used pandas DataFrames to organize the selected observations for plotting and Matplotlib to create separate time-series plots for the EFM Sensitive voltage, EFM Insensitive voltage, and MiniOFS visibility.

### F. Developing a method for selecting representative days

I developed Python code to identify dates with different proportions of MiniOFS observations below 4,000 meters.

For each date, the program counts the number of observations below 4,000 meters and compares that count with the total number of observations for the day.

I used these statistics to identify dates with consistently high visibility, predominantly low visibility, and mixed visibility.

I also experimented with using Python's `random.randint()` function to randomly select qualifying dates.

During this process, I learned that applying separate random selections to two categories can produce apparently inconsistent results, even when one category is a subset of the other. For example, a random selection from dates with more than 50% low-visibility observations can contain fewer dates than a separate random selection from dates with more than 25% low-visibility observations.

This does not indicate an error in the classification logic; it results from the independent random selections.

For the fourth category, I used the existing Excel statistics to identify a date containing numerous observations flagged by the 2-before/2-after rule.

The final dates selected for comparison were:

### G. Comparison of EFM and MiniOFS observations

I prepared a four-page PDF containing three plots for each selected date: EFM Sensitive voltage, EFM Insensitive voltage, and MiniOFS visibility.

comparing_efm_and_miniofs_clifford_hall.pdf

March 23, 2026 — Consistently high visibility

The MiniOFS reported 4,000 meters throughout the day.

The EFM Sensitive channel was relatively stable for much of the day, generally around 0.3–0.6 volts, but exhibited a substantial positive spike exceeding 6 volts around 63,000 seconds after midnight UTC. This was followed by a negative excursion and additional fluctuations.

The EFM Insensitive channel exhibited a corresponding spike approaching 0.10 volts.

Despite these substantial changes in EFM measurements, MiniOFS continued reporting 4,000 meters.

This is potentially important because it shows that a large change in the EFM measurements does not necessarily correspond to reduced optical visibility.

March 14, 2026 — Sustained low visibility

MiniOFS visibility remained approximately between 975 and 1,145 meters throughout the day.

The readings exhibited gradual variation, including a decline during the first portion of the day, a subsequent increase, and another decline later in the day.

The EFM Sensitive channel remained relatively stable during the earlier portion of the day, generally around 0.5–1 volt, but began exhibiting substantial negative excursions around 58,000 seconds after midnight UTC.

Some of these excursions reached approximately −5 volts.

The EFM Insensitive channel also changed substantially during the later portion of the day.

The timing of the EFM changes did not appear to correspond directly to the gradual variations in MiniOFS visibility.

This date is also important because it contains 2,847 MiniOFS observations below 4,000 meters, none of which was identified by the 2-before/2-after rule.

April 3, 2026 — Mixed visibility

The MiniOFS observations showed substantial variation throughout the day.

Visibility was initially below 4,000 meters, then increased to the maximum. Later, the instrument reported a sharp reduction to very low visibility, followed by recovery and another period at 4,000 meters.

The EFM Sensitive channel exhibited several pronounced negative excursions earlier in the day, with readings approaching −4 volts.

The Insensitive channel also showed negative excursions during the earlier period.

However, the EFM measurements became comparatively stable during portions of the day when MiniOFS continued exhibiting substantial changes.

Although both instruments showed periods of variability, the timing and patterns were not consistently aligned.

May 7, 2026 — Numerous potentially erroneous observations

The MiniOFS reported predominantly 4,000-meter visibility, interrupted by numerous isolated observations below the maximum.

Some observations were only slightly below 4,000 meters, while others represented much larger reductions, including readings below 2,000 meters.

This is the type of behavior the 2-before/2-after screening rule is intended to investigate.

The EFM Sensitive channel remained relatively stable during much of the day, generally around 0.2–0.3 volts, but exhibited a pronounced negative excursion around 53,000–59,000 seconds after midnight UTC.

The Insensitive channel also exhibited changes during this interval.

However, MiniOFS reductions occurred both during and outside the period of pronounced EFM activity.

The plots therefore did not reveal an obvious one-to-one relationship between changes in EFM voltage and isolated reductions in MiniOFS visibility.

Overall interpretation

The four comparisons illustrate different patterns of MiniOFS behavior, but I did not identify a consistent relationship between EFM voltage measurements and reported optical visibility.

The March 23 observations are particularly interesting because the EFM exhibited substantial activity while MiniOFS reported maximum visibility throughout the day.

The May 7 observations are also useful because the isolated MiniOFS reductions occurred during periods of both relatively stable and changing EFM measurements.

However, I cannot draw a firm conclusion without understanding the physical meaning of the EFM measurements and whether a relationship with optical visibility should be expected.

### H. Data completeness and plotting adjustments

I initially selected July 23 as the example of a day containing observations flagged by the 2-before/2-after rule.

After preparing the plots, I noticed that the EFM data covered only approximately one minute, while the MiniOFS data extended across the full day.

I checked for additional EFM files for July 23 but found none.

Because the incomplete EFM coverage prevented a meaningful full-day comparison, I replaced July 23 with May 7.

I used the daily statistics already compiled in Excel to identify May 7 as a date containing numerous candidate observations.

I also refined the plotting methods during the week.

Initially, I used line plots for the MiniOFS observations. I changed these to scatter plots with small markers so that individual observations, particularly isolated reductions, would be easier to distinguish.

I also found that fixing the vertical axis at 0–4,000 meters could make relatively small changes difficult to see.

For example, the March 14 observations vary within a range of approximately 170 meters. Displaying those readings on a 4,000-meter scale obscured much of the variation.

I therefore allowed the vertical axis to reflect the actual range of observations when detailed variation was important.

The resulting figures provide a clearer representation of the behavior of each instrument.

This week's work provided substantial additional experience with Python programming, data analysis, visualization, and the challenges of working with real instrument data.

Python dictionaries and data types

I gained additional experience organizing observations in dictionaries and using datetime objects as keys.

During development, I encountered a `KeyError` because I had created a dictionary entry using a date string but attempted to retrieve it using a datetime object.

Correcting this problem reinforced the importance of using consistent data types when organizing and retrieving observations.

Reading files with different formats

The MiniOFS and EFM datasets have different header structures and timestamp formats.

I learned how to extract the observation date from the EFM header, skip the appropriate number of header lines, and convert the observations to floating-point values.

I also gained experience working with seconds-since-midnight timestamps and full datetime timestamps.

Python string handling

I encountered a `SyntaxError` when using a Windows file path containing backslashes inside a multiline Python string.

I learned that Python can interpret combinations such as `\U` as escape sequences, even inside triple-quoted strings. Using a raw string or forward slashes avoids this problem.

Random sampling and conditional logic

I gained experience using random-number functions to select dates satisfying specified conditions.

I also learned to distinguish between the classification of qualifying dates and the random selection of a subset of those dates.

Data visualization

I gained additional experience using pandas and Matplotlib to plot large numbers of observations.

I learned that the choice between line and scatter plots can materially affect how the data are interpreted.

I also learned that axis limits should be selected carefully. A scale that is appropriate for comparing the full range of possible values may obscure important variation within a narrow range.

Evaluating a screening rule

The METAR comparison demonstrated that a screening procedure can improve overall agreement with an independent dataset while still introducing new disagreements.

This is an important distinction because an improvement in agreement does not establish that the rule correctly identifies every erroneous observation.

I also learned that the rule's design determines the types of errors it can identify. The 2-before/2-after rule is appropriate for identifying isolated anomalies but cannot detect sustained periods of erroneous readings if those readings are not surrounded by maximum-visibility observations.

Using independent observations

I gained a better understanding of the strengths and limitations of different reference datasets.

METAR visibility is directly relevant but geographically separated from Clifford Hall.

NDAWN precipitation provides useful weather information but is not a direct measure of visibility.

EFM measurements are collected locally but may not have a direct physical relationship with optical visibility.

These differences are important when determining whether an independent dataset can actually validate the MiniOFS observations.

Data quality and completeness

The July 23 EFM problem reinforced the need to verify that datasets cover comparable time periods before attempting to interpret relationships between them.

The project is increasingly demonstrating that data preparation, validation, and documentation are as important as the statistical calculations themselves.

The principal output this week is a four-page PDF titled Comparing EFM and MiniOFS Clifford Hall.

The document contains three plots for each of four dates:

1. EFM Sensitive voltage versus seconds since midnight UTC. 2. EFM Insensitive voltage versus seconds since midnight UTC. 3. MiniOFS visibility versus time UTC.

The selected dates are March 23, March 14, April 3, and May 7, 2026.

Each date represents a different pattern of MiniOFS visibility observations.

The figures provide examples for discussing whether EFM measurements might help distinguish genuine low-visibility conditions from potentially erroneous MiniOFS observations.

I also produced or updated the following materials:

- MiniOFS Python import code: Reads MiniOFS files and organizes timestamped visibility observations in a dictionary indexed by date. - EFM Python import code: Reads EFM files, extracts the observation date, and stores seconds-since-midnight, Sensitive voltage, and Insensitive voltage. - Date-selection code: Identifies dates according to the proportion of observations below 4,000 meters and supports random selection of qualifying dates. - Plotting code: Uses pandas and Matplotlib to generate comparable EFM and MiniOFS time-series plots. - Excel daily statistics: Contains the number of observations below 4,000 meters, the number identified as potential errors, and the number remaining after screening. - METAR comparison results: Documents the change in daily agreement from 57.6% to 63.8% after screening. - NDAWN comparison results: Documents the change in daily agreement from 49.7% to 51.4% after screening.

These materials document both the development of the screening procedure and the preliminary evaluation of its effectiveness.

Can EFM measurements provide meaningful evidence about the validity of MiniOFS visibility observations?

The EFM is located at Clifford Hall, which makes it potentially useful as a source of local measurements.

However, I do not yet understand the physical interpretation of the Sensitive and Insensitive voltages sufficiently to determine whether changes in these measurements should correspond to changes in optical visibility.

The four-day comparison did not reveal a simple or consistent relationship.

I would like Shawn's assistance interpreting the EFM data and determining whether further comparison is scientifically justified.

How reliably does the 2-before/2-after rule identify erroneous observations?

The rule improved agreement with METAR observations, but it also caused some previously agreeing days to become disagreements.

I need to determine whether these changes reflect genuine low-visibility observations being flagged or limitations of using geographically separated METAR observations as a reference.

How should sustained periods of potentially erroneous observations be identified?

The current rule cannot identify sustained periods of low reported visibility because those observations do not meet the surrounding-observation criterion.

The March 14 dataset illustrates this limitation.

I need to determine whether a different screening method or additional reference measurements would be necessary to identify sustained errors.

Would comparisons with Gorman Field provide stronger evidence?

The Gorman Field MiniOFS sensor may provide a useful comparison because it measures the same physical quantity using a similar instrument.

However, differences in local weather conditions and the possibility of sensor-specific problems would still need to be considered.

I evaluated the 2-before/2-after screening rule by comparing daily MiniOFS classifications with independent METAR visibility observations.

I calculated agreement before and after screening, examined how individual agreement categories changed, and investigated dates that continued to disagree after screening.

I also compared MiniOFS classifications with NDAWN precipitation observations. The smaller improvement in agreement reinforced the limitations of using precipitation as a reference for visibility.

To investigate the EFM dataset, I developed Python code to read the files, extract the observation date, organize measurements by date, and plot both EFM channels alongside MiniOFS visibility.

I selected representative days with different visibility patterns and examined the resulting plots for possible relationships between the instruments.

I experimented with random selection of qualifying dates and used the existing Excel statistics to identify a date with numerous observations flagged by the screening rule.

I initially used July 23 for this purpose, but the EFM data covered only a short interval. I checked for additional files and then replaced July 23 with May 7.

I also corrected a dictionary-key data-type problem, addressed a Python string escape-sequence error, and modified the plotting methods to make individual MiniOFS observations and smaller variations easier to see.

These efforts produced a set of comparative figures and additional statistical results, but they did not establish that EFM measurements can be used to confirm MiniOFS errors.

My immediate next step is to review the four-day EFM/MiniOFS comparison with Shawn Wagner.

I would like to clarify what the EFM Sensitive and Insensitive channels measure, what causes the large voltage excursions, and whether there is any physical reason to expect their measurements to correspond to changes in optical visibility.

Based on that discussion, I will determine whether additional EFM analysis is warranted.

I also plan to continue evaluating the effectiveness and limitations of the 2-before/2-after screening rule.

In particular, I would like to investigate whether the rule is flagging some genuine low-visibility observations and whether a different method is needed to identify sustained periods of potentially erroneous readings.

Another possible next step is to compare Clifford Hall MiniOFS observations with those from Gorman Field to determine whether similar patterns occur at both locations or are specific to the Clifford Hall instrument.

The longer-term objective remains to develop a reliable method for identifying erroneous MiniOFS observations and marking confirmed errors with the designated value of `9999`, while preserving the original observations and documenting the basis for each quality-control decision.

The principal result this week is that the current screening rule increased daily agreement with airport METAR observations from approximately 57.6% to 63.8% across 177 overlapping days. However, the results also show that additional evidence is needed before the observations identified by the rule can be considered confirmed errors.

The EFM investigation represents an effort to identify such additional evidence. The next step is to determine, with Shawn's assistance, whether the EFM measurements are physically relevant to that objective.

This week I met with Shawn and Prof Delene and provided a project update (see below).

20261105_status_update_briefing.pdf

• Developed a preliminary EFM weather-event indicator using the 95th percentile of absolute 30-minute voltage slopes.
• Analyzed 20.3 million EFM observations across 220 days; 138 days were flagged for elevated electrical activity.
• Compared EFM flags with MiniOFS reduced-visibility observations. Agreement increased from 77.2% before cleaning to 81.3% after cleaning.

• EFM data may provide supporting evidence of atmospheric activity, but electrical disturbances do not necessarily correspond to reduced visibility.
• The Clifford Hall CL61 ceilometer provides fog and precipitation indicators, with measurements every five seconds stored in ten-minute files.

• EFM slope plots for flagged and unflagged days.
• EFM daily ground-truth dataset and MiniOFS comparison results.

• Whether the MiniOFS cleaning rule removes genuine short-duration visibility reductions.
• Whether ceilometer indicators can independently corroborate individual MiniOFS dips.

• Compared daily EFM activity flags with MiniOFS visibility classifications before and after cleaning.
• Examined EFM voltage and slope plots to evaluate the threshold-based approach.

1. Identify several isolated MiniOFS visibility dips in October.
2. Examine corresponding ceilometer fog and precipitation indicators at the precise event times.
3. Determine whether the ceilometer provides evidence supporting or contradicting the MiniOFS measurements.