Atmos Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
__**Justin Abold-LaBreche - Data Skills Pathway Workspace**__

**Student Information**
  * Name: Justin Abold-LaBreche
  * Major: Electrical Engineering and Cybersecurity Engineering
  * Year: Sophomore
  * Semester: Fall 2026
  * Email: justin.abold@und.edu


**Mentor Information**
  * Mentor Name: Shawn Wagner
  * Mentor Department / Program: Atmospheric Sciences
  * Mentor Email: shawn.wagner@und.edu

Meeting Schedule: Wednesdays, 1PM Central / 2PM Eastern

**Project Interest**
Sensors
Engineering
Cybersecurity

**Project Question or Problem**
Historical visibility-sensor data contain anomalous readings during periods when visibility was otherwise clear. The sensor's wiring was corrected in early September 2026, and subsequent data no longer exhibit the known wiring-related errors. The challenge is to develop a reliable method for identifying erroneous observations in the historical data and marking them with a value of 9999, allowing researchers to distinguish invalid measurements while preserving the remainder of the historical dataset for analysis.

__Project Brief__

**Project Title:** Identification and Cleansing of Anomalous Visibility Sensor Data

**Project Goal:** Develop and test a method for identifying erroneous observations in historical visibility-sensor data caused by a known sensor wiring problem. Erroneous observations will be marked with a value of 9999 so they can be distinguished from valid measurements while preserving the historical dataset for research use.

There is a potential longer term goal of automating the data cleansing process. 

**Data Source:** Historical data from the IR System MiniBSV visibility sensor located on the roof of Clifford Hall at the University of North Dakota. Daily sensor files contain a timestamp and three data fields, including solar irradiance (Field 1), extinction coefficient (Field 2), and horizontal visibility in meters (Field 3). The sensor's maximum reported visibility is 4,000 meters.

**Method or Tools:** Python will be used to read, parse, analyze, and eventually cleanse the historical sensor files. Matplotlib is being used to visualize relationships among the sensor fields and changes over time. Initial analysis compares known good and problematic data and tests simple temporal rules for identifying anomalous visibility observations. For example, one candidate rule flags a sub-4,000-meter observation when the two preceding and two following observations all report 4,000-meter visibility. Candidate rules will be tested against additional historical and post-repair data before determining an appropriate cleansing method.

**Expected Output:** A Python-based method for identifying and marking erroneous historical visibility observations with 9999. If feasible within the project timeframe, the longer-term objective is to automate the process so that the agreed-upon cleansing procedure can be applied across multiple historical data files rather than processing each file individually.

**Why this project matters:**
Historical visibility data may be valuable for atmospheric-science research, but erroneous measurements caused by the previous wiring problem can compromise subsequent analysis if they cannot be distinguished from legitimate reductions in visibility caused by actual atmospheric conditions. Developing a reproducible method for identifying and marking those observations would improve the usability of the historical dataset while retaining valid measurements.

The project also supports my longer-term interest in cyber-physical systems, particularly power systems, smart grids, and industrial control systems. Although the application here is an atmospheric sensor rather than a power-system device, the underlying problem is similar: using data from a physical sensor to distinguish genuine changes in the physical environment from anomalous measurements caused by equipment or system faults. I am using the Data Skills Pathway to develop the Python, data-analysis, and anomaly-detection skills that I hope to apply in future undergraduate research and eventually to cyber-physical power-system research.



====== 2026-09-23 ======


**What I completed this week:**
I became familiar with the MiniOFS visibility sensor dataset and began exploring differences between a known problematic historical file (July 1) and a good comparison file (September 14). I plotted visibility over time and examined the relationship among the sensor's solar irradiance, extinction coefficient, and visibility fields. Based on the pattern in the visibility data, I began developing a simple rule-based approach for identifying potentially erroneous observations: when a sub-4000 visibility reading is surrounded by readings of 4000, mark the questionable observation as 9999 rather than deleting it.

**What I learned:**
The visibility field is capped at 4000 m, and the problematic July 1 data contain repeated brief drops below 4000 that appear different from sustained changes in visibility. I also identified unexpected negative extinction-coefficient values in the September 14 data. Discussion with Shawn clarified that negative extinction values are not physically meaningful but can result from conditions such as dew on the sensor. The absence of those values on July 1 therefore does not explain the erroneous visibility readings, but examining the additional sensor fields provided useful insight into how the instrument behaves.

**Paper, figure, dataset, tool, script, or output shared:**
Python scripts and Matplotlib plots comparing the July 1 problematic dataset with the September 14 good dataset. I produced plots of visibility over time and explored visibility in relation to solar irradiance and extinction coefficient. I also documented the negative extinction-coefficient finding for discussion with the project team.

**Question or blocker:**
The primary question is how to distinguish erroneous historical visibility readings from legitimate atmospheric changes without removing valid observations. At this stage, only a small number of example days have been examined, so a rule that appears effective on July 1 needs to be tested against additional good and problematic days.

**What I tried:**
I visually compared good and problematic datasets and explored whether the other sensor fields could help explain the erroneous visibility readings. I began testing the idea that an isolated sub-4000 observation surrounded by 4000-m readings could provide a simple temporal signature for questionable data. Rather than deleting questionable observations, the proposed approach is to preserve the record while recoding identified values as 9999.

**Next step:**
Write and test Python code implementing the temporal cleansing rule, compare its behavior on the July 1 and September 14 datasets, and begin obtaining additional historical files so the approach can be tested on more than two days. Longer term, explore whether the cleansing process can be automated across the historical dataset rather than processing files individually.

====== 2026-09-29 ======

**What I completed this week:**
I developed and tested three candidate temporal filtering rules for identifying questionable visibility observations: 2-before/2-after (2/2), 2-before/1-after (2/1), and 1-before/1-after (1/1). I tested the rules against July 1 (problematic data) and September 14 (good data). The 2/2 rule identified 92 observations on July 1 versus 10 on September 14; relaxing the rule to 2/1 produced 94 versus 13, and 1/1 produced 96 versus 24. I also conducted preliminary analysis of the magnitude of changes between consecutive visibility readings and gained access through WinSCP/VPN to the historical sensor server and MiniOFS directory, although I do not yet have permission to access the data files.

**What I learned:**
The preliminary results suggest that the conservative 2/2 rule provides better separation between the two test days than relaxing the temporal requirement. July 1 contained 104 sub-4000 observations, of which 92 (88.5%) met the 2/2 criterion, compared with only 10 of 317 (3.2%) on September 14.

The jump analysis also revealed substantial structural differences. On July 1, 108 of 204 nonzero changes (52.9%) were ≥500 m, compared with 28 of 385 (7.3%) on September 14. In addition, only 4 of July's 204 nonzero changes (2.0%) occurred between two sub-4000 readings, compared with 247 of 385 (64.2%) on September 14. The problematic day therefore appears dominated by abrupt excursions away from and back to the 4000-m ceiling, while the good day shows much more continuous variation below 4000.

I also examined what the 2/2 rule misses. The 12 remaining sub-4000 observations on July 1 occur in short clusters that prevent them from meeting the 2/2 requirement. This suggests that jump magnitude could potentially complement the temporal rule.

**Paper, figure, dataset, tool, script, or output shared:**
Python scripts implementing the 2/2, 2/1, and 1/1 rules; comparative plots and histograms for the July 1 and September 14 datasets; descriptive output comparing nonzero jumps, jump magnitude, and sub-4000 behavior. I also established a WinSCP connection to the littlestorm server through the UND VPN.

//Python Code for 2/2 Rule//
Counter = 0
Counter9999 = 0
for item in VisibilityField3:
    if item <4000 and Counter >=2 and Counter <= len(VisibilityField3)-3:
        if (VisibilityField3[Counter-2] == 4000 and VisibilityField3[Counter-1] == 4000 
            and VisibilityField3[Counter+1] == 4000 and VisibilityField3[Counter+2] == 4000):
            CleanedVisibilityField3[Counter] = 9999
            Counter9999 += 1
    Counter += 1

//Python Code for Count of Observations Less than 4K//
count4 = 0
for item in VisibilityField3:
    if item < 4000:
        count4 += 1


//Python Code for Jump Size and Jump Characterization//
JumpSize = []
JumpEnd = 0
JumpStart = 0
Non4000Jump = 0
for x in range(1,len(JumpsVisibilityField3)):
    JumpDifference = abs(JumpsVisibilityField3[x]-JumpsVisibilityField3[x-1])
    if JumpDifference != 0:
        JumpSize.append(JumpDifference)
        if JumpsVisibilityField3[x] == 4000:
            JumpEnd += 1
        elif JumpsVisibilityField3[x-1] == 4000:
            JumpStart += 1
        else:
            Non4000Jump += 1


**Question or blocker:**
I currently can connect to the historical sensor server and reach the MiniOFS directory, but I receive a permissions error when attempting to access the sensor data directory. I therefore cannot yet obtain additional historical files needed to determine whether the patterns observed in July 1 and September 14 generalize to other days. ***Update: I have access to the MiniOFS director and sensor files thanks to the help of Atmospheric Sciences team! (Thank you!) I successfully downloaded ten additional files and was able to read them in.***

**What I tried:**
I compared increasingly permissive temporal rules, examined the observations captured and missed by the 2/2 rule, and explored whether the size and structure of consecutive visibility changes provide additional information. I created histograms of jump magnitude and calculated the frequency of ≥500-m changes and whether transitions involved the 4000-m ceiling. I also connected to the server using WinSCP after determining that access requires the UND VPN.

**Next steps:**
  - Continue with access to additional historical MiniOFS files and apply the same rules and descriptive measures without further tuning to multiple good and potentially problematic days. This will test whether the differences found between July 1 and September 14 are reproducible. If they are, I will explore a possible multi-stage approach in which day-level characteristics identify potentially problematic files, the conservative 2/2 rule identifies isolated questionable observations, and jump magnitude helps identify short clusters that 2/2 misses. 
  - Explore the statistical outliers python code to identify how this might help with identifying erroneous data.
  - Begin thinking through how to automate the cleansing process - read a list of file names, iterate through and open each file - read the data, cleanse it, write the cleansed data to a few file.

====== 2026-10-06 ======

**What I completed:**
  * Expanded the Python analysis from individual MiniOFS files to batch processing of the historical data files.
  * Created functions to extract and separate sensor data and calculate descriptive statistics for each file.
  * Applied the 2-before/2-after rule across the historical dataset to identify potentially erroneous sub-4000 m observations.
  * Added statistics describing total observations, sub-4000 observations, jump frequency, jumps from 4000 m, transitions occurring entirely below 4000 m, jumps ≥500 m, and the proportion of sub-4000 observations identified by the 2/2 rule.
  * Created a pandas DataFrame and exported the results to Excel for easier review.
  * Identified substantial variation in the number of observations per file, including partial files, header-only files with zero observations, and files with substantially more observations than a typical day.    

**What I learned:**
  * The suspected erroneous behavior appears to have a distinctive temporal pattern: isolated or short excursions from the 4000 m ceiling, often involving large jumps, while legitimate reduced-visibility periods tend to contain sustained sequences of changing values below 4000 m.
  * The 2/2 rule identifies a high percentage of sub-4000 observations on many days during the suspected problem period, but a much smaller percentage during sustained low-visibility events. For example, July 1 was 88.46%, while September 14 and September 26 were 3.15% and 2.47%, respectively.    Pasted markdown    Pasted markdown
  * Data completeness and file structure need to be understood before calculating formal probability-of-detection or false-alarm statistics.
  * Gained additional Python experience with functions, nested dictionaries, exception handling, pandas DataFrames, batch file processing, and exporting results to Excel.

**Paper, figure, dataset, tool, script, or output shared:**
  * Python script for batch processing and descriptive analysis of historical MiniOFS files.
  * Excel table containing file-level descriptive statistics and results of the 2/2 potential-error rule.
  * Historical MiniOFS dataset reviewed through September 2026.

**Question or blocker:**
  * What accounts for the large variation in observations per file? A typical complete file contains approximately 2,800 observations, but some files contain only a few hundred, some contain zero observations, and some contain substantially more than a normal day (e.g., 19,751 observations on February 16).    Pasted markdown
  * What do the header-only/zero-observation files represent?
  * What are the exact dates of the wiring problem and repair?
  * What should be used as ground truth for evaluating probability of detection and false alarms?

**What I tried:**
  * Tested the 2/2 rule on multiple known or presumed good and bad days and then automated it across the historical files.
  * Compared several characteristics of the visibility data, including the proportion of observations below 4000 m, frequency of jumps, jump magnitude, and whether transitions occurred from the 4000 m ceiling or between two sub-4000 values.
  * Used exception handling to accommodate files with no observations or no jumps without stopping batch processing.
  * Examined several zero-observation files directly and confirmed that at least some contain only header information and no sensor observations.

**Next step:**
  * Review the statistical/outlier-filter Python code provided by Prof. Delene and compare its behavior with the current rule-based approach.
  * Discuss the newly identified file-structure/data-completeness questions and the historical timing of the wiring problem with Shawn.
  * Determine an appropriate ground-truth strategy and statistics for evaluating probability of detection and false alarms.
  * Continue developing the automated workflow toward producing cleansed output files while preserving the original data.

====== 2026-10-15 ======

===== What I completed this week: =====

I continued investigating possible sources of independent ground-truth data for evaluating the Clifford Hall MiniOFS visibility sensor. Our goal is to distinguish genuine periods of reduced atmospheric visibility from potentially erroneous sensor readings to be able to calculate probability of detection stats for our proposed data marking methods.

I investigated three possible reference sources:

  * Gorman Field MiniOFS: Obtained and examined observations from a second MiniOFS sensor approximately 15–20 miles west of Clifford Hall. Compared visibility patterns on selected dates, including September 14, 21, and 25.
  * Grand Forks International Airport (GFK) ASOS: Located and downloaded historical airport visibility observations through the Iowa Environmental Mesonet. Developed Python code to extract observations for selected dates and successfully examined March 16, when all 24 airport observations reported 10 statute miles of visibility.
  * North Dakota Agricultural Weather Network (NDAWN): Identified Grand Forks 3S (Station 26), near UND, as a possible source of independent meteorological observations, including temperature, relative humidity, dew point, wind, and solar radiation.
  * Still TBD: EFM on Clifford Hall and at Gorman Field

I also developed code to automate the creation of Python dictionaries for the Clifford Hall and Gorman Field datasets from all source files for 2026. Observations are now organized by their internal date stamps rather than by individual files, allowing multiple files to contribute observations to the same calendar date. This is an improvement over the past week.

I completed a preliminary inventory of the two datasets:

  * Gorman Field: 203 unique filename dates, 195 dates with observations, and 541,471 observations through approximately October 15.
  * Clifford Hall: 210 unique filename dates, 177 dates with observations, and 437,037 observations through September 29. The unique filename-date count slightly underestimates the dates represented because some early observations were stored in rolling files.
  * Average observations per day with data: 2,776.77 at Gorman Field and 2,469.14 at Clifford Hall.
  * Identified 128 dates with observations from both sensors.

I also discussed with Shawn the possibility of examining Gorman Field electric field mill (EFM) measurements to determine whether changes in atmospheric electric fields correlate with or precede changes in visibility. This may be the best source of ground truth since the sensor is collocated geographically with the minOFS visibility sensor.

===== What I learned: =====


Ground truth is more complicated than simply comparing two instruments. Gorman Field provides measurements from a similar sensor, but the stations are geographically separated, and genuine local differences in visibility are possible.

The airport ASOS provides an independent visibility measurement, but its observations are generally hourly, compared with the approximately 30-second MinOFS sampling interval. This limits its usefulness for validating individual short-duration visibility changes. Also both MinOFS and ASOS visibility measurements have upper reporting limits. A MinOFS reading of 4,000 meters represents visibility at or above its reporting ceiling, while an ASOS reading of 10 miles may also represent an upper reporting limit. Neither necessarily represents the exact atmospheric visibility. Historical GFK ASOS visibility dataset obtained through the Iowa Environmental Mesonet.

NDAWN does not directly measure visibility, but relative humidity, temperature, dew point, and wind measurements could help determine whether conditions were consistent with fog or other causes of reduced visibility. This makes using NDAWN more complicated. 

Comparing selected Clifford Hall and Gorman Field plots showed both agreements and disagreements. For example, September 21 showed generally good agreement at maximum visibility, while September 14 showed more substantial reductions at Clifford Hall than at Gorman Field. On September 25, both stations recorded reduced visibility, but their patterns differed.

These comparisons suggest that isolated reductions, abrupt changes, and differences between stations may be useful indicators of questionable observations, but none independently proves that a reading is erroneous.

I also earned improved my ability to use Python dictionaries to organize large datasets by date, including .setdefault(), .append(), .keys(), .values(), and .items(). Also learned how to handle malformed data records.

Recognized the importance of distinguishing dates represented by filenames from dates containing actual observations, as well as checking data completeness before comparing daily statistics.

===== Paper, figure, dataset, tool, script, or output shared: =====


Create Dictionaries.py — Python script for reading and aggregating Clifford Hall and Gorman Field MiniOFS observations.

{{ :atmos:dsp:workspace:fall2026:pythoncode20261015.docx |}}

Comparative visibility plots for selected Clifford Hall and Gorman Field dates.

Previously generated daily descriptive-statistics spreadsheets for both stations, which will be recalculated using the aggregated observations.

===== Question or blocker: =====


  * How can we establish sufficiently reliable ground-truth periods when the available instruments have different sampling frequencies, reporting limits, and locations?
  * How much simultaneous coverage exists within the 128 dates when both MiniOFS sensors recorded observations?
  * Earlier daily descriptive statistics may be incomplete because observations from multiple files associated with the same date were not always combined.
  * Can Gorman Field EFM measurements provide useful information about visibility changes, either through correlation or by providing an advance indication of changes?

===== What I tried: =====

  * Compared MiniOFS visibility plots from both stations on selected dates to identify agreement, disagreement, and potentially anomalous behavior.
  * Retrieved and parsed GFK ASOS observations, including a test of March 16 data, to explore their potential as an independent visibility reference.
  * Investigated NDAWN station availability and the meteorological variables that could support interpretation of MiniOFS measurements.
  * Developed a Python approach for grouping all observations by the actual dates recorded inside the data files, rather than relying on filenames.
  * Added checks to exclude malformed records with fewer than four fields.
  * Used dictionary operations to calculate observation counts, identify overlapping dates, and evaluate the completeness of the datasets.

===== Next step: =====

  * Finalize the Clifford Hall and Gorman Field dictionaries, including chronological sorting and checks for duplicate timestamps.
  * Recalculate daily descriptive statistics using all observations for each date, then compare the results for both stations, particularly on the 128 overlapping dates.
  * Begin identifying candidate ground-truth periods using agreement between MiniOFS observations and GFK ASOS reports, with NDAWN meteorological observations as additional supporting evidence.
  * Access the Gorman Field EFM dataset, determine its structure and sampling frequency, and produce initial time-aligned plots of electric field and visibility measurements.
  * Investigate whether changes in electric field measurements correlate with or precede changes in reported visibility.


====== 2026-10-21 ======

===== What I completed this week =====

  * Successfully completed dictionaries of daily aggregated data and exported descriptive statistics for each day to excel files.
  * Reviewed the daily statistics and comparisons between the two stations, including the frequency of observations below 4,000 meters and the number of observations flagged as potentially erroneous.
  * Met with Shawn to discuss the results of the 2−/2+ screening rule, which identifies isolated observations below 4,000 meters when the two preceding and two following observations are all 4,000 meters.
  * Examined how applying the rule changes the number of days containing only 4,000-meter observations. The rule identifies an additional 23 such days at Clifford Hall and 34 at Gorman Field.
  * Investigated published research and technical documentation concerning MiniOFS measurement artifacts and methods for processing visibility data.

===== What I learned =====

  * The 2−/2+ rule identifies potentially erroneous observations, but the pattern alone does not establish that an observation is incorrect.
  * Although Clifford Hall experienced a known wiring problem, no comparable physical cause has been identified for the suspicious observations at Gorman Field.
  * The frequency of observations flagged by the rule may depend on weather conditions. During sustained reduced visibility, isolated erroneous measurements may not satisfy the rule because the surrounding observations are also below 4,000 meters.
  * Rolling median filters are another established method for identifying short-duration anomalies. Depending on the window size, they can identify sequences of suspicious observations that the 2−/2+ rule would not detect.
  * Published MiniOFS studies use various approaches to data processing, including temporal averaging and removal of artifacts. However, I have not found evidence that the specific isolated downward spikes observed in this dataset are a documented characteristic of the MiniOFS.

====== Paper, figure, dataset, tool, script, or output shared ======

  * Daily statistical summaries for Clifford Hall and Gorman Field, exported to Excel.
  * Comparison of the 128 dates with observations from both stations.
  *  Results of applying the 2−/2+ screening rule, including the number of additional days classified as containing only 4,000-meter observations.
  * Reviewed published MiniOFS studies and documentation describing visibility measurement and data-processing methods.

====== Question or blocker ======

  * The underlying cause of isolated sub-4,000-meter observations remains uncertain, particularly at Gorman Field, where no physical sensor problem has been identified. Publicly available materials on the sensor as well as visibility sensors in general describe common problems, but there is a lack of specific discussions of short term spikes for MiniOFS. This may be because industry / government implementations of MiniOFS are using a longer sampling timeframe (5 min or 10 min) and this is smoothing out the spikes we are seeing in 30 second sample timeframe.
  * It is also unclear how reliably the 2−/2+ rule distinguishes erroneous observations from genuine short-duration reductions in visibility. This is an area I am working on putting more rigor around with probability of detection methods.

===== What I tried =====

  * Investigated whether other MiniOFS users have reported similar observations or published methods for filtering them.
  * Considered the potential advantages and limitations of rolling median filtering compared with the existing 2−/2+ rule based on literature review. I will likely test this experimentally with the data from Clifford Hall and Gorman Field.

===== Next step =====

  * Unordered List ItemBegin comparing the remaining sub-4,000-meter observations with independent meteorological measurements, including precipitation and electric field mill data. Specifically, for the days that have sub 4K observations after the 2+/2- rule has been applied and those observations meeting the rule criteria are "corrected" to 4K, are there independent sources that confirm a reduction in visibility for those days.
  * Examine whether observations removed by the 2−/2+ rule correspond to genuine weather events.
  * Investigate the distribution of flagged observations over time, accounting for differences in the opportunity to detect isolated anomalies during clear and reduced-visibility conditions.
  * Consider comparing the results of the 2−/2+ rule with a rolling median filter.
  * Investigate if the aggregation of data into daily files has left "gaps" (due to lack of sensor observations) that may impact the application of the 2-/2+ rule.
  * Contact researchers at the University of California, Santa Cruz (UCSC), and potentially the University of Connecticut (UConn), to request samples of their raw MiniOFS observations. Apply the same 2−/2+ screening rule to these independent datasets to determine whether similar isolated downward spikes occur at other installations.