This is an old revision of the document!
Justin Abold-LaBreche - Data Skills Pathway Workspace
Student Information Name: Justin Abold-LaBreche Major: Electrical Engineering and Cybersecurity Engineering Year: Sophomore Semester: Fall 2026 Email: justin.abold@und.edu
Mentor Information Mentor Name: Shawn Wagner Mentor Department / Program: Atmospheric Sciences Mentor Email: shawn.wagner@und.edu
Meeting Schedule: Wednesdays, 1PM Central / 2PM Eastern
Project Interest Sensors Engineering Cybersecurity
Project Question or Problem Historical visibility-sensor data contain anomalous readings during periods when visibility was otherwise clear. The sensor's wiring was corrected in early September 2026, and subsequent data no longer exhibit the known wiring-related errors. The challenge is to develop a reliable method for identifying erroneous observations in the historical data and marking them with a value of 9999, allowing researchers to distinguish invalid measurements while preserving the remainder of the historical dataset for analysis.
Project Brief
Project Title: Identification and Cleansing of Anomalous Visibility Sensor Data
Project Goal: Develop and test a method for identifying erroneous observations in historical visibility-sensor data caused by a known sensor wiring problem. Erroneous observations will be marked with a value of 9999 so they can be distinguished from valid measurements while preserving the historical dataset for research use.
There is a potential longer term goal of automating the data cleansing process.
Data Source: Historical data from the IR System MiniBSV visibility sensor located on the roof of Clifford Hall at the University of North Dakota. Daily sensor files contain a timestamp and three data fields, including solar irradiance (Field 1), extinction coefficient (Field 2), and horizontal visibility in meters (Field 3). The sensor's maximum reported visibility is 4,000 meters.
Method or Tools: Python will be used to read, parse, analyze, and eventually cleanse the historical sensor files. Matplotlib is being used to visualize relationships among the sensor fields and changes over time. Initial analysis compares known good and problematic data and tests simple temporal rules for identifying anomalous visibility observations. For example, one candidate rule flags a sub-4,000-meter observation when the two preceding and two following observations all report 4,000-meter visibility. Candidate rules will be tested against additional historical and post-repair data before determining an appropriate cleansing method.
Expected Output: A Python-based method for identifying and marking erroneous historical visibility observations with 9999. If feasible within the project timeframe, the longer-term objective is to automate the process so that the agreed-upon cleansing procedure can be applied across multiple historical data files rather than processing each file individually.
Why this project matters: Historical visibility data may be valuable for atmospheric-science research, but erroneous measurements caused by the previous wiring problem can compromise subsequent analysis if they cannot be distinguished from legitimate reductions in visibility caused by actual atmospheric conditions. Developing a reproducible method for identifying and marking those observations would improve the usability of the historical dataset while retaining valid measurements.
The project also supports my longer-term interest in cyber-physical systems, particularly power systems, smart grids, and industrial control systems. Although the application here is an atmospheric sensor rather than a power-system device, the underlying problem is similar: using data from a physical sensor to distinguish genuine changes in the physical environment from anomalous measurements caused by equipment or system faults. I am using the Data Skills Pathway to develop the Python, data-analysis, and anomaly-detection skills that I hope to apply in future undergraduate research and eventually to cyber-physical power-system research.
Week 1 Update Date: YYYY-MM-DD
What I completed this week:
What I learned:
Paper, figure, dataset, tool, script, or output shared:
Question or blocker:
What I tried:
Next step:
Week 2 Update 2026-09-29
What I completed this week: I developed and tested three candidate temporal filtering rules for identifying questionable visibility observations: 2-before/2-after (2/2), 2-before/1-after (2/1), and 1-before/1-after (1/1). I tested the rules against July 1 (problematic data) and September 14 (good data). The 2/2 rule identified 92 observations on July 1 versus 10 on September 14; relaxing the rule to 2/1 produced 94 versus 13, and 1/1 produced 96 versus 24. I also conducted preliminary analysis of the magnitude of changes between consecutive visibility readings and gained access through WinSCP/VPN to the historical sensor server and MiniOFS directory, although I do not yet have permission to access the data files.
What I learned: The preliminary results suggest that the conservative 2/2 rule provides better separation between the two test days than relaxing the temporal requirement. July 1 contained 104 sub-4000 observations, of which 92 (88.5%) met the 2/2 criterion, compared with only 10 of 317 (3.2%) on September 14. The jump analysis also revealed substantial structural differences. On July 1, 108 of 204 nonzero changes (52.9%) were ≥500 m, compared with 28 of 385 (7.3%) on September 14. In addition, only 4 of July's 204 nonzero changes (2.0%) occurred between two sub-4000 readings, compared with 247 of 385 (64.2%) on September 14. The problematic day therefore appears dominated by abrupt excursions away from and back to the 4000-m ceiling, while the good day shows much more continuous variation below 4000.
I also examined what the 2/2 rule misses. The 12 remaining sub-4000 observations on July 1 occur in short clusters that prevent them from meeting the 2/2 requirement. This suggests that jump magnitude could potentially complement the temporal rule.
Paper, figure, dataset, tool, script, or output shared: Python scripts implementing the 2/2, 2/1, and 1/1 rules; comparative plots and histograms for the July 1 and September 14 datasets; descriptive output comparing nonzero jumps, jump magnitude, and sub-4000 behavior. I also established a WinSCP connection to the littlestorm server through the UND VPN.
Python Code for 2/2 Rule Counter = 0 Counter9999 = 0 for item in VisibilityField3:
if item <4000 and Counter >=2 and Counter <= len(VisibilityField3)-3:
if (VisibilityField3[Counter-2] == 4000 and VisibilityField3[Counter-1] == 4000
and VisibilityField3[Counter+1] == 4000 and VisibilityField3[Counter+2] == 4000):
CleanedVisibilityField3[Counter] = 9999
Counter9999 += 1
Counter += 1
Python Code for Count of Observations Less than 4K count4 = 0 for item in VisibilityField3:
if item < 4000:
count4 += 1
Python Code for Jump Size and Jump Characterization JumpSize = [] JumpEnd = 0 JumpStart = 0 Non4000Jump = 0 for x in range(1,len(JumpsVisibilityField3)):
JumpDifference = abs(JumpsVisibilityField3[x]-JumpsVisibilityField3[x-1])
if JumpDifference != 0:
JumpSize.append(JumpDifference)
if JumpsVisibilityField3[x] == 4000:
JumpEnd += 1
elif JumpsVisibilityField3[x-1] == 4000:
JumpStart += 1
else:
Non4000Jump += 1
Question or blocker: I currently can connect to the historical sensor server and reach the MiniOFS directory, but I receive a permissions error when attempting to access the sensor data directory. I therefore cannot yet obtain additional historical files needed to determine whether the patterns observed in July 1 and September 14 generalize to other days.
What I tried: I compared increasingly permissive temporal rules, examined the observations captured and missed by the 2/2 rule, and explored whether the size and structure of consecutive visibility changes provide additional information. I created histograms of jump magnitude and calculated the frequency of ≥500-m changes and whether transitions involved the 4000-m ceiling. I also connected to the server using WinSCP after determining that access requires the UND VPN.
Next step: Obtain access to additional historical MiniOFS files and apply the same rules and descriptive measures without further tuning to multiple good and potentially problematic days. This will test whether the differences found between July 1 and September 14 are reproducible. If they are, I will explore a possible multi-stage approach in which day-level characteristics identify potentially problematic files, the conservative 2/2 rule identifies isolated questionable observations, and jump magnitude helps identify short clusters that 2/2 misses.