techniques may perform well, it is rarely the case, so you need a few backup.
Identifying the Type of Missingness
The first step to implementing an effective imputation strategy is identifying why the values are missing. Even though each case is unique, missingness can be grouped into three broad categories:
Missing Completely At Random (MCAR): this is a genuine case of data missing randomly. Examples are sudden mistakes in data entry, temporary sensor failures, or generally missing data that is not associated with any outside factor. The amount of missingness is low.
Missing At Random (MAR): this is a broader case of MCAR. Even though missing data may seem random at first glance, it will have some systematic relationship with the other observed features — for example — data missing from observational equipment during scheduled maintenance breaks. The number of null values may vary.
Missing Not At Random (MNAR):
Due to temporary inabilities of the models to match the real values with the predictions, random spikes can arise in the “alarm” time series. Thus, in order to make the alarm system more reliable, we use a two-level structure: this first alarm, the one defined above, is merely a warning signal and is processed again to produce a more accurate second level alarm signal. Using a Moving Aggregation node, the moving averages are calculated on a backward window of 21 samples of the level 1 alarm (warning) signals. This moving average operation smooths out all short random spikes in the level 1 alarm time series, retaining only the ones that persist over time.
Bookmarks on this page are managed by an admin user. 