This repo contains the project for the Big Data course at NYU.
The count.py, clean_data.py, and LOF.py are written in python Spark.
count.py counts the number of nulls and computes percentage of nulls for each column in a given dataset. The null values are user-specified. By calling -getmerge on Hadoop streaming, the function returns a single csv file listing each column name, followed by the number of nulls in that column, and percentage of nulls for that column.
clean_data.py returns a cleaned version of given dataset that does not contain any null values in any column.
LOF.py computes the LOF score for each instance in a dataset. A value close to 1 or less than 1 indicate a normal instance.
loop_spark.py and loop_spark_with_sampling.py are pyspark implementations of the local outlier probabilities algorithm to detect outliers.
ABOD.py and ABOD_sampling.py are both angle based outlier detection methods that one is combined with our novel sampling technique in order to improve time complexity and one without.
Visualization.py is how we implement t-SNE and PCA to help visualize high-dimensional dataset and identify outliers.
To run the code, please follow Instruction.txt.