Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation

Deep Neural Networks (DNNs) have become the tool of choice for machine learning practitioners today. One important aspect of designing a neural network is the choice of the activation function to be used at the neurons of the different layers. In this work, we introduce a four-output activation func...

Full description

Saved in:
Bibliographic Details
Main Authors: Chaity Banerjee, Tathagata Mukherjee, Eduardo Pasiliao Jr.
Format: Article
Language:English
Published: Tsinghua University Press 2020-06-01
Series:Big Data Mining and Analytics
Subjects:
Online Access:https://www.sciopen.com/article/10.26599/BDMA.2019.9020024
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1832572925434134528
author Chaity Banerjee
Tathagata Mukherjee
Eduardo Pasiliao Jr.
author_facet Chaity Banerjee
Tathagata Mukherjee
Eduardo Pasiliao Jr.
author_sort Chaity Banerjee
collection DOAJ
description Deep Neural Networks (DNNs) have become the tool of choice for machine learning practitioners today. One important aspect of designing a neural network is the choice of the activation function to be used at the neurons of the different layers. In this work, we introduce a four-output activation function called the Reflected Rectified Linear Unit (RReLU) activation which considers both a feature and its negation during computation. Our activation function is "sparse", in that only two of the four possible outputs are active at a given time. We test our activation function on the standard MNIST and CIFAR-10 datasets, which are classification problems, as well as on a novel Computational Fluid Dynamics (CFD) dataset which is posed as a regression problem. On the baseline network for the MNIST dataset, having two hidden layers, our activation function improves the validation accuracy from 0.09 to 0.97 compared to the well-known ReLU activation. For the CIFAR-10 dataset, we use a deep baseline network that achieves 0.78 validation accuracy with 20 epochs but overfits the data. Using the RReLU activation, we can achieve the same accuracy without overfitting the data. For the CFD dataset, we show that the RReLU activation can reduce the number of epochs from 100 (using ReLU) to 10 while obtaining the same levels of performance.
format Article
id doaj-art-166ca277897d4c9791286ccd9798d767
institution Kabale University
issn 2096-0654
language English
publishDate 2020-06-01
publisher Tsinghua University Press
record_format Article
series Big Data Mining and Analytics
spelling doaj-art-166ca277897d4c9791286ccd9798d7672025-02-02T05:59:18ZengTsinghua University PressBig Data Mining and Analytics2096-06542020-06-013210212010.26599/BDMA.2019.9020024Feature Representations Using the Reflected Rectified Linear Unit (RReLU) ActivationChaity Banerjee0Tathagata Mukherjee1Eduardo Pasiliao Jr.2<institution content-type="dept">Department of Idustrial & Systems Engineering</institution>, <institution>University of Central Florida</institution>, <city>Orlando</city>, <state>FL</state> <postal-code>32816-2368</postal-code>, <country>USA</country>.<institution content-type="dept">Department of Computer Science</institution>, <institution>University of Alabama in Huntsville</institution>, <city>Huntsville</city>, <state>AL</state> <postal-code>35806</postal-code>, <country>USA</country>.<institution content-type="dept">Air Force Research Labs, United States Air Force</institution>, <institution>Eglin Air Force Base</institution>, <city>Shalimar</city>, <state>FL</state> <postal-code>32579</postal-code>, <country>USA</country>.Deep Neural Networks (DNNs) have become the tool of choice for machine learning practitioners today. One important aspect of designing a neural network is the choice of the activation function to be used at the neurons of the different layers. In this work, we introduce a four-output activation function called the Reflected Rectified Linear Unit (RReLU) activation which considers both a feature and its negation during computation. Our activation function is "sparse", in that only two of the four possible outputs are active at a given time. We test our activation function on the standard MNIST and CIFAR-10 datasets, which are classification problems, as well as on a novel Computational Fluid Dynamics (CFD) dataset which is posed as a regression problem. On the baseline network for the MNIST dataset, having two hidden layers, our activation function improves the validation accuracy from 0.09 to 0.97 compared to the well-known ReLU activation. For the CIFAR-10 dataset, we use a deep baseline network that achieves 0.78 validation accuracy with 20 epochs but overfits the data. Using the RReLU activation, we can achieve the same accuracy without overfitting the data. For the CFD dataset, we show that the RReLU activation can reduce the number of epochs from 100 (using ReLU) to 10 while obtaining the same levels of performance.https://www.sciopen.com/article/10.26599/BDMA.2019.9020024deep learningfeature spaceapproximationsmulti-output activationsrectified linear unit (relu)
spellingShingle Chaity Banerjee
Tathagata Mukherjee
Eduardo Pasiliao Jr.
Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation
Big Data Mining and Analytics
deep learning
feature space
approximations
multi-output activations
rectified linear unit (relu)
title Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation
title_full Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation
title_fullStr Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation
title_full_unstemmed Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation
title_short Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation
title_sort feature representations using the reflected rectified linear unit rrelu activation
topic deep learning
feature space
approximations
multi-output activations
rectified linear unit (relu)
url https://www.sciopen.com/article/10.26599/BDMA.2019.9020024
work_keys_str_mv AT chaitybanerjee featurerepresentationsusingthereflectedrectifiedlinearunitrreluactivation
AT tathagatamukherjee featurerepresentationsusingthereflectedrectifiedlinearunitrreluactivation
AT eduardopasiliaojr featurerepresentationsusingthereflectedrectifiedlinearunitrreluactivation