NEURAL NET TRAINING SPEED USING A MODIFIED ELLIOT SIGMOID
The 2012b NN TBX features the option of replacing the TANSIG activation
function with the ELLIOT SIGMOID for faster training.
% speed-and-memory-optimizations.htmlFast Elliot Sigmoid
% Some simple computing hardware might not support the exponential function
% directly, and software implementations can be slow. The Elliot sigmoid
% elliotsig function performs the same role as the symmetric sigmoid tansig
% function, but avoids the exponential function.
% Here, the times to execute elliotsiggh and tansig are compared.
% elliotsig is approximately four times faster on the test system.
Deleted computation time comparison (see the website)
% However, while simulation is faster with elliotsig, training is not
% guaranteed to be faster, due to the different shapes of the two transfer
% functions. Here, 10 networks are each trained for tansig and elliotsig,
% but training times vary significantly even on the same problem with the
% same network.
Deleted training time comparison (see the website)
The Elliot Sigmoid has the shape
elliotsig(x) = x ./ (1 + abs(x) );
tansig(x) = 2 ./ (1 + exp(-2*x)) - 1;
The comparison plot shows that the S-shaped elliotsig approaches the
asymptotes [ -1, +1 ] more slowly. Therefore, even though it is computed
faster, it generally takes more epochs (iterations) to converge.
In order to mitigate this effect, I compared tansig with
elliotsigs = (s*x) ./ ( 1 + abs(s*x) );
x = 3*randn(1,e7); s=1:10;
Using the above equations instead of calling the functions, I found that
the minimum RMSE between tansig and elliotsigs occured when s = 4. In
RMSE4 = 0.1051
RMSE1 = 0.2285
For an impressive visual comparison, plot tansig, elliotsig and elliotsig4 on
the same graph.
A fast implementaion of elliotsig4 is
elliotsig4(x) = x ./ (0.25 + abs(x) );
which is just as fast as the original elliotsig.
To make a fair training comparison, I created a MATLAB-type elliotsig4
function by modifying tansig. The resulting 225 lines of code will mask the
the basic difference in computation times of the two formulae.
I will post results later.
Hope this helps.
You can think of your watch list as threads that you have bookmarked.
You can add tags, authors, threads, and even search results to your watch list. This way you can easily keep track of topics that you're interested in. To view your watch list, click on the "My Newsreader" link.
To add items to your watch list, click the "add to watch list" link at the bottom of any page.
To add search criteria to your watch list, search for the desired term in the search box. Click on the "Add this search to my watch list" link on the search results page.
You can also add a tag to your watch list by searching for the tag with the directive "tag:tag_name" where tag_name is the name of the tag you would like to watch.
To add an author to your watch list, go to the author's profile page and click on the "Add this author to my watch list" link at the top of the page. You can also add an author to your watch list by going to a thread that the author has posted to and clicking on the "Add this author to my watch list" link. You will be notified whenever the author makes a post.
To add a thread to your watch list, go to the thread page and click the "Add this thread to my watch list" link at the top of the page.
No tags are associated with this thread.
A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.
Anyone can tag a thread. Tags are public and visible to everyone.
The newsgroups are a worldwide forum that is open to everyone. Newsgroups are used to discuss a huge range of topics, make announcements, and trade files.
Discussions are threaded, or grouped in a way that allows you to read a posted message and all of its replies in chronological order. This makes it easy to follow the thread of the conversation, and to see what’s already been said before you post your own reply or make a new posting.
Newsgroup content is distributed by servers hosted by various organizations on the Internet. Messages are exchanged and managed using open-standard protocols. No single entity “owns” the newsgroups.
There are thousands of newsgroups, each addressing a single topic or area of interest. The MATLAB Central Newsreader posts and displays messages in the comp.soft-sys.matlab newsgroup.
You can use the integrated newsreader at the MATLAB Central website to read and post messages in this newsgroup. MATLAB Central is hosted by MathWorks.
Messages posted through the MATLAB Central Newsreader are seen by everyone using the newsgroups, regardless of how they access the newsgroups. There are several advantages to using MATLAB Central.
Your MATLAB Central account is tied to your MathWorks Account for easy access.
Use the Email Address of Your Choice
The MATLAB Central Newsreader allows you to define an alternative email address as your posting address, avoiding clutter in your primary mailbox and reducing spam.
Most newsgroup spam is filtered out by the MATLAB Central Newsreader.
Messages can be tagged with a relevant label by any signed-in user. Tags can be used as keywords to find particular files of interest, or as a way to categorize your bookmarked postings. You may choose to allow others to view your tags, and you can view or search others’ tags as well as those of the community at large. Tagging provides a way to see both the big trends and the smaller, more obscure ideas and applications.
Setting up watch lists allows you to be notified of updates made to postings selected by author, thread, or any search variable. Your watch list notifications can be sent by email (daily digest or immediate), displayed in My Newsreader, or sent via RSS feed.
You can also select a location from the following list: