About Presenter James Weaver

Java Champion, JavaOne Rockstar, plays well with others, etc :-)

Author of several Java/JavaFX/RaspPi books

Developer Advocate & International Speaker for

@JavaFXpert

Mission: "Transform how the world builds software"

@JavaFXpert

Mission: "Transform how the world builds software"

@JavaFXpert

Some Pivotal involvement in machine learning

Example: Real-time Twitter Sentiment Analytics with TensorFlow and Spring Cloud Dataflow

@JavaFXpert

From introductory video in Machine Learning course (Stanford University & Coursera) taught by Andrew Ng.

@JavaFXpert

Self-driving cars

@JavaFXpert

Generating image descriptions

@JavaFXpert

Supervised Learning

@JavaFXpert

Supervised learning regression problem

(from Andrew Ng’s Machine Learning course)

@JavaFXpert

Unsupervised Learning

@JavaFXpert

Unsupervised learning finds structure in unlabeled data

(e.g. market segment discovery, and social network analysis)

@JavaFXpert

Reinforcement Learning

@JavaFXpert

**AlphaGo is a recent reinforcement learning success story**

Source: https://gogameguru.com/i/2016/03/AlphaGo-Lee-Sedol-game-3-game-over.jpg

@JavaFXpert

Supervised Learning

(Let's dive in now)

@JavaFXpert

Supervised learning classification problem

(using the Iris flower data set)

@JavaFXpert

Visualizing Iris dataset with TensorFlow tool

@JavaFXpert

projector.tensorflow.org

Modeling the brain works well with machine learning
(ya think?)

(inputs)

(output)

@JavaFXpert

Anatomy of an Artificial Neural Network

(aka Deep Belief Network when multiple hidden layers)

@JavaFXpert

Neural net visualization app (uses Spring and DL4J)

github.com/JavaFXpert/visual-neural-net-server

github.com/JavaFXpert/ng2-spring-websocket-client

@JavaFXpert

Entering feature values for prediction (classification)

@JavaFXpert

Simple neural network trained for XOR logic

forward propagation

@JavaFXpert

Feedforward calculations with XOR example

For each layer:

Multiply inputs by weights:

(1 x 8.54) + (0 x 8.55) = 8.54

Add bias:

8.54 + (-3.99) = 4.55

Use sigmoid activation function:

1 / (1 + e

-4.55

) = 0.99

@JavaFXpert

Excellent video on neural networks

Deep Learning, Part 1 (3Blue1Brown)

@JavaFXpert

Simple neural network trained for XOR logic

back propagation (minimize cost function)

@JavaFXpert

https://github.com/katharinebeaumont/Linear-Regression

@katharineCodes

Lab Exercise / Visualizing Gradient Descent:

Linear Regression app developed by Katharine Beaumont

@JavaFXpert

Make Your Own Neural Network (book)

Contains details on how weights and biases are adjusted during back propagation

@JavaFXpert

amazon.com/Make-Your-Own-Neural-Network-ebook/dp/B01EER4Z4G

Excellent video on gradient descent

Deep Learning, Part 2 (3Blue1Brown)

@JavaFXpert

Visual Neural Network application architecture

Spring makes REST services and WebSockets easy as π

@JavaFXpert

The app leverages machine learning libraries found at deeplearning4j.org

@JavaFXpert

Code that configures our speed dating neural net

@JavaFXpert

To quickly create a Spring project, visit start.spring.io

@JavaFXpert

Lab Exercise:

@JavaFXpert

playground.tensorflow.org

For each of the four dataset icons (Circle, Exclusive Or, Gaussian, and Spiral):

Select only the X1 & X2 features
Modify the hyperparameters in such a way that minimizes the number of Epochs required to make the Test loss and Training loss each <= 0.009
Tweet screenshot with your lowest Epochs result tagging @JavaFXpert in the message.

Practice tuning neural network hyperparameters

Is Optimizing your Neural Network a Dark Art ?

Excellent article by Preetham V V on neural networks and choosing hyperparameters

@JavaFXpert

Various Neural Networks

Convolutional Neural Network for recognizing images

@JavaFXpert

cs231n.stanford.edu/

Convolutional neural network architecture

@JavaFXpert

adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/

[by Adit Deshpande]

Peeking into a convolutional neural network

http://scs.ryerson.ca/~aharley/vis/ [by Adam Harley]

@JavaFXpert

Time series prediction with neural networks

What is happening? What is most likely to happen next?

@JavaFXpert

This is a job for a Recurrent Neural Network

What is happening? What is most likely to happen next?

@JavaFXpert

Recurrent Neural Network

@JavaFXpert

sciencedirect.com/science/article/pii/S088523081400093X

vs. traditional feed-forward network

Music composition with an RNN

From: Music and Art Generation using Machine Learning | Curtis Hawthorne

@JavaFXpert

Predicting the most likely next note

From: Music and Art Generation using Machine Learning | Curtis Hawthorne

@JavaFXpert

Playing a duet with neural networks

https://aiexperiments.withgoogle.com/ai-duet/view/

@JavaFXpert

Playing a duet with neural networks

https://aiexperiments.withgoogle.com/ai-duet/view/

@JavaFXpert

Unsupervised Learning

@KatharineCodes @JavaFXpert

(Let's dive in now)

Using unsupervised learning to map artworks

@JavaFXpert

artsexperiments.withgoogle.com/tsnemap

Euclidian distance for high-dimensional vectors

@JavaFXpert

http://www.ltcconline.net/greenl/courses/107/vectors/vect.htm

Using unsupervised learning to map words

@JavaFXpert

The amazing power of word vectors - Adrian Colyer

word2vec vector representations of words

Using unsupervised learning to map words

@JavaFXpert

The amazing power of word vectors - Adrian Colyer

word2vec vector offsets for gender relationships

Using unsupervised learning to map words

@JavaFXpert

The amazing power of word vectors - Adrian Colyer

word2vec vector offsets for plural relationships

Using unsupervised learning to map words

@JavaFXpert

The amazing power of word vectors - Adrian Colyer

word2vec vector arithmetic

@JavaFXpert

The amazing power of word vectors - Adrian Colyer

King – Man + Woman = Queen

Visualizing word2vec words & points

@JavaFXpert

using Tensorflow Embedding Projector

Inspecting word embeddings

@JavaFXpert

wevi: Word Embedding Visual Inspector by Xin Rong

Reinforcement Learning

(Let's dive in now)

@JavaFXpert

Reinforcement Learning tabula rasa

DeepMind AlphaGo Zero video

@JavaFXpert

Nature article

Using BURLAP for Reinforcement Learning

burlap.cs.brown.edu

@JavaFXpert

Learning to Navigate a Grid World with Q-Learning

@JavaFXpert

Rules of this Grid World

Agent may move left, right, up, or down (actions)
Reward is 0 for each move
Reward is 5 for reaching top right corner (terminal state)
Agent can't move into a wall or off-grid
Agent doesn't have a model of the grid world. It must discover as it interacts.

Challenge: Given that there is only one state that gives a reward, how can the agent work out what actions will get it to the reward?

(AKA the credit assignment problem)

Goal of an episode is to maximize total reward

@JavaFXpert

This Grid World's MDP (Markov Decision Process)

In this example, all actions are deterministic

@JavaFXpert

Agent learns optimal policy from interactions with the environment (s, a, r, s')

Source: http://www.mdpi.com/sensors/sensors-15-06668/article_deploy/html/images/sensors-15-06668-g002-1024.png

@JavaFXpert

Visualizing training episodes

From BasicBehavior example in https://github.com/jmacglashan/burlap_examples

@JavaFXpert

Expected future discounted rewards, and polices

@JavaFXpert

This example used discount factor 0.9

Low discount factors cause agent to prefer immediate rewards

@JavaFXpert

Exploration vs. Exploitation

How often should the agent try new paths vs. greedily taking known paths?

@JavaFXpert

Q-Learning approach to reinforcement learning

	Left	Right	Up	Down
...
2, 7	2.65	4.05	0.00	3.20
2, 8	3.65	4.50	4.50	3.65
2, 9	4.05	5.00	5.00	4.05
2, 10	4.50	4.50	5.00	3.65
...

Q-Learning table of expected values (cumulative discounted rewards) as a result of taking an action from a state and following an optimal policy. Here's an explanation of how calculations in a Q-Learning table are performed.

Actions

States

@JavaFXpert

Tic-Tac-Toe with Reinforcement Learning

Learning to win from experience rather than by being trained

@JavaFXpert

Inspired by the Tic-Tac-Toe Example section...

...of Reinforcement Learning: An Introduction

@JavaFXpert

Tic-Tac-Toe Learning Agent and Environment

X

O

Our learning agent is the "X" player, receiving +5 for winning, -5 for losing, and -1 for each turn

The "O" player is part of the Environment. State and reward updates that it gives the Agent consider the "O" play.

@JavaFXpert

Tic-Tac-Toe state is the game board and status

States	0	1	2	3	4	5	6	7	8
O I X I O X X I O, O won	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
I I I I I I O I X, in prog	1.24	1.54	2.13	3.14	2.23	3.32	N/A	1.45	N/A
I I O I I X O I X, in prog	2.34	1.23	N/A	0.12	2.45	N/A	N/A	2.64	N/A
I I O O X X O I X, in prog	+4.0	-6.0	N/A	N/A	N/A	N/A	N/A	-6.0	N/A
X I O I I X O I X, X won	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
...