First Steps: A Structure of Neural Networks

Neural Networks consist of many nodes, edges, and algorithms.

Nodes: Input layer, Hidden Layers, Output Layer

Input Layer: Nodes with input values, each dataset consists of many different input layers.

Hidden Layers: Consist of Values + Algorithm,

Algorithm: activation functions that change values. Edges consist of weights, more layers mean higher power function capability

Each edge is a linear function of y=w*x+b, use gradient descent algorithm to calculate best fitting w and b, newtonian based minimizing SE.

 

More Hidden Layers: More Accuracy

 

# x=[1,2,3,4]

# y=[2,5,10,17]

# a=1

# b=1

# c=1

# run=100000

# rate=0.002

# while run>0:

# run-=1

# la=0.0

# lb=0.0

# lc=0.0

# for i in range(4):

# la+=2*x[i]*x[i]*(a*x[i]*x[i]+b*x[i]+c-y[i])

# lb+=2*x[i]*(a*x[i]*x[i]+b*x[i]+c-y[i])

# lc+=2*(a*x[i]*x[i]+b*x[i]+c-y[i])

# a-=la*rate

# b-=lb*rate

# c-=lc*rate

# print(a,b,c)

Simple take on gradient descent: Calculate a,b,c for (x,y), use derivative of SE to calculate.

Using tensorflow: 

import tensorflow as tf

x_data = [[1.],[2.],[3.],[4.],[6.]]

y_data = [[3.],[5.],[7.],[9.],[13.]]

x=tf.placeholder(tf.float32,[None,1],'in')

y=tf.placeholder(tf.float32,[None,1],'out')

w=tf.Variable(tf.ones([1,3]))

b=tf.Variable(tf.ones([3]))

w2=tf.Variable(tf.ones([3,1]))

b2=tf.Variable(tf.ones([1]))

h=tf.nn.tanh(tf.matmul(x,w)+b)

out=tf.matmul(h,w2)+b2

loss=tf.reduce_mean(tf.squared_difference(out,y))

#opt=tf.train.GradientDescentOptimizer(0.01)

opt=tf.train.AdamOptimizer(0.01)

train_opt=opt.minimize(loss)

sess=tf.Session()

sess.run(tf.global_variables_initializer())

for i in range(10000):

sess.run(train_opt,feed_dict={x:x_data,y:y_data})

print(sess.run(out,feed_dict={x:x_data}))