Confusion Matrix in Tensorflow - tensorflow

I am using finetune AlexNet architecture written by #kratzert on my own dataset which, works properly (I got the code from here: https://github.com/kratzert/finetune_alexnet_with_tensorflow) and I want to figure out how to build confusion matrix from his code. I have tried to use tf.confusion_matrix(labels, predictions, num_classes) to build confusion matrix but I can't. I am confused what should be the values for labels and predictions, I mean, I know what should be but each time I feed these value got an error. Can anyone help me on this or have a look at the code (above link) and guide me?
I added these two lines in finetune.py exactly after calculating accuracy to make the labels and the predictions as the number of the class.
with tf.name_scope("accuracy"):
correct_pred = tf.equal(tf.argmax(score, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
**true_class = tf.argmax(y, 1)
predicted_class = tf.argmax(score, 1)**
and I have added tf.confusion_matrix() inside my session at the very bottom before saving checkpoint of the model
for _ in range(val_batches_per_epoch):
img_batch, label_batch = sess.run(next_batch)
acc, cost = sess.run([accuracy, loss], feed_dict={x: img_batch,
y: label_batch,
keep_prob: 1.})
test_acc += acc
test_count += 1
test_acc /= test_count
print("{} Validation Accuracy = {:.4f} -- Validation Loss = {:.4f}".format(datetime.now(),test_acc, cost))
print("{} Saving checkpoint of model...".format(datetime.now()))
**print(sess.run(tf.confusion_matrix(true_class, predicted_class, num_classes)))**
# save checkpoint of the model
checkpoint_name = os.path.join(checkpoint_path,
'model_epoch'+str(epoch+1)+'.ckpt')
save_path = saver.save(sess, checkpoint_name)
print("{} Model checkpoint saved at {}".format(datetime.now(),
checkpoint_name))
I have tried other places as well but each time I will get an error:
Caused by op 'Placeholder_1', defined at:
File "/home/armin/Desktop/Alexnet_DataPipeline/finetune.py", line 85, in <module>
y = tf.placeholder(tf.float32, [batch_size, num_classes])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_ops.py", line 1777, in placeholder
return gen_array_ops.placeholder(dtype=dtype, shape=shape, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 4521, in placeholder
"Placeholder", dtype=dtype, shape=shape, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3290, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1654, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder_1' with dtype float and shape [128,3]
any help will be appreciated, Thanks.

It's a fairly long piece of code you're referring to, and you did not specify where you put your confusion matrix line.
Just by experience, the most frequent problem with confusion matrices is that tf.confusion_matrix() requires both the labels and the predictions as the number of the class, not as one-hot vectors. In other words, the label and the prediction should be in the form of the number 5 instead of [ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0 ].
In the code you refer to, y is in the one-hot format. The output of the network, score is a vector, giving the probability of each class. That is also not the required format. You need to do something like
true_class = tf.argmax( y, 1 )
predicted_class = tf.argmax( score, 1 )
and use those with the confusion matrix like
tf.confusion_matrix( true_class, predicted_class, num_classes )
(Basically, if you take a look at line 123 of finetune.py, that has both of those elements for determining accuracy, but they are not saved in separate tensors.)

Related

Tensorflow won't matmul inputs and weights. “Dimensions must be equal”

I've been working on a simple tensor flow neural network. My input placeholder is
x = tf.placeholder(tf.float32, shape=[None, 52000, 3]).
My weight matrix is initialized to all zeros as
W = tf.Variable(tf.zeros([52000, 10])).
I tried different combinations with and without the 3 for color channels, but I guess I'm just not understanding the dimensionality because I got the error:
Traceback (most recent call last): File
"C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py",
line 686, in _call_cpp_shape_fn_impl
input_tensors_as_shapes, status) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py",
line 473, in exit
c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape
must be rank 2 but is rank 3 for 'MatMul' (op: 'MatMul') with input
shapes: [?,52000,3], [52000,10].
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "rating.py", line 65, in
y = tf.matmul(x, W) + b # "fake" outputs to train/test File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\math_ops.py",
line 1891, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name) File
"C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\gen_math_ops.py",
line 2436, in _mat_mul
name=name) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\op_def_library.py",
line 787, in _apply_op_helper
op_def=op_def) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py",
line 2958, in create_op
set_shapes_for_outputs(ret) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py",
line 2209, in set_shapes_for_outputs
shapes = shape_func(op) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py",
line 2159, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py",
line 627, in call_cpp_shape_fn
require_shape_fn) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py",
line 691, in _call_cpp_shape_fn_impl
raise ValueError(err.message) ValueError: Shape must be rank 2 but is rank 3 for 'MatMul' (op: 'MatMul') with input shapes: [?,52000,3],
[52000,10].
At first, I thought my next_batch() function was the culprit because I had to make my own due to the fact that I uploaded my images "manually" using scipy.misc.imread(), whose definition reads:
q = 0
def next_batch(batch_size):
x = images[q:q + batch_size]
y = one_hots[q:q + batch_size]
q = (q + batch_size) % len(images)
return x, y
However, after looking through, I don't see what's wrong with this, so I imagine that I'm just confused about dimensionality. It is supposed to be a "flattened" 200x260 color image. It just occurred to me now that maybe I have to flatten the color channels as well? I will place my full code below if curious. I'm a bit new to Tensorflow, so thanks, all. (Yes, it is not a CNN yet, I decided to start simple just to make sure I'm importing my dataset right. And, I know it is tiny, I'm starting my dataset small too.)
############# IMPORT DEPENDENCIES ####################################
import tensorflow as tf
sess = tf.InteractiveSession() #start session
import scipy.misc
import numpy as np
######################################################################
#SET UP DATA #########################################################
images = []
one_hots = []
########### IMAGES ##################################################
#put all the images in a list
for i in range(60):
images.append(scipy.misc.imread('./shoes/%s.jpg' % str(i+1)))
print("One image appended...\n")
#normalize them, "divide" by 255
for image in images:
print("One image normalized...\n")
for i in range(260):
for j in range(200):
for c in range(3):
image[i][j][c]/=255
for image in images:
tf.reshape(image, [52000, 3])
########################################################################
################# ONE-HOT VECTORS ######################################
f = open('rateVectors.txt')
lines = f.readlines()
for i in range(0, 600, 10):
fillerlist = []
for j in range(10):
fillerlist.append(float(lines[i+j][:-1]))
one_hots.append(fillerlist)
print("One one-hot vector added...\n")
########################################################################3
#set placeholders and such for input, output, weights, biases
x = tf.placeholder(tf.float32, shape=[None, 52000, 3])
y_ = tf.placeholder(tf.float32, shape=[None, 10])
W = tf.Variable(tf.zeros([52000, 10])) # These are our weights and biases
b = tf.Variable(tf.zeros([10])) # initialized as zeroes.
#########################################################################
sess.run(tf.global_variables_initializer()) #initialize variables in the session
y = tf.matmul(x, W) + b # "fake" outputs to train/test
##################### DEFINING OUR MODEL ####################################
#our loss function
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y, y_))
#defining our training as gradient descent
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
###################### TRAINING #############################################
#################### OUR CUSTOM BATCH FUNCTION ##############################
q = 0
def next_batch(batch_size):
x = images[q:q + batch_size]
y = one_hots[q:q + batch_size]
q = (q + batch_size) % len(images)
return x, y
#train
for i in range(6):
batch = next_batch(10)
train_step.run(feed_dict={x: batch[0], y_: batch[1]})
print("Batch Number: " + i + "\n")
print("Done training...\n")
################ RESULTS #################################################
#calculating accuracy
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
#print accuracy
print(accuracy.eval(feed_dict={x: images, y_: one_hots}))
Your placeholder should have the dimension [None, 200, 260, 3] where None is the batch size, 200, 260 is the image size, and 3 is the channels.
Your weights should be [filter_height, filter_width, num_channels, num_filters]
Your bias should be [num_filters]
And the dimensions for the labels should be [None, num_classes] where None is the batch size, and num_classes is the number of classes that your images have.
These are just to make sure that math works.
I took these codes from here

Rank error in tf.nn.dynamic_rnn

I am trying to build a CNN + RNN model and I am getting the following error.
Any help will be appreciated.
fc2 has shape (?,4096)
cell = tf.contrib.rnn.BasicLSTMCell(self.rnn_hidden_units)
stack = tf.contrib.rnn.MultiRNNCell([cell]*self.rnn_layers)
initial_state = cell.zero_state(self.batch_size, tf.float32)
initial_state = tf.identity(initial_state, name='initial_state')
outputs, _ = tf.nn.dynamic_rnn(stack, fc2,dtype=tf.float32)
File "rcnn.py", line 182, in model
outputs, _ = tf.nn.dynamic_rnn(stack, [fc2],dtype=tf.float32)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.py", line 574, in dynamic_rnn
dtype=dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.py", line 637, in _dynamic_rnn_loop
for input_ in flat_input)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.py", line 637, in
for input_ in flat_input)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 649, in with_rank_at_least
raise ValueError("Shape %s must have rank at least %d" % (self, rank))
ValueError: Shape (4096, ?) must have rank at least 3
Copying the answer of #jdehesa from his comment for better visibility:
The error seems fairly clear, tf.nn.dynamic_rnn expects a 3-dimensional tensor as input (i.e. rank 3), but fc2 has only two dimensions. The shape of fc2 should be something like (<batch_size>, <max_time>, <num_features>) (or (<max_time>, <batch_size>, <num_features>) if you pass time_major=True)
The default input of tf.nn.dynamic_rnn has a dimension of 3 (Batchsize, sequence_length, num_features). Since your num_features is 1 you can expand your fc_seq with
fc2 = tf.expand_dims(fc2, axis = 2)
and then use the code you have above.

How can I do adjustable gradient clipping with tensorflow

I want use this gradient clipping technique in tensorflow. So, I have next function:
def create_optimizers(cost, collections, learning_rate):
'''
Create optimizer for collections with associate learning rate
Args:
cost: Euclid cost functions
collections: training collections
learning_rate: learning rate for gradient descent
Return:
Gradient functions associate with tenosrflow collections
'''
optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate, momentum=0.9)
gradients = {}
sigma = 0.01
clip_val = tf.div(sigma, learning_rate)
#clip_val = 0.1
# clip gradient
for collection in collections:
tvars = tf.get_collection(collection)
grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), clip_norm=clip_val)
train_op_wc = optimizer.apply_gradients(zip(grads, tvars))
return gradients
Now, it done with constant clipping norm. But when I fill it by placeholder like:
weight_learning_rate = tf.placeholder(tf.float32, shape=[])
weight_optimizers = sp_net_util.create_optimizers(cost, weights_coll, weight_learning_rate)
And run it as
sess.run(weight_optimizers[i],feed_dict={weight_learning_rate: private_weight_learning_rate,...})
I got this error:
File ".../sp_train.py", line 70, in train
weight_optimizers = sp_net_util.create_optimizers(cost, weights_coll, weight_learning_rate)
File "".../sp_net_util.py", line 218, in create_optimizers
grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), clip_norm=clip_val)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/clip_ops.py", line 208, in clip_by_global_norm
constant_op.constant(1.0 / clip_norm, dtype=use_norm.dtype))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 163, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 353, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 290, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).__name__))
TypeError: Expected float32, got <tf.Tensor 'weight_optimizer/clip_by_global_norm/truediv_1:0' shape=() dtype=float32> of type 'Tensor' instead.
If I put constant everything OK.
So, any ways to do this with placeholder in tensorflow?

ctc loss error - sequence_length(0) <= 3

i am facing an error similar to the one stated here CTC Loss InvalidArgumentError: sequence_length(b) <= time but there seems to be no explanation of what the error actually means. Based on the reading i did, does it mean that the sequence length
of the example "0" in the minibatch is less than 3 ? in which case why is it an error (since as explained in the tf doc and the question above, the length of all sequences has to be lesser than time, right ?) ..could anyone kindly explain how i could debug the issue and make sense of the error? i am using an existing conv2d example and trying to merge ctc loss using some audio files i had
the code is present here https://github.com/takingstock/ServerSide-Algos/blob/master/ctc-conv.py and the problem occurs on line 213 (apologies for pasting the code github url instead of the code here..i felt it might be cleaner this way)
the stack trace
Caused by op u'CTCLoss', defined at:
File "conv_train.py", line 279, in <module>
loss = tf.nn.ctc_loss(Y , logits, seq_len)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/ctc_ops.py", line 156, in ctc_loss
ignore_longer_outputs_than_inputs=ignore_longer_outputs_than_inputs)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_ctc_ops.py", line 224, in _ctc_loss
name=name)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): sequence_length(0) <= 3
[[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=false, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/device:CPU:0"](transpose, _arg_Placeholder_3_0_3, _arg_Placeholder_2_0_2, _arg_Placeholder_4_0_4)]]
turns out the error was because of the way i was feeding input into the ctc_loss function. The logits should have been in the shape [ max_timestep, batch_size, num_classes/labels ] but i was sending it the other way round. Kindly look at the updated code in the url below ..hopefully it might be of some use to some folks.
https://github.com/takingstock/ServerSide-Algos/blob/master/ctc_conv_corrected.py
to be precise , this is the part of the code that was creating issues
conv1 = conv2d(x, weights['wc1'], biases['bc1'])
# Max Pooling (down-sampling)
conv1 = maxpool2d(conv1, k=2)
# Convolution Layer
conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
# Max Pooling (down-sampling)
conv2 = maxpool2d(conv2, k=2)
# Fully connected layer
# Reshape conv2 output to fit fully connected layer input
fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])
fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
fc1 = tf.nn.relu(fc1)
# Apply Dropout
fc1 = tf.nn.dropout(fc1, dropout)
if you notice, the addition of pooling reduces the dimensionality of data that needs to be input into ctc_loss. Also, in my personal experience (and quite a bit of literature i have read) pooling doesn't do much good( at least not in non image convolutions) hence i replaced the above with
x = tf.reshape(X, shape=[-1, num_features, 399 , 1])
# Convolution Layer
conv1 = conv2d(conv1, weights['wc1'], biases['bc1'], 1)
fc1 = tf.reshape(conv1, [batch_size , 399 ,
weights['wd1'].get_shape().as_list()[0]])
fc1 = tf.layers.dense( fc1, 1024 , activation=tf.nn.relu)
# Apply Dropout
fc1 = tf.nn.dropout(fc1, keep_prob)
# Output, class prediction
logits = tf.layers.dense(inputs=fc1, units=num_classes, activation=tf.nn.relu)
logits = tf.transpose(logits, (1, 0, 2))
loss = tf.nn.ctc_loss(Y , logits, seq_len)
this way, the input going into ctc_loss has the exact required [ max_ts, batch, label] format. Also the results of using just 1 layer of conv is way superior to BiRNN (**for my data)
..also this post proved to be of immense intuitive help (for using convolutions with ctc_loss)
How to use tf.nn.ctc_loss in cnn+ctc network

Custom loss function: perform a model.predict on the data in y_pred

I am training a network to denoise images,for this I am using the CIFAR10 dataset. I am trying to generate a custom loss function so that the loss is mse / classification_accuracy.
Given that my network receives as input 32x32 (noisy) images and predicts 32x32 (denoised) images, I am assuming that y_pred and Y_true would be arrays of 32x32 images. Thus my custom loss functions looks like this:
def custom_loss():
def joint_optimized_loss(y_true, y_pred):
mse = K.mean(K.square(y_pred - y_true), axis=-1)
preds = classif_model.predict(y_pred)
correctPreds = 0
totPreds = 0
for pred in preds:
predictedClass = pred.index(max(pred))
totPreds += 1
if predictedClass == currentClass:
correctPreds += 1
classifAccuracy = correctPreds / totPreds
loss = mse / classifAccuracy
return loss
return joint_optimized_loss
myModel.compile(optimizer='adadelta', loss=custom_loss())
classif_model is a pre-trained model that classifies CIFAR10 images into one of the 10 classes. It receives an array of 32x32 images.
However when I run my code I get the following error:
Traceback (most recent call last):
File "myCode.py", line 94, in
myModel.compile(optimizer='adadelta', loss=custom_loss())
File "/home/rvidalma/anaconda2/envs/tensorUpdated/lib/python2.7/site-packages/keras/engine/training.py",
line 850, in compile
sample_weight, mask)
File "/home/rvidalma/anaconda2/envs/tensorUpdated/lib/python2.7/site-packages/keras/engine/training.py",
line 450, in weighted
score_array = fn(y_true, y_pred)
File "myCode.py", line 57, in joint_optimized_loss
preds = classif_model.predict(y_pred)
File "/home/rvidalma/anaconda2/envs/tensorUpdated/lib/python2.7/site-packages/keras/models.py",
line 913, in predict
return self.model.predict(x, batch_size=batch_size, verbose=verbose)
File "/home/rvidalma/anaconda2/envs/tensorUpdated/lib/python2.7/site-packages/keras/engine/training.py",
line 1713, in predict
verbose=verbose, steps=steps)
File "/home/rvidalma/anaconda2/envs/tensorUpdated/lib/python2.7/site-packages/keras/engine/training.py",
line 1260, in _predict_loop
batches = _make_batches(num_samples, batch_size)
File "/home/rvidalma/anaconda2/envs/tensorUpdated/lib/python2.7/site-packages/keras/engine/training.py",
line 374, in _make_batches
num_batches = int(np.ceil(size / float(batch_size)))
AttributeError: 'Dimension' object has no attribute 'ceil'
I think this has something to do with the fact that y_true and y_pred are both tensors that, before training, are empty thus classif_model.predict fails as it is expecting an array. However I am not sure on how to fix this...
I tried getting instead the value of y_pred using K.get_value(y_pred), but that gives me the following error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape
[-1,32,32,3] has negative dimensions [[Node: input_1 =
Placeholderdtype=DT_FLOAT, shape=[?,32,32,3],
_device="/job:localhost/replica:0/task:0/cpu:0"]]
You cannot use accuracy as a loss function, as it is not differentiable. This is why upper bounds on accuracy like the cross-entropy are used instead.
Additionally, the way you implemented accuracy is also non-symbolic, you should have used only functions in keras.backend to implement a loss for it to work properly.

Resources