Detecting objects starting from street-scene images

As a remark, neural networks consider equally-sized inputs, which is not the case of Mapillary images. A first approximation could be to resize every image as the most encountered size (2448*3264), however we choose to resize them at a smaller size (576*768) for computation purpose.

Handling image within neural network is generally done with the help of convolutional neural networks. They are composed of several kinds of layers that must be described:

• convolutional layers, in which images are filtered by several learnable image kernels, so as to extract image patterns based on pixels (this layer type is of first importance in convolutional neural network);
• pooling layers, in order to reduce the size of images and converge towards the output layer, as well as to extract feature rough locations (the max pooling operation is the most common one, i.e. consider the maximal value over a local set of pixels);
• fully-connected layers, where every neuron of the current layer are connected to every neuron of the previous layer.

We’ve carried out a set of tests with different hyperparameter values, i.e. different amounts of each layer kinds. The results are globally stable if we consider more than one convolutional layer. Here comes the way to define a neural network with TensorFlow, the dedicated Python library.

Inputs and outputs are defined as « placeholders », aka a sort of variables that must be fed by real data.

X = tf.placeholder(tf.float32, [None, 576, 768, 3], name='X')
Y = tf.placeholder(tf.float32, [None, 66], name='Y')

After designing the kernel and the biases, we can use the TensorFlow function conv2d to build this layer.

kernel1 = tf.get_variable('kernel1', [8, 8, 3, 16], initializer=tf.truncated_normal_initializer())
biases1 = tf.get_variable('biases1', [16], initializer=tf.constant_initializer(0.0))
# Apply the image convolution with a ReLu activation function
conv_layer1 = tf.nn.relu(tf.add(tf.nn.conv2d(X, kernel1, strides=[1, 1, 1, 1], padding="SAME"),
                                biases1))

As for convolutional layer, there is a ready-to-use function in the TensorFlow API, i.e. max_pool.

pool_layer1 = tf.nn.max_pool(conv_layer1, ksize=[1, 4, 4, 1],
                             strides=[1, 4, 4, 1], padding='SAME')

This operation corresponds to a standard matrix multiplication; we just have to reshape the output of the previous layer so as to consider comparable structures. Let’s imagine we have added a second convolutional layer as well as second max-pooling layer, the full-connected layer definition is as follows:

reshaped = tf.reshape(pool_layer2, [-1, int((576/(4*4))*(768/(4*4))*24)])
# Create weights and biases
weights_fc = tf.get_variable('weights_fullconn', [int((576/(4*4))*(768/(4*4))*24), 1024],
                    initializer=tf.truncated_normal_initializer())
biases_fc = tf.get_variable('biases_fullconn', [1024],
                    initializer=tf.constant_initializer(0.0))
# Apply relu on matmul of reshaped and w + b
fc = tf.nn.relu(tf.add(tf.matmul(reshaped, weights_fc), biases_fc), name='relu')
# Apply dropout
fc_layer = tf.nn.dropout(fc, 0.75, name='relu_with_dropout')

Here we have defined the major part of our network. However the output layer is still missing…

The predicted labels are given after a sigmoid activation in the last layer: the sigmoid function allows to consider independant probabilities in multilabel context, i.e. if the presence of different object types on images is possible.

The sigmoid function gives probabilities of appearance for each object type, in a given picture. The predicted labels are built as simply as possible: a threshold of 0.5 is set to differentiate negative and positive predictions.

# Create weights and biases for the final fully-connected layer
weights_sig = tf.get_variable('weights_s', [1024, 66],
                              initializer=tf.truncated_normal_initializer())
biases_sig = tf.get_variable('biases_s', [66], initializer=tf.random_normal_initializer())
logits = tf.add(tf.matmul(fc_layer, weights_sig), biases_sig)
Y_raw_predict = tf.nn.sigmoid(logits)
Y_predict = tf.to_int32(tf.round(Y_raw_predict))

Although several metrics may measure the model convergence, we choose to consider classic cross-entropy between true and predicted labels.

entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=Y, logits=logits)
loss = tf.reduce_mean(entropy, name="loss")
optimizer = tf.train.AdamOptimizer(0.01).minimize(loss)

In this snippet, we are using AdamOptimizer, however other solutions do exist (e.g. GradientDescentOptimizer).

Several way of measuring the model quality may be computed, see e.g.:

• accuracy (number of good predictions, over total number of predictions)
• precision (number of true positives over all positive predictions)
• recall (number of true positives over all real positive values)

They can be computed globally, or by label, as we are in a multilabel classification problem.