License Plate Recognition using OpenCV, YOLO and Keras
Our goal was to recognize license plates in real time. We focused on the Belgian cars.
In that purpose, we used the following python libraries :
There are 3 steps in our process: First we need to detect the plate then perform character segmentation and finally read the plate.
Hardware : Google cloud compute engine (8 vCPU, 30 Go memory, Tesla K80, Ubuntu 18.04)
1. Plate detection
Using Darkflow, we trained a YOLO (You Only Look Once) model, with 1900 images of car with annotated plate. LabelImg is a great tool witch allowed us to annotate our images in Pascal VOC format. This dataset was composed of car images we found online, some we took on the street and data augmentation (Vertical Flip, Brightness modification) using Keras (ImageDataGenerator).
Training code :
import numpy as np
from darkflow.net.build import TFNet
import cv2
options = {"model": "cfg/yolo-1c.cfg",
"load": "bin/yolo.weights",
"batch": 8,
"epoch": 100,
"gpu": 0.9,
"train": True,
"annotation": "./data/AnnotationsXML/007/",
"dataset": "./data/Images/007/"}
tfnet = TFNet(options)
tfnet.train()
tfnet.savepb()
Then we load weights from our training to make new predictions :
options = {"pbLoad": "yolo-plate.pb", "metaLoad": "yolo-plate.meta", "gpu": 0.9}
yoloPlate = TFNet(options)firstCrop function use best prediction from our YOLO model and return the license plate.
def firstCrop(img, predictions):
predictions.sort(key=lambda x: x.get('confidence'))
xtop = predictions[i].get('topleft').get('x')
ytop = predictions[i].get('topleft').get('y')
xbottom = predictions[i].get('bottomright').get('x')
ybottom = predictions[i].get('bottomright').get('y') firstCrop = img[ytop:ybottom, xtop:xbottom]
cv2.rectangle(img,(xtop,ytop),(xbottom,ybottom),(0,255,0),3) return firstCrop
secondCrop function use OpenCV functions to crop a little more the plate to avoid noise in the background.
def secondCrop(img):
gray=cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
ret,thresh = cv2.threshold(gray,127,255,0)
contours,_ = cv2.findContours(thresh,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)
areas = [cv2.contourArea(c) for c in contours]
if(len(areas)!=0):
max_index = np.argmax(areas)
cnt=contours[max_index]
x,y,w,h = cv2.boundingRect(cnt)
bounds = cv2.boundingRect(cnt)
cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),2)
secondCrop = img[y:y+h,x:x+w]
else:
secondCrop = img
return secondCropMain code using functions explained above :
predictions = yoloPlate.return_predict(frame)
firstCropImg = firstCrop(frame, predictions)
secondCropImg = secondCrop(firstCropImg)
2. Character Segmentation
We used two methods for more accuracy :
In the first one, we used an another YOLO model trained with images of license plates where characters have been annotated. There is only one label “character”. Around 1400 characters.
Weights loading :
options = {"pbLoad": "yolo-character.pb", "metaLoad": "yolo-character.meta", "gpu":0.9}
yoloCharacter = TFNet(options)In the second method, we used OpenCV’s functions to process the plate.
def auto_canny(image, sigma=0.33):
# compute the median of the single channel pixel intensities
v = np.median(image)
# apply automatic Canny edge detection using the computed median
lower = int(max(0, (1.0 - sigma) * v))
upper = int(min(255, (1.0 + sigma) * v))
edged = cv2.Canny(image, lower, upper)
# return the edged image
return edgedgray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
thresh_inv = cv2.adaptiveThreshold(gray,255,cv2.ADAPTIVE_THRESH_MEAN_C,cv2.THRESH_BINARY_INV,39,1)
edges = auto_canny(thresh_inv)ctrs, _ = cv2.findContours(edges.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
sorted_ctrs = sorted(ctrs, key=lambda ctr: cv2.boundingRect(ctr)[0])
img_area = img.shape[0]*img.shape[1]for i, ctr in enumerate(sorted_ctrs):
x, y, w, h = cv2.boundingRect(ctr)
roi_area = w*h
roi_ratio = roi_area/img_areaif((roi_ratio >= 0.015) and (roi_ratio < 0.09)):
if ((h>1.2*w) and (3*w>=h)):
cv2.rectangle(img,(x,y),( x + w, y + h ),(90,0,255),2)
3. Character Recognition
We train a CNN with Tensorflow and Keras libraries. There are 35 classes ( 10 for numbers and 25 for alphabet without “O”). We used approximately 1000 images for each classes. We collected a sample of characters images and then practiced data augmentation (rotation and brightness).
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(height, width, channel)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(35, activation='softmax'))
model.summary()
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(train_images, train_labels, epochs=8)
test_loss, test_acc = model.evaluate(test_images, test_labels)
print(test_acc)
model.save("model_char_recognition.h5")
Predictions code:
def cnnCharRecognition(img):
dictionary = {0:'0', 1:'1', 2 :'2', 3:'3', 4:'4', 5:'5', 6:'6', 7:'7', 8:'8', 9:'9', 10:'A',
11:'B', 12:'C', 13:'D', 14:'E', 15:'F', 16:'G', 17:'H', 18:'I', 19:'J', 20:'K',
21:'L', 22:'M', 23:'N', 24:'P', 25:'Q', 26:'R', 27:'S', 28:'T', 29:'U',
30:'V', 31:'W', 32:'X', 33:'Y', 34:'Z'} blackAndWhiteChar= cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
blackAndWhiteChar = cv2.resize(blackAndWhiteChar,(75,100))
image = blackAndWhiteChar.reshape((1, 100,75, 1))
image = image / 255.0
new_predictions = characterRecognition.predict(image)
char = np.argmax(new_predictions)
return dictionary[char]
4. Final Results
5. Axis of improvement
It’s hard to make a robust solution which works even if there are a bad weather or a lot of light. The weakness of our process is the character segmentation. We could have improve our YOLO model in charge of that but it would need a lot more data and it’s very painful to annotated these images.
You can also improve OpenCV character segmentation by adding more image processing like dilate or close function however it’s depend of the quality and size of the license plate image returned by the prediction. This may work well on some cases but not in others, we think a deep learning solution is more robust.
We also tried to train an OCR model with YOLO but we do not have enough quality data to make it work properly. We used the same images than in our second YOLO model by annotating each character by its corresponding letter or number. The dataset was unbalanced because in Belgian license plates there usually are 4 numbers and 3 characters. So we had much more number than letters. The model was able to recognize numbers only.
You can find the whole code in our GitHub
If you have any questions, or ideas of improvement, do not hesitate to give us a feedback in comments ! :)
Jordan Van Eetveldt & Théophile Buyssens (Junior Data Scientists at Intys Data)
