Generic and attribute-specific deep representations for maritime vessels

MARVEL dataset consists of 2 million marine vessel images collected from Shipspotting website [12]. For most of the images in the dataset, the following attributes are available: beam, year built, draught, flag, gross tonnage, IMO number, name, length, category, summer deadweight, MMSI, vessel type.

Among the above attributes, we observe that the most useful and visually relevant categories are as follows: (1) Vessel type, (2) category, (3) draught, (4) gross tonnage, (5) length, (6) summer deadweight, and (7) IMO number. Vessel type is assigned based on the type of cargo a vessel will be transporting. For instance, if a vessel carries passengers, its type is very likely to be a Passengers Ship. The dataset contains 1,607,190 images with valid annotated type labels belonging to one of 197 categories. Vessel type histogram, highlighting the major categories, is depicted in Fig. 1 c. Another available attribute is category, which is another vessel description. Example categories with a substantial number of members are chemical and products tankers, containerships built 2001–2010, and Tugs (please see Fig. 1 a). All collected images have been assigned a category out of 185 categories in MARVEL dataset. IMO number is another category, which is an abbreviation for International Maritime Organization number. Similar to the chassis numbers of cars, IMO numbers of vessels uniquely identify the ships registered to IMO regardless of any changes made in their names, flags, or ownerships. Of the collected images, 1,628,056 are annotated with IMO numbers (please refer to Fig. 1 b). There are a total of 103,701 unique IMO numbers in MARVEL dataset.

Considering the fact that images which have been assigned identical IMO numbers belong to the same vessels, we are able to check the consistency of other attribute annotations and fill out the missing entries when necessary. First, zero or invalid entries are discarded. Next, we convert all attribute labels to metric unit system to account for the presence of some labels in an imperial system. Finally, we maintain the consistency of labels for each vessel separately by applying median filters on available annotations. Engaging such preprocessing procedures, we obtain very large groups of images that include valid attribute labels. The attributes we focus on are IMO number, vessel type label, draught, gross tonnage, length, and summer deadweight (Fig. 2). For draught, an attribute which is defined as the vertical distance between the bottom of vessel hull and waterline, there are 1,067,946 images carrying validated labels. Gross tonnage is a unit-less index calculated using the internal volume of vessels. There are 1,583,882 images with valid annotated labels for gross tonnage. Validated annotations for summer deadweight, a measure of carrying capacity of a ship, are provided for 1,508,974 of all images. Length data of the maritime vessels are made available for 1,107,907 images. In summary, when combined, a total of 1,006,868 images retain valid annotated labels for all vessel type, IMO number, draught, length, summer deadweight, and gross tonnage attributes.

Huge quantity of images and their annotations, existing in MARVEL, makes it applicable to directly employ recent methods utilizing deep architectures such as AlexNet [2] for vessel categorization. One may choose one of the provided vessel attributes such as vessel type or category and apply classification methods for categorizing images according to the selected attribute.

In MARVEL there are more than 8000 unique vessels (carrying unique IMO numbers) having more than 50 example images as shown in Fig. 1 b. It is also feasible to use the dataset for both vessel verification and identity recognition, which could be a vital part of a maritime security system, analogous to a scenario where vehicle make and model recognition is crucial for a traffic security system.

The main foci of this study on MARVEL dataset are five folds: (1) vessel classification since content of cargo that a ship carries, specified by its type, is crucial for maritime surveillance, (2) identity verification where the ultimate goal is to find out if a pair of images belong to the same vessel with a unique IMO number, (3) retrieval where one might desire to query a vessel image and retrieve a certain number of similar images from a database, (4) identity recognition which is a challenging though interesting task which aims at recognizing a specific vessel within vessels of same type or among all other vessels (This might be likened to a facial recognition task.), and finally (5) specific attribute prediction and classification, where the objective is to grasp draught, length, gross tonnage, and summer deadweight of a vessel by simply analyzing the 2-D visual content. With an aim to achieve these goals, we design generic and attribute specific representations which are powerful in describing marine vessel images.

For vessel classification, one of the most important tasks, we first generate a set of superclasses which may contain vessels of more than one type, since some subsets of vessel types are not visually distinguishable even with human supervision. The sole differences within the subsets arises from the invisible content of cargo rather than the visual appearance of ships. A concrete example of such a case arises for the pair of vessel types: crude oil tanker and oil products tanker, which is illustrated in Fig. 3. Although the two vessel types have distinct functional differences, their visual characteristics are congruent especially when images are captured by cameras located far away from these vessels; when the vessels occupy a small portion of images and their decks are not visible from such a view point, it is tough to distinguish them. Hence, we merge some of the types to generate superclasses which are semantically correct and visually discriminable. In Section 4, we describe the details for combining vessel types. As inspired by [15], the presence of multi-level relevance information and hierarchical grouping of vessels may allow exploitation of MARVEL dataset for a further performance improvement for particular marine vessel recognition tasks in the future.

Vessel verification task serves for deciding whether a pair of vessel images belong to the same vessel or not. This may be beneficial for a naval surveillance scenario, where a specific vessel is required to be tracked using an electro-optic imaging system.

For the task of vessel retrieval relating to vessel classification, the goal is to retrieve images belonging to providing a query image, several images with similar content are retrieved from the database.

Vessel recognition aims at revealing the accurate identity of a vessel by analyzing an unseen example image of it and finding out the matching vessel within a group of vessels. This task may be particularly useful for scenarios of marine surveillance and port registration. For this task, first, we performed recognition for vessels considering their type labels, for instance, identifying a passenger ship among other passenger ships. Next, we attempt a more challenging recognition problem, identifying vessels where no additional cues such as vessel type labels or category labels are given.

Moreover, as novel problems, we attempt tasks of predicting and classifying vessel attributes: draught, gross tonnage, length, and summer deadweight. The objective here is to quantify these attributes based on 2-D visual content only, which may ameliorate the practicality of coastal surveillance systems, since that avoids the need for retaining meta-data for optical systems, namely camera parameters, camera position, and distance to the vessel, while estimating physical dimensions of a vessel based on its appearance. Another beneficial use of this task may be for safe marine traffic routing as well as for the calculation of port access and transit fees, when vessel dimensions need to be known. Furthermore, there are studies, proving that presence of attribute-based representations are helpful for several computer vision tasks including object recognition [16], detection [17], and identification [18]. The attribute-based learned representations for marine vessels in this work may be utilized in a similar fashion aiding other visual analysis tasks.

To generate superclasses from vessel types, the first 50 major vessel types containing the largest amount of example images are selected and sorted according to their quantity. The vessel type with the largest number of images which is employed in our superclass generation, is general cargo, consisting of 324,561 example images. The class with the smallest number of images is the timber carrier, accommodating only 1837 images. In this work, to investigate the visual similarities among vessel types, MatConvNet Toolbox [19] implementation of a pre-trained convolutional neural network (CNN) architecture, VGG-F [20], is adopted. Features are extracted posterior to resizing images to 224×224. Utilizing the penultimate layer acctivations of VGG-F [20] as visual representations of images, each image is described by a 4096-dimensional feature vector. Based on these feature vectors, we calculated a dissimilarity matrix for the 50 major vessel classes. To generate superclasses, 1/10 of all collected images belonging to 50 major classes are randomly selected (approximately 130,000 images) and individual class statistics are estimated. Prior to calculating a dissimilarity matrix, we removed outliers following the preprocessing step explained below.

4.2 Dissimilarity matrix and superclass generation

Once outliers are removed from each vessel type class by the above procedure, the remaining examples are used to compute a dissimilarity matrix. We compute symmetrized divergence as the dissimilarity index. Symmetrized divergence D _S (P,Q) of two classes, namely P and Q, is defined as \(D_{S}(P,Q) = \frac {1}{2} D_{KL}(P\lvert \rvert Q)+\frac {1}{2} D_{KL}(Q\lvert \rvert P)\), where D _KL (.||.) stands for Kullback-Liebler divergence of two multivariate Gaussian distributions. The computed dissimilarity matrix is depicted in Fig. 4.

By exploiting the dissimilarity matrix, we merge similar vessel type classes using a threshold. Prior to thresholding, we applied spectral clustering methods with the help of the dissimilarity matrix. Nevertheless, the resulting groups were not semantically meaningful. Hence, we opt to continue by increasing the threshold for the similarities of the pairs of classes (i.e., this corresponds to each entry of the dissimilarity matrix). If dissimilarity index of a pair of classes is below a threshold, the pair is assigned to the same superclass. We keep increasing the threshold before it reaches to a point where semantically irrelevant classes (human supervision is adopted here) start to merge, and we define it as the final threshold for clustering. The majority of the resulting superclasses contain reasonable classes. The generated vessel type superclasses with more than one vessel type are (1) tankers (consisting of oil products tanker, oil/chemical tanker, tanker, chemical tanker, crude oil tanker, lpg tanker, lng tanker, ore carrier), (2) carrier/floating (consisting of timber carrier, floating storage production, self discharging bulk carrier), (3) supply vessels (which contain offshore supply ship, supply vessel, tug/supply vessel, anchor handling vessel, multi purpose offshore vessel), (4) fishing vessels (which include trawler, fishing vessel, factory trawler, fish carrier), and (5) dredgers (which contain suction dredger, hopper dredger). Finally, marginal adjustments are done manually to make all superclasses as meaningful as possible. These adjustments include merging superclass containing only trailing suction hopper dredger with superclass consisting of Suction Dredger and Hopper Dredger. In addition, seven vessel types are removed entirely from the set of superclasses. The classes to be eliminated are decided according to the average dissimilarity of the classes to the rest. The salient overall dissimilarity scores are detected manually. The removed classes are, namely (1) general cargo (it is significantly confusing with the container ship and ro-ro cargo), (2) cargo/containership, (3) research/survey vessel, (4) cement carrier, (5) multi purpose offshore vessel, (6) passenger/cargo ship, and (7) cable layer. The removed classes both visually and functionally contain more than at least two separate classes, i.e., passenger/cargo ship involve both passenger vessels and general cargo vessels. The merged classes with thresholding also contain visually very meaningful vessel types, i.e., all of the fish-related vessels are clustered within the same superclass. The distribution of final 26 superclasses can be viewed in Fig. 5.

4.3 Superclass classification

As demonstrated in Fig. 5, there exists an imbalance between superclasses. Nevertheless, even the superclass with the least amount of examples has a large quantity of examples. Therefore, to classify superclasses of vessels, it is feasible to train a deep CNN architecture AlexNet [2]. To avoid the imbalance between superclasses, we acquire equal numbers of samples from each class for both training and testing, as 8192 and 1024 images, respectively. For superclasses with examples less than the required amount, we generate additional examples by data augmentation (using different croppings of images). Consequently, our training and test sets contain 212,992 and 26,624 examples, respectively, although we have 140,000 unique examples. We should also note that no images of the same vessels appear in both training and test sets. The classification performance is quantified by the help of a normalized confusion matrix [7]. The practical + metric for a fine-grained classification task can be the class-normalized average classification accuracy, which is calculated as the average of diagonal elements of a normalized confusion matrix, C, entries of which are defined as follows [6]:

$$ C_{pq} = \frac{\lvert \{i:\hat{y_{i}}=q \wedge y_{i}=p\}\rvert}{\lvert\{i:y_{i}=p\}\rvert}, $$

(1)

where |.| denotes the cardinality of the set, \(\hat {y_{i}}\) indicates the estimated class label, and y _i is the actual label for the i ^th training example. The final performance measure is the mean of the diagonal elements of the matrix C. This value for 26 superclasses is 73.14% for the normalized confusion matrix depicted in Fig. 6. To emphasize the validity and efficacy of the learned network, we also compare it with another method utilizing multi-class support vector machine (SVM) with the Crammer and Singer multi-class SVM [21] implementation of [22] in LIBLINEAR [23] library. The feature vectors for training SVM are extracted from the VGG-F network of [20], their dimensionality is reduced to 256, and PCA whitening is applied. Due to memory requirements and computational complexity in optimization, we use half of the training set. We report the class-normalized average classification accuracy in testing as 53.89%. Compared to the use of pre-learned VGG-F weights with an SVM classifier, AlexNet trained from scratch has 35% improvement in accuracy.

In this section, we make use of our dataset, MARVEL, for potential maritime applications and vessel verification, retrieval, identity recognition, and attribute prediction and classification. In the following subsections, these applications and necessary experimental settings are explained.

During all experiments, we follow training and testing strategies similar to [10]. First, 8000 vessels with unique IMO numbers are selected such that each vessel will have 50 example images, resulting in a total of 400,000 images. This data is divided into two splits: training and testing. The training set consists of 4035 vessels (201,750 example images in total), and the test set contains 3965 vessels (198,250 example images in total). There exist 109 vessel type labels among 400,000 examples, and training and test sets are split in a way that the number of vessel types are identical in both sets. In the rest of the paper, we call the training split of this subset as IMO training set, and the test split as IMO test set.

We propose three deep CNN-based generic representations for marine vessels on IMO training set by making use of vessel type and/or vessel IMO labels. Hence, we train the same architecture of [2] as in vessel classification task and modify it accordingly with an aim to capture more details in vessel images: For the last layer, rather than 26 label classes, we use 109, 4035, and 4144 label classes. These three different classifiers focus on discriminating vessel types, vessel IMO numbers (classifying individual vessels on IMO training set), and both vessel types and IMO numbers (jointly classifying type and IMO numbers of vessels on IMO training set), respectively. We compare the performances of these three representations over computer vision tasks, which are described below in details.

Deep representations for example images are extracted as the penultimate layer activations of the trained networks (as in the superclass generation part in Section 4) with 4096 dimensions. More discriminative features being desired, we extract the penultimate layer activations prior to the rectified linear unit (ReLU) layer, which carry more information than the layer after ReLU since the negative values are cast to zero after ReLU. This choice makes our vessel verification performance better than the case with the deep representations after ReLU case.

During all experiments utilizing convolutional neural networks, we select batch sizes as 256 without normalization and decaying learning rates, consisting of logarithmically equally spaced values between 0.01 and 0.0001. For superclass classification, we train the networks for 60 epochs and for attribute classification and prediction, we train the networks for 50 epochs, since we notice that the training error does not decrease with further training. The implementation of the networks are based on the MatConvNet Toolbox [19].

5.1 Vessel verification, retrieval, and recognition

5.1.1 Vessel verification

Akin to face verification [24], car model verification is applied in CompCars dataset [10] to serve for conceivable purposes in transportation systems. That kind of task is claimed to be more complicated compared to face verification, since car model verification is performed on images with unconstrained viewpoints. On MARVEL dataset, we perform maritime vessel verification where the attribute to be verified is the vessel identity. Please note that our task is more challenging compared to identifying other attributes such as category or vessel type. Furthermore, this problem is more challenging than both car model and face verification tasks, since it is desired to identify/verify pairs of individual vessels by looking only at their appearances which have more diversity.

After extracting the generic deep representations (109 and 4144-dimensional output based), 50,000 positive pairs (belonging to same vessels) and 50,000 negative pairs (belonging to different vessels) are picked randomly from both training and test splits out of 201,750 training examples and 198,250 test examples, respectively¹. For all 400,000 training and testing examples, feature vector dimensionality is reduced to 100 by PCA exploited with only training examples. Moreover, all 100-dimensional examples are PCA whitened since whitening increases performance of SVM classifier. Concatenating two 100-dimensional vectors, we describe each pair of vessel during verification experiments. Finally, for each generic representation, we train a binary SVM classifier with a radial basis function kernel on the generated training set by using the implementation of LIBSVM library [25]. Additionally, we attempt end-to-end learning for verification. For this experiment, we construct a Siamese neural network, based on AlexNet architecture, with shared weights, and added a contrastive loss layer after the last fully connected layers. Contrastive loss [26], incurring for similar and dissimilar pairs of images is defined as,

$$ L = (1-Y)\frac{1}{2}(D_{W})^{2}+Y\frac{1}{2}\left\{max(0,m-D_{W}) \right\}^{2} $$

(2)

where Y is a binary label, assigned to 1 for similar images, otherwise set to 0. m>0 is a margin set for dissimilar pairs, and D _W is the distance to be learned for pairs of images, \(\vec {X_{1}}\) and \(\vec {X_{2}}\). D _W is calculated as the Euclidean distance between outputs of parametrized function G _W .

$$ D_{W}(\vec{X_{1}},\vec{X_{2}}) = \left\| G_{W}(\vec{X_{1}})-G_{W}(\vec{X_{2}}) \right\|_{2} $$

(3)

The precision recall curves for the two generic representations and the Siamese network-based representation, obtained by varying the classification thresholds, are plotted in Fig. 7. We also compare the performance of SVMs with the nearest neighbor (NN) classifiers. For NN classifier, each test pair is assigned the label of the training pair for which the Euclidean descriptor distances are the smallest. The resulting precision and recall values of SVM and NN classifier are presented in Table 1. All classifiers are quite satisfactory, which is very promising for a real-world verification application. SVM performs better than NN for all tested representations, since it generalizes better, making use of all training data while learning support vectors. The 4144-dimensional output-based generic representation, carrying finer details for the vessels performs the best for both classifiers. Verification performance is slightly lower for end-to-end learning -based representation compared to the 4144-dimensional output-based vessel representation. One reason may be the limitation in random and insufficient sampling of image pairs out of 4035 different vessels during training.

	Representation	True positives	True negatives	False positives	False negatives	Accuracy	Precision	Recall
NN	109-dimensional output based	44,978	40,198	9,802	5,022	85.18%	0.82	0.90
SVM	109-dimensional output based	45,503	45,422	4,578	4,497	90.93%	0.91	0.91
NN	4144-dimensional output based	47,305	41,148	8,852	2,695	88.45%	0.84	0.95
SVM	4144-dimensional output based	46,225	47,744	2,256	3,775	93.97%	0.95	0.92
NN	Siamese network based	44,459	40,390	9,610	5,541	84.85%	0.82	0.89
SVM	Siamese network based	45,869	46,150	3,850	4,131	92.02%	0.92	0.92

5.1.2 Vessel retrieval

Compelling amount of research efforts [27–30] have been put on content-based image retrieval (CBIR) as volumes of image databases are dramatically growing. Particularly, vessel retrieval is another promising application, potentially required in a maritime security system, where a user would like to query a database with a vessel image and retrieve similar images. It may also help annotating vessel images uploaded to a database when no meta-data is present. In our application, the retrieved content is not chosen as either the superclasses of vessel types that we constructed as the coarse attribute in Section 4.3, or the IMO number (aiming to identify the exact vessel), which is too fine for a retrieval task (This is studied as a recognition problem in Section 5.1.3.). Instead, we use 109 vessel types of the 8000 unique vessels with 50 example images, as the content for the retrieval task. We perform content based vessel retrieval (CBIR), using Euclidean (L ²) and chi-squared (χ ²) distances as the similarity metric for four different vessel representations.

The first representation is one of the presented generic descriptions for marine vessels, a 109-dimensional classifier output of the network, trained on IMO training set. The second representation is the 4144 dimensional output-based generic description designed for distinguishing both vessel types and identities. Third representation is based on a Siamese network similar to the one, end-to-end trained in Section 5.1.1. However, this network focuses on matching vessel types. On the other hand, we also compare these learned deep representations (employing the content information) with another effective representation, designed for object classification. Hence, we use pre-learned VGG-F weights to extract 4096-dimensional features. We train a multi-class SVM to train a classifier for 109 vessel types on the IMO training set. For each example, classifier responses of dual combinations of 109 classes (generated during the multi-class SVM phase) are utilized as \(\dbinom {109}{2}\) dimensional feature vectors. By utilizing these four representations, various numbers of images are retrieved and mean average precision curves are generated, as depicted in Fig. 8.

Here, the deep representations learned specifically for maritime vessels significantly outperform the deep representation (VGG-F) learned for general object categorization for 1000 classes [2, 20] for both distance metrics. In addition, χ ² distance is superior in CBIR than L ² distance, for the tested representations. A 109-dimensional output-based generic representation performs the best in this experiment, since it is specifically designed for learning vessel types. The retrieval performance of Siamese network, utilizing end-to-end learning, is lower, compared to 109 and 4144-dimensional representations.

5.1.3 Vessel recognition

Visual object recognition is one of the most crucial topics of computer vision. Especially, face recognition has been studied extensively, and state-of-the-art methods [31, 32], which perform effectively on the benchmark datasets [33–35], have been proposed. Since encouraging performance results are obtained with recent methods, another application performed, utilizing MARVEL, is vessel recognition task, where the ultimate goal is to perceive a vessel’s identity by its visual appearance. It might not be meaningful for object types, other than maritime vessels or faces, such as cars, since same car models with same color have no visual differences and technically are not distinguishable. Nevertheless, individual vessels generally carry distinctive features, as shapes of vessels belonging to the same vessel type category may vary significantly due to their customized construction processes. Here, we utilize the learned generic vessel representations as feature vectors for vessels.

We perform identification for two scenarios. First, we assume the vessel type labels are provided. Hence, recognition is performed among individual classes separately, e.g., vessels belonging to the passenger ships class are learned and recognized. Multi-class SVMs are trained for images belonging to each vessel type and classification is done. Among the 3965 vessels in IMO test set, there exist 29 vessel types that have at least 10 unique vessels, and each unique vessel has 50 example images. For recognition, we first divide the examples of each vessel into fivefolds where each fold has 10 examples per vessel. The training and testing sets contain fourfolds (40 examples) and onefold (10 examples) per vessel, respectively. We perform fivefold cross-validation for classifying all 50 example images of each vessel. For each multi-class SVM, the number of classes equals the number of unique vessels of that particular vessel type. In Fig. 9, the recognition performances are illustrated for each vessel type and by using each generic vessel representation as feature vectors. Representations trained over 4035- and 4144-dimensional output labels, which aim to learn specific vessels in IMO training set, perform significantly better than the representation trained on 109-dimensional output labels which only learns vessel types on IMO training set. Being able to learn both, hence extracting both coarse and fine details, 4144 dimensional output-based representation is the best of three for generic vessel description. Random chance for recognition is also depicted in the figure in order to prove the success of the presented generic marine vessel representations. Additionally, we tested the performance of 4144 dimensional representation when employing a deeper neural network VGG-VD-19 [20], and we obtain high performance similarly.

Vessels belonging to research survey vessels, suction dredgers, and supply vessels type classes of are the most distinguishable ones with recognition accuracies above 90%. On the other hand, vessels of crude oil tankers, vehicle carriers, and containership classes have less distinct differences and a slightly lower recognition performances are achieved, compared to the rest of the classes. Please note that, as number of unique vessels increase in a vessel type group, the random chance and recognition rates slightly decrease as expected, since it becomes a more challenging recognition problem. Yet, recognition accuracies over 77% can be obtained even though the number of unique vessels exceeds a hundred, such as in ro-ro cargo and chemical tanker vessel types.

As a second scenario for recognition, we attempt recognition of vessels when there is no prior information, namely, when type labels are not present. Here, the goal is to classify images of 3965 vessels in IMO testing set by the use of generic vessel representations learned on images of IMO training set. Large number of classes makes it computationally infeasible to train models with a SVM; thus, we employ a nearest neighbor classifier for this experiment. In a similar setting, we split images of individual vessels in IMO testing set into five non-overlapping folds (fourfolds as a training and onefold as a testing split), and we perform fivefold cross-validation for and classify all 50 example images of each vessel. For each image in a testing fold, we find the best matching image among training images and assign its label for the test image. Repeating the same experiment for four generic representations, we conclude that 4144-dimensional output-based representations (AlexNet based and VGG-VD-19 based) perform better than the other two. The recognition rates are listed in Table 2.

	109-dim. output based representation	4035-dim. output based representation	4144-dim. output based representation	4144-dim. output based representation (VGG-VD-19)
Recognition accuracy	23.87%	59.25%	65.13%	65.78%

5.2 Vessel attribute prediction and classification

MARVEL dataset includes several labeled vessel attributes some of which relate to the visual content. Here, as interesting applications, by studying only the visual content, we targeted predicting and classifying four important attributes: draught, gross tonnage, length, and summer deadweight.

The draught of a vessel is a measure describing the vertical distance between the waterline and the bottom of vessel hull. Draught, defining the minimum depth of water a vessel can operate, is an important factor for navigating and routing vessels while avoiding shallow water pathways. Length of a vessel does matter for navigation and marine traffic routing, as well as for calculating fees during vessel registration. Consequently, estimating length of a vessel effectively from a single image may be very beneficial for maritimeapplications. Gross tonnage is a nonlinear measure calculated based on overall interior volume (from keel to funnel) of a vessel. It is important in determining the number of staff, safety rules, registration fees, and port dues. Summer deadweight defines how much mass a ship can safely carry. It excludes the weight of the ship and includes the sum of the weights of cargo, fuel, fresh water, ballast water, provisions, passengers, and crew [36].

Such efforts of attribute estimation is especially valuable for coastal guarding and surveillance, since it allows grasping the physical specifications of a vessel remotely and only by a captured image. In order to achieve these objectives, we both test the use of our powerful 4144-dimensional output-based generic vessel representation and also employ specific attribute-based deep representations. Please note that estimating these attributes are very challenging due to the lack of notion of scale, pose, perspective, camera parameters, etc. The only available information is the appearance of a vessel. For all experiments of attribute prediction, we learn models in IMO training set and evaluate performances of the learned models in IMO testing set. Images missing valid attribute labels were not used in these experiments. Attribute labels, as opposed to being discrete numbers as in vessel type labels or IMO number labels, are continuous and might be unique for each vessel.

We design two sets of experiments: regression and classification. Approaching the problem as a regression task, we represent vessel images by either generic deep models we designed for marine vessels or deep models trained for estimating specific attributes. As in the previous experiments, we extract the penultimate layer activations of the trained networks as feature vectors and utilize a support vector regressor [25, 37] for prediction. For learning attribute-specific deep models, we use AlexNet as a base CNN architecture and modify the last loss layer with an objective to minimize an L2-norm loss, approaching the problem as a least squares regression. For performance evaluation, we compute two measures.

The first measure is Pearson correlation coefficient between predicted labels and manual truth. It is defined as,

$$ r = \frac{\sum_{i=1}^{N}(\hat{y}_{i}-\bar{\hat{y}})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{N}(\hat{y}_{i}-\bar{\hat{y}})^{2}}\sqrt{\sum_{i=1}^{N}(y_{i}-\bar{y})^{2}}}, $$

(4)

where \(\hat {y}_{i}\) and y _i are single indexed samples of predicted labels and true labels, respectively. N is the sample size, which is 158,850, corresponding to all test images with valid attribute labels. These results are given in Table 3. The highest correlations obtained are 0.9042 for length, 0.7911 for draught, 0.8301 for gross tonnage, and 0.7930 for summer deadweight.

	Draught	Gross tonnage	Length	Summer deadweight
SVM	0.7556	0.8301	0.8696	0.7930
CNN	0.7911	0.2699	0.9042	0.0830

The second measure we report is the coefficient of determination, namely R ², which quantifies how well regression model fits the data. It is calculated as,

$$ R^{2} = \frac{\sum_{i=1}^{N}(\hat{y}_{i}-\bar{y})^{2}}{\sum_{i=1}^{N}(y_{i}-\bar{y})^{2}}. $$

(5)

Table 4 shows the R ² values when predicting four attributes. SVM classifier employs the generic representation learnt for vessel type classification, whereas CNN employs a representation specifically learnt for predicting attributes. Table 4 shows that attribute-based representation performs better for predicting length and draught; nevertheless, it performs slightly worse for gross tonnage and summer deadweight. Thus, we may conclude that for predicting physical attributes, values of which are visually explicit, specific representations are more effective. For predicting attributes such as weight, our method relies on vessel type classification.

	Draught	Gross tonnage	Length	Summer deadweight
SVM	0.598	0.554	0.743	0.481
CNN	0.770	0.419	0.863	0.466

For further analysis, we plot predicted draught values for four example vessel categories separately in Fig. 10. The annotated attributes differ for individual vessels within specific vessel categories. However, the significant correlations, between the true values and predicted values for vessels belonging to the same types, show that learnt representations, capturing visual cues, are effective in attribute prediction. The trained neural networks simply try to estimate vessel attributes similar to how human can do, based on clues such as vessel type and also appearance (visible parts of a vessel).

As another experiment, we quantize the attribute labels and relabel and assign the images in IMO training set accordingly to 20 distinct classes such that each class has equivalent number of examples for a balanced training. Next, we train a multi-class classifier, using both the generic vessel representation (combined with a nonlinear SVM) and also specific deep representations (softmax classifier) for each attribute. For instance, in training, we use a total of 134,000 images for draught, 142,000 images for gross tonnage, 140,000 images for length and 148,000 images for summer deadweight. For testing, we use all 158,850 images of IMO test set for which all attribute annotations are present. Top five classification accuracies for the attributes and employed representations are summarized in Table 5. Though generic vessel representation performs reasonably well, trained deep models which focus on specific attributes are significantly better in attribute categorization. The classification results are also depicted as normalized confusion matrices in Figs. 11, 12, 13, and 14. The imbalance of the training set results in coarser ranges for classes around the extrema values and very fine classes otherwise. The entries of the confusion matrices are high valued along the diagonal entries, which shows that the learned models are effective in capturing the desired attribute information.

Classified attribute	Employed representation	Top 1 accuracy	Top 2 accuracy	Top 3 accuracy	Top 4 accuracy	Top 5 accuracy
Draught	Generic model + SVM	0.1302	0.3104	0.4432	0.5506	0.6320
Gross tonnage	Generic model + SVM	0.4755	0.6393	0.7418	0.8178	0.8678
Length	Generic model + SVM	0.4539	0.6345	0.7317	0.8019	0.8510
Summer deadweight	Generic model + SVM	0.4304	0.6209	0.7310	0.7998	0.8525
Draught	Attribute-specific trained CNN	0.1834	0.4159	0.5761	0.6884	0.7774
Gross tonnage	Attribute-specific trained CNN	0.5515	0.7492	0.8556	0.9131	0.9454
Length	Attribute-specific trained CNN	0.5289	0.7266	0.8257	0.8896	0.9328
Summer deadweight	Attribute-specific trained CNN	0.5155	0.7364	0.8317	0.8938	0.9288

¹ A negative pair indicates a pair of different vessel images, whereas a positive pair corresponds to a pair of vessel images belonging to a unique vessel.