automating mashup service recommendation via semantic and...

Research ArticleAutomating Mashup Service Recommendation via Semantic andStructural Features

Wei Xiong 12 Zhao Wu 1 Bing Li 2 and Bo Hang 1

1Hubei University of Arts and Science Xiangyang 441000 China2International School of Software Wuhan University Wuhan 430072 China

Correspondence should be addressed to Zhao Wu wuzhao73163com

Received 4 February 2020 Revised 9 June 2020 Accepted 6 July 2020 Published 12 August 2020

Guest Editor Chunlai Chai

Copyright copy 2020 Wei Xiong et al -is is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Increasing physical objects connected to the Internet make it possible for smart things to access all kinds of cloud services Mashuphas been an effective way to the rapid IoT (Internet of -ings) application development It remains a big challenge to bridge thesemantic gap between user expectations and application functionality with the development of mashup services -is paperproposes a mashup service recommendation approach via merging semantic features from API descriptions and structuralfeatures from the mashup-API network To validate our approach large-scale experiments are conducted based on a real-worldaccessible service repository ProgrammableWeb -e results show the effectiveness of our proposed approach

1 Introduction

Internet of -ings (IoT) was firstly introduced to thecommunity in 1999 for supply chain management Now wewill arrive to the post-cloud era where there will be largeamounts of smart things to access all kinds of cloud servicesand the capabilities of smart things can be enhanced byinteracting with other functional entities through the in-terfaces of cloud services Since the functionality of an in-dividual service is too simple to satisfy the complexrequirements of users in an IoTapplication people prefer tocombine several cloud services together In recent yearsmashup technology has gained great attention since it cansupport the development of IoT applications by composingexisting cloud services in the form of web APIs

Several platforms including Seekda1 Google andProgrammableWeb [1] enabled vibrant software ecosystemsbetween service providers and developers where developersutilize one or more query items such as keywords andcategory However keyword-based service recommendationmechanisms usually have low accuracy [2] NLP techniquesare increasingly applied in the software engineering domainwhich have been shown to be useful in requirements en-gineering [3] usability of API documents [4] and other

areas [5] -us semantic queries may get more accurateresults

Most of API descriptions and the tags are all text -enthese approaches use similar semantic metrics that capturethe service similarity such as the similarity of service de-scriptions and tags Indeed most of similarity metrics whichare proposed to quantify the similarity of service descrip-tions and the similarity of tags are based on the semanticinformation in the text Some works utilized structuralmetrics to quantify the structural similarity where themetrics reached the topological information extracted frommethods attributes classes and their couplings etc [6ndash8]However there are very few works in which structuralsimilarity has been introduced to guide the service ranking

We present AMSRSSF (automating mashup servicerecommendation via semantic and structural features) aframework that utilizes NLP and graph-embedding tech-niques to recommend services for developers in this studyAMSRSSF takes as input the developersrsquo personalized re-quirements and structural semantic and determines whichservices can be recommended for developers Furthermorewe show that the structural semantics can be generated froma two-mode graph which can describe mashups web APIsand their relations We evaluate AMSRSSF against a dataset

HindawiMathematical Problems in EngineeringVolume 2020 Article ID 4960439 10 pageshttpsdoiorg10115520204960439

of description documents including 10050 web-based ser-vices and 7155 mashups Our experiments demonstrate thatour approach can rank mashup services efficiently and itsperformance is better than semantic-based mashup serviceranking approaches alone

-is paper makes the following main contributions

(1) We propose a mashup service recommendationapproach via merging semantic features from APIdescriptions and structural features from themashup-API network

(2) We conduct comprehensive experiments on a real-world dataset demonstrating the effectiveness of ourapproach

-e remainder of this paper is organized as followsSection 2 gives a motivating scenario and presents ourapproach Section 3 describes the experiments in detailSection 4 discusses the related works and Section 5 con-cludes the paper with future work directions

2 Automating Service Recommendation viaSemantic Features and Structural Features

In this section we first present a scenario to illustrate themotivation of our work in Section 21 -en we discuss theissues ofmashup service recommendation in Section 22 nextwe propose a mashup service recommendation approach viasemantic and structural features in Sections 23ndash28

21 Motivation Under Internet scenario more structuresare presented between data objects A typical scenario is aknowledge graph that consists of a global relationship di-agram of user behavior data and an item with more attri-butes Structural and semantic similarities characterizedifferent aspects of the service similarity Indeed twosimilarities are orthogonal which motivate our work Weexpected that better service recommendation approachesmight be proposed by using structural features In particularwe propose a simple but effective approach AMSRSSF torank mashup services by using semantic and structural fea-tures First it applies a two-mode graph to describe mashupsweb APIs and their relations formally Second we quantifythe structural similarity between every pair ofmashup servicesbased on the two-mode graph-e structural similarity resultsfrom the structural context in which web APIs are used bymashup services Finally we introduce a merging embeddingvectors algorithm that only considers the pairwise similaritiesbetween mashup services to rank them effectively

According to the above it is certainly valuable to in-troduce structural features and then improve the accuracy ofrecommendation based on structural semantics

Finally there are some other issues which need to beaddressed

(1) How to extract semantic from natural language APIdescriptions

(2) How to generate structural semantics of the mashup-mashup network

(3) How to merge multiple embedding vectors for betteraccuracy

(4) How do we design experiments for performanceevaluation

22 Problem Description Our work considers the task ofrecommendation via representation learning as followsgiven a requirement q described in multidimensional in-formation which includes semantic and structural featuresthe corresponding recommended services are given viasimilarity calculation which exists in terms of the repre-sentation learning model

Our model consists in learning a function S(middot) which canscore requirement-service pairs (q t) Hence finding thetop-ranked answer t(q) to a requirement q is directly carriedout by

1113954t(q) argmaxtprimeisinK

S q tprime( 1113857 (1)

and to handle multiple recommended services wepresent the results as a ranked list rather than taking the topservice

23 Overview of Our Framework In this section we willpresent an overview of our approach As shown in Figure 1our approach mainly consists of four components semanticextraction based on NLP pipeline structural embeddinggeneration of the mashup-mashup network merging ofmultiple embeddings and recommendation based on fusionembedding More specifically the descriptions of web-basedservices will be crawled from ProgrammableWeb

We firstly preprocess natural language API descriptionsusing NLP pipeline and extract text semantic embeddingWe then generate structural embedding of the mashup-mashup network next we merge semantic embedding andstructural embedding Finally the developers give multi-dimensional information of requirements which will beparsed into semantic and structural features and passed tosimilarity calculation for ranking (Figure 2)

24 Semantic Extraction Based on the NLP Pipeline -issection preprocesses the sentences in API descriptions usingNLP pipeline named as NLTK [9] which resembles thefollowing high-level flow in Figure 3

For purposes of illustration we introduce a sample asfollows

ldquo-e Twitter microblogging service includes twoRESTful APIs -e Twitter REST API methods allowdevelopers to access core Twitter datardquo

We have the following steps

241 EOS (End of Sentence) Detection Breaking the textsinto paragraphs might be important However it may beunlikely to help EOS detection whichmarks the boundary ofa sentence from texts and breaks them into sentences We

2 Mathematical Problems in Engineering

Structural embedding

NLP

hellip

H

Output matrix

Classifier

Hidden representation

Dense embeddingsSemantic embedding Stucture embedding

ProgrammableWeb

User Recommendation forweb-based services

Recommendationresults

Candidate poolof web services

Google AJAX Search Facebook Google Maps Flickr GlobalBiodiversity Information

Facility

Facebookfriend plotter

Biodiversitytaxonomic catalogue

1

2

API-mashup network

Blog on a map

Facebookfriend plotter Blog on a map


Mashup-mashup network

Fusion embedding

Semantic embedding α0 α1

Figure 1 Overview of our approach

End ofsentence (EOS)

detectionTokenization Tokenization Chunking Named entity

extraction

Figure 2 Overview of the NLP pipeline


Facility



1

2

API-mashup network

Blog on a map


Blog on a map



Figure 3 Mashup-API network and mashup-mashup network

Mathematical Problems in Engineering 3

utilize them to represent logical units of thought and tend togive a predictable syntax for further analysis

We will obtain the following results by parsing a sen-tence with NLTK

[ldquo-e Twitter micro-blogging service includes twoRESTful APIs -e Twitter REST API methods allowdevelopers to access core Twitter datardquo]

A period ldquordquo is utilized to mark the end of a sentenceHowever there are other legal usages of the period such

as decimal (periods between numbers) ellipsis ldquordquo andshorthand notations just like ldquoMrrdquo and ldquoDrrdquo We canovercome this difficulty by looking up abbreviations fromWordNet and detecting decimals including period characterby regular expressions

Moreover there are instances where an enumeration listis used to describe data type such as the following

ldquo-is service where you can obtain the following data (a)abc (b) xyz rdquo

It is prone to understand the implication for a humanbut it is arduous to seek appropriate boundaries as far as amachine is concerned An improvement over breaking oncharacters ldquordquo is leveraging the Syntax information

(1) Placements of tabs(2) Bullet points (numbers characters roman numerals

and symbols)(3) Delimiters such as ldquordquo to detect appropriate

boundariesWe further improve the EOS detection using thefollowing patterns

(1) We remove the leading and trailing ldquolowast rdquo and ldquo-rdquocharacters in a sentence

(2) We consider the following characters as sentenceseparators ldquondashrdquo ldquo-rdquo ldquo1113928rdquo ldquosectrdquo ldquo￥rdquo ldquordquo ldquordquo ldquo|rdquo ldquosimrdquoand ldquordquo

(3) For an enumeration sentence we split the sentenceinto short ones for each enumerated item

Currently sentence detection has occurred for arbitrarytext

242 Tokenization We then split individual sentences intotokens leading to the following results

[[ldquo-erdquo ldquoTwitterrdquo ldquomicrobloggingrdquo ldquoservicerdquo ldquoin-cludesrdquo ldquotwordquo ldquoRESTfulrdquo ldquoAPIsrdquo ldquordquo] [ldquo-erdquo ldquoTwit-terrdquo ldquoRESTrdquo ldquoAPIrdquo ldquomethodsrdquo ldquoallowrdquo ldquodevelopersrdquoldquotordquo ldquoaccessrdquo ldquocorerdquo ldquoTwitterrdquo ldquodatardquoldquordquo]]

Here tokenization appeared to split on whitespace andit tokenized out EOS correctly as well We further improvethe discrimination by determining whether a period is theend of sentence marker or part of an abbreviation

243 POS (Parts of Speech) Tagging Part of speech of eachtoken in a sentence means ldquoword taggingrdquo or ldquogrammaticaltaggingrdquo where the state-of-the-art approaches have beenshown to achieve 97 accuracy in classifying POS tags for

well-written news articles We can obtain the followingresults

[[(ldquo-erdquo ldquoDTrdquo) (ldquoTwitterrdquo ldquoNNPrdquo) (ldquomicro-blog-gingrdquo ldquoNNPrdquo) (ldquoservicerdquo ldquoNNPrdquo) (ldquoincludesrdquoldquoVBZrdquo) (ldquotwordquo ldquoNUMrdquo) (ldquoRESTfulrdquoldquorsquoJJrdquo)(ldquoAPIsrdquoldquorsquoNNPrdquo)] [(ldquo-erdquo ldquoDTrdquo) (ldquoTwitterrdquo ldquoNNPrdquo)(ldquoRESTrdquo ldquoNNPrdquo) (ldquoAPIrdquo ldquoNNPrdquo) (ldquomethodsrdquoldquoNNPrdquo) (ldquoallowrdquo ldquoVBrdquo) (ldquodevelopersrdquo ldquoNNPrdquo) (ldquotordquorsquoINrdquo) (ldquoaccessrdquo ldquoVBrdquo) (ldquocorerdquo ldquoJJrdquo) (ldquoTwitterrdquoldquoNNPrdquo) (ldquodatardquo ldquoNNPrdquo) (ldquordquo ldquordquo)]]

ldquoNNPrdquo manifests a noun as the part of a noun phraseldquoVBDrdquo manifests a verb in simple past tense ldquoDTrdquo manifestsan article ldquoINrdquo manifests a preposition ldquoNUMrdquo manifests anumeral and ldquoJJrdquo manifests an adjective

After finishing POS tagging we can chunk nouns as partof noun phrases and then try to deduce the types of entities(eg people places organizations etc)

244 Chunking Chunking means to analyze tagged tokensfrom sentence and identify logical concepts such as ldquoPan-dora Internet radiordquo and ldquoGoogle Mapsrdquo We annotate suchphrases as single lexical units such as a noun phrase and averb phrase -e state-of-the-art approaches achieve around97 accuracy in classifying phrases and clauses over well-written news articles4 We develop chunking with morecomplex grammars to substitute default one in NLTK

245 Named Entity Extraction Named Entity Extractionmeans analyzing chunks and further tagging the chunks asnamed entities such as people organizations and objectsWe will obtain the following results

[Tree(ldquoSrdquo [Tree(ldquoOBJrdquo [(ldquo-erdquo ldquoDTrdquo) (ldquoTwitterrdquoldquoNNPrdquo) (ldquomicrobloggingrdquo ldquoNNPrdquo) (ldquoservicerdquoldquoNNPrdquo)]) (ldquoincludesrdquo ldquoVBZrdquo) Tree(ldquoOBJrdquo [(ldquotwordquoldquoNUMrdquo) (ldquoRESTfulrdquo ldquoJJrdquo) (ldquoAPIsrdquo ldquoNNPrdquo) (ldquordquoldquordquo)]) Tree(ldquoSrdquo [Tree(ldquoOBJrdquo [(ldquo-erdquo ldquoDTrdquo) (ldquoTwit-terrdquo ldquoNNPrdquo) (ldquoRESTrsquordquo ldquoNNPrdquo) (ldquoPIrdquo ldquoNNPrdquo)(ldquomethodsrdquo ldquoNNPrdquo)]) (ldquoallowrdquo ldquoVBrdquo) Tree(ldquo-PERSONrdquo [(ldquodevelopersrdquo ldquoNNPrdquo)]) (ldquotordquo ldquoINrdquo)(ldquoaccessrdquo ldquoVBrdquo) (ldquocorerdquo ldquoJJrdquo) (ldquoTwitterrdquo ldquoNNPrdquo)(ldquodatardquo ldquoNNPrdquo) (ldquordquo ldquordquo)]

We can identify that ldquo-e Twitter microblogging ser-vicerdquo ldquotwo RESTful APIsrdquo and ldquo-e Twitter REST APImethodsrdquo have been tagged as an object and ldquodevelopersrdquohas been tagged as a person

In addition abbreviations usually lead to incorrectparsing for sentence for example ldquoApplication ProgrammingInterfaces (APIs)rdquo is treated as single lexical unit We canwork it out looking up abbreviations from WordNet [10]

246 Semantic Embedding Generation In order to convertnamed entity into feature vectors and utilize them in lattermodels we utilize word2vector to encode named entitiesfrom API descriptions


Word2vector is a powerful method to extract meaningfulgrammatical and semantic features which can transform aword into a fixed-length vector Each named entity fromAPIdescription can be reduced to embedding vector Vtext andsemantic embedding of API description can then be rep-resented as a fixed-length vector as follows

Vm average Vtexti( 1113857

I 1 2 Nm1113864 1113865(2)

where Nm is the number of named entities of APIdescription

25 Two-Mode Graph of Mashup Services and Web APIs-e first step of structural embedding generation is togenerate the mashup-mashup network -e network can begenerated through mashup behavior sequence -e infor-mation such as the same APIs between mashups can beutilized to establish the edges between mashups as well so asto generate structural graph -e embedding vectors basedon this graph can be called structural embedding vectors

In this article two networks are utilized to represent allrelationships of mashup services and APIs one is the net-work between API and mashup services and the other is thenetwork between mashup services

Definition 1 MAN (mashup-API network) is a network of2 modes where the nodes represent the API and themashup services and edges represent the relationshipbetween them

Definition 2 MMN (mashup-mashup network) is aweighted network between mashups where each noderepresents a mashup service and edges represent the rela-tionship between them the weight of edge represents therelevancy between them MMN can be represented with atriple

MMN N E W (3)

where N is a node set of all mashup services E is a set of alledges in MMN network W is an edge weights matrix Forexample if Nodei and Nodej are linked then Wij isdefined as the number of API intersections of two mashupservices

26 Structural EmbeddingGeneration of theMashup-MashupNetwork Structural embedding is designed to represent anetwork in a low-dimensional space and retain as muchinformation as possible By using structural embeddingeach node in the network can be represented as a vector

-e traditional sequence embedding is inadequate fornetwork structure We can conduct a random walk on thegraph structure to generate a large number of node se-quences which were input into word2vec as trainingsamples and obtained structural embedding Figure 4 il-lustrates the random walk process graphically

As shown in Figure 4 random walk process firstly selectsrandom mashup as starting point and produces the

sequence where the hop probability of the random walkneeds to be formally defined that is the probability oftraversing vj after reaching node vi If the graph is anundirected graph then the hop probability from node vi tonode vj is defined as follows

P vi vj

111386811138681113868111386811138681113874 1113875

(1 minus d) + dwij

1113936jisinN vi( )wij

⎛⎝ ⎞⎠

0 eij notin ε

⎧⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎩

(4)

where N(vi) is the set of all outgoing edges of node vi andwij

is the weight of edge from nodevi to node vjWe further tradeoff the homogeneity and structural

equivalence of network embedding by adjusting the weightof random walk

Specifically the homogeneity of the network means thatthe embeddings of nodes that are close to each other shouldbe as similar as possible As shown in Figure 5 the em-bedding expression of node u should be close to the nodeswhich are connected to s1 s2 s3 and s4 -at is the em-bodiment of homogeneity

Structural equivalence means that the embeddings ofnodes with similar structures should be as close as possibleIn Figure 4 both node u and node s6 are the central node oftheir local networks respectively which are similar instructure -eir embedding expressions should be similarwhich is the embodiment of structural equivalence

Random walk needs to be inclined to the process ofbreadth first search (BFS) in order to express the ho-mogeneity of the network -ere will be more homoge-neous information in generated sequence because BFSprefers access to direct connection On the other handdepth-first search (DFS) can grasp the overall structure ofthe network through multiple hops However it is im-portant to trade off the tendency of BFS and DFS For-mally the hop probability from node v to node x isenhanced as follows

P(v|x) λpq(t x) middot P(v|x) (5)

where λpq(t x) is defined as follows

A

B

D E

CF

A B E F

B E C B A

C B A B E

D A B E C

E C B A

A B E C B

Figure 4 Random walk on network


λpq(t x)

1pr

if dtx 0

1 if dtx 1

1pio

if dtx 2

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

(6)

where dtxrefers to the distance from node t to node x andthe parameters pr and pio jointly control the tendency ofrandom walk Parameter pr is known as return parameterwhich controls the probability of accessing the nodevisited repeatedly Parameter pio is called in-out param-eter which controls the outward or inward tendencyRandom walk tends to access nodes close to node t ifpio ge 1 and tends to access nodes far from node t if pio le 1Our approach enhances Node2vec [11] by replacing wij asP(v|x)

Figure 6 shows the hop probability of our approach whilejumping from node t to node v in the next step

-e homogeneity and structure equivalence of thenetwork embodied by equation (5) can be intuitivelyexplained in the recommendation system Items with thesame homogeneity are likely to be items of the same cat-egory and attributes while items with the same structureare items with similar trends or structural attributes -ereis no doubt that both of them are very important features inrecommendation systems

Next we should encode above node sequences intostructural embedding Our model named sequence2-vector is derived from skip-gram where the nodesequence also utilizes central node sequence to predictthe context and its data structure can be represented by atriplet stminus1 st st+11113864 1113865 st is input and stminus1 st+11113864 1113865 is output-e neural network structure of the sequence2vectormodel is the common encoder-decoder architectureusing the GRU model -erefore word vector is utilizedwhile training sequence vector the output of encoder isembedding vector of last word of node sequences -eformulas for the realization of encoding stage are asfollows

rt

σ Wrxt

+ Urhtminus 1

1113872 1113873

zt

σ Wzxt

+ Uzhtminus 1

1113872 1113873

ht tan h Wdx

t+ U r

t⊙htminus 11113872 11138731113872 1113873

ht

1 minus zt

1113872 1113873⊙htminus 1+ z

t⊙ht

(7)

where htrepresents the output of the hidden layer at the timeof t While we take st+1 as an example the formula for therealization of decoding stage is as follows

rt

σ Wdr x

tminus 1+ U

dr h

tminus 1+ Crhi1113872 1113873

zt

σ Wdzx

tminus 1+ U

dzh

tminus 1+ Czhi1113872 1113873

ht tan h Wdx

tminus 1+ U

dr

t⊙htminus 11113872 1113873 + Chi1113872 1113873

hti+1 1 minus z

t1113872 1113873⊙htminus 1

+ zt⊙ht

(8)

where Cr Cz and C are used as vector bias for reset gateupdate gate and hidden layer respectively Sequence2vectorcan even be combined into the subsequent deep learningnetwork to retain different features of objects (Algorithm 1)

27 Embedding Fusion From the above we have got se-mantic embedding of API description and structural em-bedding through mashup behavior sequence -en it isimportant to integrate multiple embedding vectors to form thefinal embedding of mashup -e simplest approach is to addthe average pooling layer into the deep neural network toaverage different kinds of embedding On this basis we haveadded weights to each embedding as shown in Figure 7 -ehidden representation layer is the layer that performs theweighted average operation on different kinds of embeddingAfter the weighted average embedding vector is obtained it isdirectly input into the softmax layer In this way the weightedαi(i 0 1) can be trained

X1X2

V

t

X3

λ = 1 λ = 1pio

λ = 1pioλ = 1pr

Figure 6 Hop probability of the network

S1S2

S3

S4

S5

S6

S7

S8

S9

u

Figure 5 Homogeneity and structural equivalence of the network


In the actual model we adopted eαi instead of αi as theweight of corresponding embedding which can avoid theweight being 0 and own good mathematical propertiesduring the process of gradient descent

Our model finally inputs semantic and structureembedding into the model to generate the final fusionembedding

28 Interfering Semantic-Related Services In this section weutilize fusion embedding from the above to calculate thesimilarity between mashups as follows

sim hi hj1113872 1113873 exp minus hi minus hj

11113874 1113875 (9)

where 1 is theManhattan distance Finally we recommendthe top-k mashups to developers

3 Experiments

In this section we conduct experiments to compare ourapproach with state-of-the-art method Our experiments areintended to address the following questions

(1) Does structural embedding improve the effectivenesswith respect to semantic similarities

(2) What is the performance of top-k rankings of ourembedding models

31 Data Set Description We adopt a real-world datasetProgrammableWeb which was crawled from all registeredservices in publically available repository until Nov 252015 -e dataset contains description documents in-cluding 12250 web-based services and 7875 mashupswhich have attributes such as name tags summary anddescription

32 Metrics In statistical classification [12] Precision isdefined as the ratio of the number of true positives to thetotal number of items reported to be true and Recall isdefined as the ratio of the number of true positives to thetotal number of items that are true F-score is defined as theweighted harmonic mean of Precision and Recall Accuracyis defined as the ratio of sum of true positives and truenegatives to the total number of items Higher values ofprecision recall F-score and accuracy indicate higherquality of identifying the semantic relationship Based on thetotal number of TPs FPs TNs and FNs we computed theprecision recall and F-score as follows

Classifier

Output matrix


Dense embeddings

Semantic embedding Structural embedding

H

α0 α1

Figure 7 Embedding fusion

InputMMNOutputstructure embedding vector hBodyRandom select a starting point for random walk using (5) and produce sequence sh sequence2vector (s)return h

ALGORITHM 1 Structure embedding algorithm


precision TP

TP + FP

recall TP

TP + FN

F minus score 2 timesprecision times recallprecision + recall

(10)

where TP denotes true positives FP denotes false positivesFN denotes false negatives and TN denotes true negatives

MAP (mean average precision) is the average of preci-sions computed at the point of each of the relevant docu-ments in the ranked sequence whose equation is as follows

MAP 1

QR

11138681113868111386811138681113868111386811138681113868

1113944qisinQR

AP(q) (11)

where AP is average precision and QR denotes relevantdocuments

We utilize the normalized discounted cumulative gain(NDCG) [13] metric as well which is a popular metric forevaluating ranking results Given ideal rankings (used asground truth) and the derived rankings the NDCG value oftop-k ranked services can be calculated by the followingformula

NDCGk DCGk

IDCGk

(12)

where DCGk and IDCGk denote the discounted cumulativegain (DCG) values of derived rankings and ideal rankings oftop-k services

-e value of DCGk can be calculated by the followingformula

DCGk rel1 + 1113944k

i2

relilog2 i

(13)

where reli is the score of the service at position i of rankings-e premise of DCG is that a high-quality service appearinglower in a ranking list should be penalized as the score isreduced logarithmically proportional to the position of theresult divided by log2 i

33 Baseline Approaches In this section to show the ef-fectiveness of our approach we compare our approach withthe following approaches

(1) SimpleGraph [14] this approach extracts features ofsubgraphs of question-answer pairs and utilizes alogistic regression classifier to predict the probabilitythat a candidate answer is correct

(2) AvgPara the semantics for a given question in thismodel are directly averaged to predict the answers

(3) DataAugment this approach employs semantics fordata augmentation during training which utilize thequestion-answer pairs and semantics to automati-cally generate new training samples

(4) SepPara [15] this approach is the semantic scoringmodel which is trained on classification datawithout considering question-answer pairs

(5) Para4QA [15] this approach results in learning se-mantics where semantic scoring and QA models aretrained end-to-end on question-answer pairs

34 Evaluation Protocols In this section we have designedthe experiments to evaluate our approach We then created aset of (q t l) triples where q is a API description t is amashup-mashup network and l is a label (classes ofmashups) We manually tagged each pair with a label l -isresulted in a set of approximately 20 human judgments -eunion of them composes set T1 which is considered as thebaseline to evaluate the precision of mashuprecommendation

Five authors pick out 30 API descriptions of mashups aswell top-20 rankings are artificially extracted by two authorsat least after analyzing and augmenting them which isconsidered as the baseline to evaluate the rankings of mashuprecommendation -e union of them composes set T2

35 Training Details We utilized encoder-decoder archi-tecture using the GRU model with 2 layers with 800 cells ateach layer and 800 dimensional word embeddings -ecomplete training details are given below

(1) We used word2vector to initialize the word em-bedding matrix Out-of-vocabulary words werereplaced with a special unknown symbol We alsoaugmented API descriptions with start-of-sequenceand end-of-sequence symbols Word vectors forthese special symbols were updated during trainingModel hyperparameters were validated on set T1Each of the structural and semantic features isrepresented by a vector of 64 dimensions -edropout rate was selected from 02 03 04

(2) Parameters were randomly initialized from a uni-form distribution U(minus008008) -e learning ratewas 07 After 5 epochs we began halving thelearning rate every half epoch We trained ourmodels for a total of 20 epochs-e batch size was setto 150 To alleviate the exploding gradient problemthe gradient norm was clipped to 5

(3) We used the DNN model to train weightedαi(i 0 1) -e DNN model takes the combi-nation of semantic and structural features as inputand each feature is represented by a vector of 16dimensions -e total input vector dimension is 32(16 + 16) and output vector dimension vector is thetag To convert the input into a neural node thehidden layer utilizes a linear transformation and thencompresses the nonlinearity -e DNN model uti-lizes back-propagation and adjusts the weight αi

according to the training set Finally there is asoftmax layer that translates the issue into classifi-cation problems


36 Results

361 Answer to RQ1 Table 1 presents the results of theaccuracy experiments AvgPara achieves better accuracycompared with SimpleGraph since semantic embeddingscontain more information than the features of subgraphsDataAugment achieves better accuracy compared withAvgPara since data augmentation works Our approachachieves better accuracy compared with DataAugment sincestructural embedding and semantic embedding are fusedtogether

362 Answer to RQ2 Table 2 presents the results of theranking experiments Our approach obtains the best pre-diction accuracy (largest NDCG values) under all the ex-perimental settings consistently since our approachconsiders fusion embedding of semantic and structuralfeatures which improved the accuracy of similaritycalculation

37 Breats to Validity -e threats to external validity arerelated to the representativeness of the dataset To evaluatethe quality of semantic extraction and service recommen-dation we applied our approach on real-world datasetProgrammableWeb which is suitable for our study due topublic availability and activeness -is threat has beenpartially mitigated by the fact that we collected descriptiondocuments including 12250 web-based APIs and 7875mashups Furthermore we applied our approach on top-10subjects of them as well

-e threats to internal validity of our study are related tothe results of the pairwise comparison on the criteria (seeSection 35) -ere is a possibility that the results are sub-jective as people may be biased It may affect the weighted αi

which is assigned to the semantic and structural embedding-is threat has been partially mitigated by the fact that fiveauthors elicit 10 service description documents from each oftop-10 subjects independently to evaluate the quality of

semantic relationship extraction and query where we man-ually achieve top-10 rankings as the baseline in ourevaluations

4 Related Works and Discussion

Our approach touches some areas such as recommendationDL artifacts and requirements engineering We next discussrelevant works pertinent to our proposed approach in thesedomains

Most of recommendations focused on keyword-basedsearching where the keywords are usually tokenized from theinterface of web-based services such as input output pre-condition and postcondition For example Junghans et algive an oriented functionalities and requests formalizationmethod [16] OWLS-MX is proposed to calculate the simi-larity of concept in OWL-S for recommendation [17] Chenet al give a clustering approach named as WTCluster usingWSDL documents and service tags [18] Mueller et al give anapproach based on nature-inspired swarm intelligence [19]

Semantic parsing offers NLP-based techniques such assyntax directed [20] parse trees [21] combinatory cate-gorical grammars [22 23] and dependency-based semantics[24 25] -ese approaches typically have high precision butlower recall which are sensitive to grammatically incorrector ill-formed descriptions Several approaches utilize eitherNLP or a merging of NLP and machine learning algorithms(SVM driven etc) to infer specifications such as API de-scriptions [26ndash29]

Generally most ranking approaches ranked services byutilizing the text in service profiles (mashup descriptionsAPI descriptions WSDL documents tags etc) to calculatethe semantic similarity between every pair of services Incontrast our proposed approach utilized structural simi-larities to guide the ranking activities which can reduce falsepositives and reach higher performance

5 Conclusion and Future Work

-is paper introduces a mashup service recommendationapproach via merging semantic features from API de-scriptions and structural features from the mashup-APInetwork Our approach can significantly outperform pre-vious works after encoding semantic and structural features

In spite of these promising results some exciting chal-lenges remain especially in order to scale up this model tomore dimensions Furthermore many more modes have tobe carried out to encode the complex semantics into theembedding space

Data Availability

-e data used to support the findings of this study areavailable from the corresponding author upon request

Conflicts of Interest

-e authors declare that there are no conflicts of interestregarding the publication of this paper

Table 1 Performance of variants from the T1 test set

Method Precision () Recall () F-score () Map ()SimpleGraph 8632 2655 4060 1654AvgPara 7799 3213 4551 2262DataAugment 7166 3555 4752 3297Our approach 6643 4258 5189 7365

Table 2 Performance of top-k from the T2 test set

Method NDCG1 () NDCG10 () NDCG20 ()SimpleGraph 4333 5794 6672AvgPara 5756 6344 7854DataAugment 6893 7668 7934SepPara 7230 8084 8249Para4QA 8173 8270 8784Our approach 8327 8583 8876


Acknowledgments

-e work described in this study was fully supported by theNational Key Research and Development Program of China(no 2017YFB1400602) the National Natural ScienceFoundation of China (nos 61572371 61702377 6172820261572137 and 61873309) Science and Technology Devel-opment Program of Central Guide to Local Government ofChina (no 2019ZYYD043) International Science andTechnology Cooperation Program of Hubei Province (no2019AHB059) the Natural Science Foundation of HubeiProvince in China (no 2019CFC870) and Hubei Superiorand Distinctive Discipline Group of ldquoMechatronics andAutomobilesrdquo (XKQ2020016)

References

[1] httpswwwprogrammablewebcom[2] J Lu D Wu M Mao W Wang and G Zhang ldquoRecom-

mender system application developments a surveyrdquo DecisionSupport Systems vol 74 pp 12ndash32 2015

[3] V Gervasi and D Zowghi ldquoReasoning about inconsistenciesin natural language requirementsrdquo ACM Transactions onSoftware Engineering and Methodology vol 14 no 3pp 277ndash330 2005

[4] U Dekel and J D Herbsleb ldquoImproving API documentationusability with knowledge pushingrdquo in Proceedings of the 2009IEEE 31st International Conference on Software EngineeringVancouver BC Canada May 2009

[5] G Little and R C Miller Keyword programming in Java vol16 Springer US Boston MA USA 2007

[6] W Pan B Li J Liu Y Ma and B Hu ldquoAnalyzing thestructure of java software systems by weighted k-core de-compositionrdquo Future Generation Computer Systems vol 83pp 431ndash444 2018

[7] W Pan B Song K Li and K Zhang ldquoIdentifying key classesin object-oriented software using generalized k-core de-compositionrdquo Future Generation Computer Systems vol 81pp 188ndash202 2018

[8] W Pan H Ming C K Chang Z Yang and D K KimldquoElementRank ranking java software classes and packagesusing multilayer complex network-based approachrdquo IEEETransactions on Software Engineering 2019

[9] httpwwwnltkorg[10] httpswordnetprincetonedu[11] A Grover and J Leskovec ldquonode2vec scalable feature

learning for networksrdquo in Proceedings of the 22nd ACMSIGKDD International Conference on Knowledge Discoveryand Data Mining pp 855ndash864 New York NY USA August2016

[12] D Olson Advanced Data Mining Techniques Springer-Verlag Berlin Germany 2008

[13] K Kekalainen and J Kekalainen ldquoCumulated gain-basedevaluation of IR techniquesrdquo ACM Transactions on Infor-mation Systems (TOIS) vol 20 no 4 pp 422ndash446 2002

[14] S Reddy O Tackstrom M Collins et al ldquoTransformingdependency structures to logical forms for semantic parsingrdquoTransactions of the Association for Computational Linguisticsvol 4 pp 127ndash140 2016

[15] L Dong J Mallinson S Reddy et al ldquoLearning to paraphrasefor question answeringrdquo 2017 httparxivorgabs170806022

[16] M Junghans S Agarwal and R Studer ldquoTowards practicalsemantic web service discoveryrdquo in Proceedings of the Ex-tended Semantic Web Conference pp 15ndash29 HeraklionGreece June 2010

[17] M Klusch B Fries and K Sycara ldquoOWLS-MX a hybridsemantic web service matchmaker for OWL-S servicesrdquoJournal of Web Semantics vol 7 no 2 pp 121ndash133 2009

[18] L Chen L Hu Z Zheng et al ldquoUtilizing tags for web servicesclusteringrdquo in Proceedings 2011 International Conference onService-Oriented Computing pp 204ndash218 Paphos Cyprus2011

[19] C Mueller N Kiehne and N Kiehne ldquoKeyword-basedservice matching in a cloud environment using nature-in-spired swarm intelligencerdquo Lecture Notes on Software Engi-neering vol 3 no 4 pp 299ndash302 2015

[20] R J Mooney ldquoLearning for semantic parsingrdquoComputationalLinguistics and Intelligent Text Processing Springer BerlinGermany pp 311ndash324 2007

[21] R Ge and R J Mooney ldquoA statistical semantic parser thatintegrates syntax and semanticsrdquo in Proceedings of the 9thConference on Computational Natural Language Learningpp 9ndash16 Stroudsburg PA USA 2005

[22] L Zettlemoyer and M Collins ldquoLearning to map sentences tological form structured classification with probabilistic cat-egorial grammarsrdquo in Proceedings of the 21st Conference onUncertainty in Artificial Intelligence pp 658ndash666 EdinburghUK 2005

[23] L S Zettlemoyer and M Collins ldquoOnline learning of relaxedCCG grammars for parsing to logical formrdquo in Proceedings ofthe EMNLP-CoNLL pp 678ndash687 Prague Czech RepublicJune 2007

[24] P Liang M I Jordan and D Klein ldquoLearning dependency-based compositional semanticsrdquo in Proceedings of the 49thAnnual Meeting of the Association for Computational Lin-guistics Human Language Technologies pp 590ndash599 Port-land OR USA June 2011

[25] H Poon ldquoGrounded unsupervised semantic parsingrdquo inProceedings of the 51st Annual Meeting of the Association forComputational Linguistics pp 933ndash943 Sofia BulgariaAugust 2013

[26] L Tan D Yuan G Krishna and Y Zhou ldquolowasticommentrdquoACM SIGOPS Operating Systems Review vol 41 no 6pp 145ndash158 2007

[27] R J Kate and R J Mooney ldquoUsing string-Kernels for learningsemantic parsersrdquo in Proceedings of the 21st InternationalConference on Computational Linguistics pp 913ndash920 Syd-ney Australia July 2006

[28] W Xiong Z Lu B Li B Hang and Z Wu ldquoAutomatingsmart recommendation from natural language API descrip-tions via representation learningrdquo Future Generation Com-puter Systems vol 87 pp 382ndash391 2018

[29] W Xiong Z Wu B Li Q Gu L Yuan and B Hang ldquoIn-ferring service recommendation from natural language APIdescriptionsrdquo in Proceedings of the IEEE International Con-ference on Web Services (ICWS) pp 316ndash323 San FranciscoCA USA June 2016


of description documents including 10050 web-based ser-vices and 7155 mashups Our experiments demonstrate thatour approach can rank mashup services efficiently and itsperformance is better than semantic-based mashup serviceranking approaches alone

-is paper makes the following main contributions

(1) We propose a mashup service recommendationapproach via merging semantic features from APIdescriptions and structural features from themashup-API network

(2) We conduct comprehensive experiments on a real-world dataset demonstrating the effectiveness of ourapproach

-e remainder of this paper is organized as followsSection 2 gives a motivating scenario and presents ourapproach Section 3 describes the experiments in detailSection 4 discusses the related works and Section 5 con-cludes the paper with future work directions

2 Automating Service Recommendation viaSemantic Features and Structural Features

In this section we first present a scenario to illustrate themotivation of our work in Section 21 -en we discuss theissues ofmashup service recommendation in Section 22 nextwe propose a mashup service recommendation approach viasemantic and structural features in Sections 23ndash28

21 Motivation Under Internet scenario more structuresare presented between data objects A typical scenario is aknowledge graph that consists of a global relationship di-agram of user behavior data and an item with more attri-butes Structural and semantic similarities characterizedifferent aspects of the service similarity Indeed twosimilarities are orthogonal which motivate our work Weexpected that better service recommendation approachesmight be proposed by using structural features In particularwe propose a simple but effective approach AMSRSSF torank mashup services by using semantic and structural fea-tures First it applies a two-mode graph to describe mashupsweb APIs and their relations formally Second we quantifythe structural similarity between every pair ofmashup servicesbased on the two-mode graph-e structural similarity resultsfrom the structural context in which web APIs are used bymashup services Finally we introduce a merging embeddingvectors algorithm that only considers the pairwise similaritiesbetween mashup services to rank them effectively

According to the above it is certainly valuable to in-troduce structural features and then improve the accuracy ofrecommendation based on structural semantics

Finally there are some other issues which need to beaddressed

(1) How to extract semantic from natural language APIdescriptions

(2) How to generate structural semantics of the mashup-mashup network

(3) How to merge multiple embedding vectors for betteraccuracy

(4) How do we design experiments for performanceevaluation

22 Problem Description Our work considers the task ofrecommendation via representation learning as followsgiven a requirement q described in multidimensional in-formation which includes semantic and structural featuresthe corresponding recommended services are given viasimilarity calculation which exists in terms of the repre-sentation learning model

Our model consists in learning a function S(middot) which canscore requirement-service pairs (q t) Hence finding thetop-ranked answer t(q) to a requirement q is directly carriedout by

1113954t(q) argmaxtprimeisinK

S q tprime( 1113857 (1)

and to handle multiple recommended services wepresent the results as a ranked list rather than taking the topservice

23 Overview of Our Framework In this section we willpresent an overview of our approach As shown in Figure 1our approach mainly consists of four components semanticextraction based on NLP pipeline structural embeddinggeneration of the mashup-mashup network merging ofmultiple embeddings and recommendation based on fusionembedding More specifically the descriptions of web-basedservices will be crawled from ProgrammableWeb

We firstly preprocess natural language API descriptionsusing NLP pipeline and extract text semantic embeddingWe then generate structural embedding of the mashup-mashup network next we merge semantic embedding andstructural embedding Finally the developers give multi-dimensional information of requirements which will beparsed into semantic and structural features and passed tosimilarity calculation for ranking (Figure 2)

24 Semantic Extraction Based on the NLP Pipeline -issection preprocesses the sentences in API descriptions usingNLP pipeline named as NLTK [9] which resembles thefollowing high-level flow in Figure 3

For purposes of illustration we introduce a sample asfollows

ldquo-e Twitter microblogging service includes twoRESTful APIs -e Twitter REST API methods allowdevelopers to access core Twitter datardquo

We have the following steps

241 EOS (End of Sentence) Detection Breaking the textsinto paragraphs might be important However it may beunlikely to help EOS detection whichmarks the boundary ofa sentence from texts and breaks them into sentences We



NLP

hellip

H

Output matrix

Classifier



ProgrammableWeb





Facility



1

2

API-mashup network

Blog on a map




Fusion embedding





extraction



Facility



1

2

API-mashup network

Blog on a map


Blog on a map





































I 1 2 Nm1113864 1113865(2)






MMN N E W (3)






P vi vj

111386811138681113868111386811138681113874 1113875

(1 minus d) + dwij


⎛⎝ ⎞⎠

0 eij notin ε

⎧⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎩

(4)









A

B

D E

CF

A B E F

B E C B A

C B A B E

D A B E C

E C B A

A B E C B



λpq(t x)

1pr

if dtx 0

1 if dtx 1

1pio

if dtx 2

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

(6)





rt

σ Wrxt

+ Urhtminus 1

1113872 1113873

zt

σ Wzxt

+ Uzhtminus 1

1113872 1113873

ht tan h Wdx

t+ U r

t⊙htminus 11113872 11138731113872 1113873

ht

1 minus zt

1113872 1113873⊙htminus 1+ z

t⊙ht

(7)


rt

σ Wdr x

tminus 1+ U

dr h

tminus 1+ Crhi1113872 1113873

zt

σ Wdzx

tminus 1+ U

dzh

tminus 1+ Czhi1113872 1113873

ht tan h Wdx

tminus 1+ U

dr

t⊙htminus 11113872 1113873 + Chi1113872 1113873

hti+1 1 minus z

t1113872 1113873⊙htminus 1

+ zt⊙ht

(8)



X1X2

V

t

X3

λ = 1 λ = 1pio

λ = 1pioλ = 1pr


S1S2

S3

S4

S5

S6

S7

S8

S9

u







11113874 1113875 (9)


3 Experiments






Classifier

Output matrix


Dense embeddings


H

α0 α1





precision TP

TP + FP

recall TP

TP + FN


(10)



MAP 1

QR

11138681113868111386811138681113868111386811138681113868

1113944qisinQR

AP(q) (11)



NDCGk DCGk

IDCGk

(12)




i2

relilog2 i

(13)
















36 Results















Data Availability









Acknowledgments


References































NLP

hellip

H

Output matrix

Classifier



ProgrammableWeb





Facility



1

2

API-mashup network

Blog on a map




Fusion embedding





extraction



Facility



1

2

API-mashup network

Blog on a map


Blog on a map





































I 1 2 Nm1113864 1113865(2)






MMN N E W (3)






P vi vj

111386811138681113868111386811138681113874 1113875

(1 minus d) + dwij


⎛⎝ ⎞⎠

0 eij notin ε

⎧⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎩

(4)









A

B

D E

CF

A B E F

B E C B A

C B A B E

D A B E C

E C B A

A B E C B



λpq(t x)

1pr

if dtx 0

1 if dtx 1

1pio

if dtx 2

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

(6)





rt

σ Wrxt

+ Urhtminus 1

1113872 1113873

zt

σ Wzxt

+ Uzhtminus 1

1113872 1113873

ht tan h Wdx

t+ U r

t⊙htminus 11113872 11138731113872 1113873

ht

1 minus zt

1113872 1113873⊙htminus 1+ z

t⊙ht

(7)


rt

σ Wdr x

tminus 1+ U

dr h

tminus 1+ Crhi1113872 1113873

zt

σ Wdzx

tminus 1+ U

dzh

tminus 1+ Czhi1113872 1113873

ht tan h Wdx

tminus 1+ U

dr

t⊙htminus 11113872 1113873 + Chi1113872 1113873

hti+1 1 minus z

t1113872 1113873⊙htminus 1

+ zt⊙ht

(8)



X1X2

V

t

X3

λ = 1 λ = 1pio

λ = 1pioλ = 1pr


S1S2

S3

S4

S5

S6

S7

S8

S9

u







11113874 1113875 (9)


3 Experiments






Classifier

Output matrix


Dense embeddings


H

α0 α1





precision TP

TP + FP

recall TP

TP + FN


(10)



MAP 1

QR

11138681113868111386811138681113868111386811138681113868

1113944qisinQR

AP(q) (11)



NDCGk DCGk

IDCGk

(12)




i2

relilog2 i

(13)
















36 Results















Data Availability









Acknowledgments


References






























































I 1 2 Nm1113864 1113865(2)






MMN N E W (3)






P vi vj

111386811138681113868111386811138681113874 1113875

(1 minus d) + dwij


⎛⎝ ⎞⎠

0 eij notin ε

⎧⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎩

(4)









A

B

D E

CF

A B E F

B E C B A

C B A B E

D A B E C

E C B A

A B E C B



λpq(t x)

1pr

if dtx 0

1 if dtx 1

1pio

if dtx 2

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

(6)





rt

σ Wrxt

+ Urhtminus 1

1113872 1113873

zt

σ Wzxt

+ Uzhtminus 1

1113872 1113873

ht tan h Wdx

t+ U r

t⊙htminus 11113872 11138731113872 1113873

ht

1 minus zt

1113872 1113873⊙htminus 1+ z

t⊙ht

(7)


rt

σ Wdr x

tminus 1+ U

dr h

tminus 1+ Crhi1113872 1113873

zt

σ Wdzx

tminus 1+ U

dzh

tminus 1+ Czhi1113872 1113873

ht tan h Wdx

tminus 1+ U

dr

t⊙htminus 11113872 1113873 + Chi1113872 1113873

hti+1 1 minus z

t1113872 1113873⊙htminus 1

+ zt⊙ht

(8)



X1X2

V

t

X3

λ = 1 λ = 1pio

λ = 1pioλ = 1pr


S1S2

S3

S4

S5

S6

S7

S8

S9

u







11113874 1113875 (9)


3 Experiments






Classifier

Output matrix


Dense embeddings


H

α0 α1





precision TP

TP + FP

recall TP

TP + FN


(10)



MAP 1

QR

11138681113868111386811138681113868111386811138681113868

1113944qisinQR

AP(q) (11)



NDCGk DCGk

IDCGk

(12)




i2

relilog2 i

(13)
















36 Results















Data Availability









Acknowledgments


References






























λpq(t x)

1pr

if dtx 0

1 if dtx 1

1pio

if dtx 2

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

(6)





rt

σ Wrxt

+ Urhtminus 1

1113872 1113873

zt

σ Wzxt

+ Uzhtminus 1

1113872 1113873

ht tan h Wdx

t+ U r

t⊙htminus 11113872 11138731113872 1113873

ht

1 minus zt

1113872 1113873⊙htminus 1+ z

t⊙ht

(7)


rt

σ Wdr x

tminus 1+ U

dr h

tminus 1+ Crhi1113872 1113873

zt

σ Wdzx

tminus 1+ U

dzh

tminus 1+ Czhi1113872 1113873

ht tan h Wdx

tminus 1+ U

dr

t⊙htminus 11113872 1113873 + Chi1113872 1113873

hti+1 1 minus z

t1113872 1113873⊙htminus 1

+ zt⊙ht

(8)



X1X2

V

t

X3

λ = 1 λ = 1pio

λ = 1pioλ = 1pr


S1S2

S3

S4

S5

S6

S7

S8

S9

u







11113874 1113875 (9)


3 Experiments






Classifier

Output matrix


Dense embeddings


H

α0 α1





precision TP

TP + FP

recall TP

TP + FN


(10)



MAP 1

QR

11138681113868111386811138681113868111386811138681113868

1113944qisinQR

AP(q) (11)



NDCGk DCGk

IDCGk

(12)




i2

relilog2 i

(13)
















36 Results















Data Availability









Acknowledgments


References


































11113874 1113875 (9)


3 Experiments






Classifier

Output matrix


Dense embeddings


H

α0 α1





precision TP

TP + FP

recall TP

TP + FN


(10)



MAP 1

QR

11138681113868111386811138681113868111386811138681113868

1113944qisinQR

AP(q) (11)



NDCGk DCGk

IDCGk

(12)




i2

relilog2 i

(13)
















36 Results















Data Availability









Acknowledgments


References






























precision TP

TP + FP

recall TP

TP + FN


(10)



MAP 1

QR

11138681113868111386811138681113868111386811138681113868

1113944qisinQR

AP(q) (11)



NDCGk DCGk

IDCGk

(12)




i2

relilog2 i

(13)
















36 Results















Data Availability









Acknowledgments


References






























36 Results















Data Availability









Acknowledgments


References






























Acknowledgments


References






























automating mashup service recommendation via semantic and...

Documents