Call for Task Participation NTCIR-12 Pilot Task: Short Text Conversation (STC) Lifeng Shang, Zhengdong Lu, Hang Li (Huawei Noah’s Ark Lab, Hong Kong) Tetsuya Sakai (Waseda University, Japan) http://ntcir12.noahlab.com.hk/stc.htm Twitter: @ntcirstc February 27, 2015@NTCIR-12 Kickoff
Microblogs: Twitter, Weibo... Over 40 million users
What is STC? (1) POST: “Dr. Hang Li’s Learning to Rank for IR and NLP second edition released! Follow!”
What is STC? (2) POST: “Dr. Hang Li’s Learning to Rank for IR and NLP second edition released! Follow!” COMMENT by Hang Li: “Thanks ZhiYuan! I’ve added detailed explanations of the LambdaMART algorithms etc. “ Coherent AND useful
What is STC? (3) POST: “How’s the hair?”
What is STC? (4) POST: “How’s the hair?” COMMENT by Tetsuya Sakai: “I don’t have any.” Coherent but NOT useful
Objectives LOL • The ultimate objective Build an open-domain system that can interact naturally with humans • The objective for NTCIR-12/13 STC Build an IR system that effectively reuses past comments to respond to a post. Coherence: the post-comment pair makes sense as a consecutive short text exchange between two people. Usefulness: the comment contains information or an opinion that might be useful to the author of the post.
Given a new post, can a coherent and STC research questions useful comment be returned by searching a post-comment repository? What are the challenges and limitations of this IR-based STC approach? [Ji14] post STC Search and reuse post-comment repository post comment post comment post comment post comment comment comment comment comment
We also provide English machine Data and language scope translations of the Chinese posts and comments. Number of test posts will be determined using topic set size design [Sakai14CIKM,Sakai14EVIA]
Task design and evaluation measures • Ad hoc IR design: given a “new” post, retrieve coherent and useful comments from repository. • Pooling and graded relevance assessments L2: coherent and useful L1: coherent but not useful L0: not coherent (and therefore not useful either) • Evaluation measures (basically one good comment is enough): G@1 (normalised gain at rank 1) ERR (expected reciprocal rank) P+ (Similar to Q-measure, suitable for navigational intents) [Sakai14PROMISE]
Plans for STC-2@NTCIR-13 • Follow the INTENT-2 “revived run” model [Sakai13INTENT] • STC-1 participants will keep their systems in the fridge • When they come back at STC-2, they use both STC-1 and STC-2 systems to handle the STC-2 posts • Compare STC-1 and STC-2 systems on the STC-2 test collection STC-1 new posts STC-2 new posts STC-1 runs STC-2 new runs STC-1 systems Revived runs STC-2 systems
Schedule We will give you training data as soon as you register! Sooner the better! Feb 27, 2015 NTCIR-12 kickoff Oct 31, 2015 NTCIR-12 task registration deadline Nov 2, 2015 STC test topics released Nov 30, 2015 STC run submission deadline Dec 2015-Jan 2016 STC relevance assessments + evaluation Feb 1, 2015 STC results sent to participants + STC draft overview released Mar 1, 2015 NTCIR-12 participants’ draft papers due / Task organisers’ feedback May 1, 2015 NTCIR-12 all camera ready papers due Jun 7-10, 2015 NTCIR-12 conference
Prospective participants and budget Huawei will cover the relevance assessment cost. No seeding funding from NTCIR required.
Related tasks • TREC Microblog (2011-) [Lin13] Data: twitter, NOT distributed to participants Tweets2011: only IDs distributed, data downloaded individually Tweets2013: Evaluation as a Service (access through APIs) Ad hoc search etc. Evaluation based on binary relevance • NTCIR Community Question Answering (2010) [Ishikawa10] Data: Japanese Yahoo! Answers (Chiebukuro) Given a Q and its responses, rank the responses (which is the best answer?). Evaluation using G@1 etc.
References [Ishikawa10] Ishikawa, D., Sakai, T. and Kando, N.: Overview of the NTCIR-8 Community QA Pilot Task (Part I): The Test Collection and the Task, Proceedings of NTCIR-8, pp.421-432, 2010. [Ji14] Ji, Z., Lu, Z. and Li, H.: An Information Retrieval Approach to Short Text Conversation, 2014. http://arxiv.org/abs/1408.6988 [Lin13] Lin, J and Efron, M.: Overview of the TREC-2013 Microblog Track, Proceedings of TREC 2013, 2013. [Sakai13INTENT] Sakai, T. et al.: Overview of the NTCIR-10 INTENT-2 Task, Proceedings of NTCIR-10, pp.94-123, 2013. [Sakai14CIKM] Sakai, T.: Designing Test Collections for Comparing Many Systems, Proceedings of ACM CIKM 2014, pp.61-70, 2014. [Sakai14EVIA] Sakai, T.: Topic Set Size Design with Variance Estimates from Two-Way ANOVA, Proceedings of EVIA 2014, pp.1-8, 2014. http://www.f.waseda.jp/tetsuya/CIKM2014/ir0030-sakai.pdf [Sakai14PROMISE] Sakai, T.: Metrics, Statistics, Tests, PROMISE Winter School 2013: Bridging between Information Retrieval and Databases (LNCS 8173), 2014. http://research.microsoft.com/en-us/people/tesakai/metrics.pdf