seabet app download title：seabet app download Quality and Learning with Crowdsourcing
seabet app download time: July 10, 2019 (Wednesday) 3:00 pm
Reporting location：seabet app download Building 313
Reporter: Sheng Shengli, associate professor at the seabet app download of Central Arkansas, USA，Director of Data Analysis Laboratory
Abstract
Crowdsourcing systems provide convenient platforms to collect human intelligence for a variety of tasks (e.g., labeling objects) from a vast pool of independent workers (a crowd). Compared with traditional expert labeling methods, crowdsourcing is obviously more efficient and cost-effective, but the quality of a single labeler cannot be guaranteed. In taking advantage of the low cost of crowdsourcing, it is common to obtain multiple labels per object (i.e., repeated labeling) from the crowd. In this talk, I outline our seabet app download on crowdsourcing from three aspects: (1) crowdsourcing mechanisms, specifically on repeated labeling strategies; (2) ground truth inference, specifically on noise correction after inference and biased wisdom of the crowd; and (3) learning from crowdsourced data.
I first present repeated-labeling strategies of increasing complexity to obtain multiple labels. Repeatedly labeling a carefully chosen set of points is generally preferable. A robust technique that combines different notions of uncertainty to select data points for more labels is recommended. Recent seabet app download on crowdsourcing focuses on deriving an integrated label from multiple noisy labels via expectation-maximization based (EM-based) ground truth inference. I present a novel framework that introduces noise correction techniques to further improve the label quality of the integrated labels obtained after ground truth inference. I further show that biased labeling is a systematic tendency. State-of-the-art ground truth inference algorithms cannot handle the biased labeling issue very well. Our simple consensus algorithm performs much better. Finally, I present pairwise solutions for maximizing the utility of multiple noisy labels for learning. Pairwise solutions can completely avoid the potential bias introduced in ground truth inference. They have both sides (potential correct and incorrect/noisy information) considered, so that they have very good performance whenever there are a few or many labels available.
Bio
Victor S. Sheng received the M.Sc. degree from the seabet app download of New Brunswick, Fredericton, NB, Canada, and the Ph.D. degree from the seabet app download of Western Ontario, London, ON, Canada, both in computer science, in 2003 and 2007, respectively.
He is an Associate Professor of computer science and the Founding Director of Data Analytics Laboratory at seabet app download of Central Arkansas. After receiving the Ph.D. degree, he was an Associate Research Scientist and NSERC Postdoctoral Fellow in information systems with the Stern Business School at New York seabet app download. His research interests include data mining, machine learning, crowdsourcing, and related applications in business, industry, medical informatics, and software engineering. He has published more than 140 research papers in conferences and journals of machine learning and data mining. Most papers are published in top journals and conferences in data science, such as PAMI, TNNNLS, TKDE, JMLR, AAAI, KDD, IJCAI, and ACMMM.
Prof. Sheng is a senior member of IEEE. He is a conference organizer for several conferences, and an editorial board member for several journals. He also is a SPC and PC member for many international conferences (such as IJCAI, AAAI, and KDD) and a reviewer of more than twenty international journals (such as PAMI, TNNNLS, TKDE, and JMLR). He was the recipient of the Best Paper Award Runner Up seabet app download KDD’08, the Best Paper Award seabet app download ICDM’11, the Best Student Paper Award Finalist seabet app download WISE’15, and the Best Paper Award seabet app download ICCCS’18.
About the speaker:
VICTOR S. SHENG) is an associate professor and director of the Data Analysis Laboratory in the Department of Computer Science at the University of Central Arkansas。Research fields are data mining and machine learning、Artificial Intelligence、Data Security and Decision Support。Published more than 100 papers in international seabet app download conferences and journals，A single paper has been cited up to more than 680 times。Research results have been published in top conferences and journals on data mining and machine learning, International seabet app download journals including TPAMI, TKDE, JMLR, TMM, TNNLS and DMKD, etc.。International seabet app download conferences including IJCAI, KDD, ICML, AAAI, ECML, ICDM, DASFAA, ACM MM, ICMR, ICME, CIKM etc.。Won the Best Poster Award of IEEE Kitchener-Waterloo Section Joint Workshop on Knowledge and Data Mining in 2006；Won the runner-up for the Best Paper Award at the KDD Conference in 2008；2008 Machine Learning Workshop Google Student Award Winner；Won the Best Paper Award at ICDM Conference in 2011；Won the WISE Best Student Paper Award finalist in 2015。Current financial chair of ICDM 2017 and editorial board member of multiple international journals。Host and participates in the National Natural Science Foundation of America、More than 10 Canadian Natural Sciences and Engineering Research Funds。Served as a member of the National Science Foundation review committee and chairman of the branch of international seabet app download conferences for many times。
