Lydia/TextMap Publications

Websites presenting Lydia analysis include:

• TextMap identifies temporal and geographic trends in entities such as people, places, and things by analyzing roughly 500 daily English language newspapers.
• TextMed identifies relationships between medical and biological entities through analysis of PubMed/Medline abstracts.
• TextBlog identifies trends and entity relationships through large-scale analysis of blogs.
• TextBiz uses random-walk models to generate a probability distribution on the future prices for all NASDAQ, NYSE, and AMEX stocks. We soon should add analysis of business news sources.

See our October 2009 overview presentation on the Lydia project. Research papers describing our work are:

• Access: News and Blog Analysis for the Social Sciences (with M. Bautin, C. Ward, and A. Patil) 19th Int. World Wide Web Conference (WWW 2010), Raleigh NC, April 26-30, 2010. Access Access
• Trading Strategies to Exploit Blog and News Sentiment (with W. Zhang). Fourth Int. Conf. on Weblogs and Social Media (ICWSM 2010), Washington DC, May 23-26, 2010.
• The Wisdom of Bookies? Sentiment Analysis vs. the NFL Point Spread (with Y. Hong). Fourth Int. Conf. on Weblogs and Social Media (ICWSM 2010), Washington DC, May 23-26, 2010.
• A Dynamic Visual Interface for News Stream Analysis (with W. Cui, H. Zhou, H. Qu, and W. Zhang). First International Workshop on Intelligent Visual Interfaces for Text Analysis, (IUI 2010), February 7, 2010. Hong Kong, China.
• Expanding Network Communities from Representative Examples (with A. Mehler). ACM Trans. Knowledge Discovery from Data (TKDD), Special Issue on Social Computing, Behavioral Modeling, and Prediction.
• Name-Ethnicity Classification from Open Sources (with A. Ambekar, C. Ward, J. Mohammed, and S. Male) 15th ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD 2009), Paris France, June 28-July 1, 2009.
• Improving Movie Gross Prediction Through News Analysis (with W. Zhang) IEEE/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology (WI 2009), Milan Italy, September 15-18, 2009.
• Identifying Differences in News Coverage Between Cultural/Ethnic Groups (with C. Ward and M. Bautin) News Analysis Workshop of IEEE/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology (WI 2009), Milan Italy, September 15-18, 2009.
• International Sentiment Analysis for News and Blogs (with M. Bautin and L. Vijayarenu). Second Int. Conf. on Weblogs and Social Media (ICWSM 2008), Seattle WA, March 26-28, 2008.
• Large-Scale Sentiment Analysis for News and Blogs (with N. Godbole and M. Srinivasaiah). Int. Conf. on Weblogs and Social Media (ICWSM 2007), Denver CO, March 26-28, 2007. Also see our system demonstration description.
• Concordance-Based Entity-Oriented Search (with M. Bautin) IEEE/ACM Web Intelligence (WI-07), Silicon Valley CA, November 2-5, 2007. Full version to appear in Web Intelligence and Agent Systems: An International Journal
• Newspapers vs. Blogs: Who Gets the Scoop? by L. Lloyd, P. Kaulgud, and S. Skiena, AAAI Symp. Computational Approaches to Analysing Weblogs (AAAI-CAAW 2006), Stanford University, March 27-29, 2006 provides an comparison of entity frequencies between blogs and more formal news sources.
• Identifying co-referential Names Across Large Corpora by L. Lloyd, A. Mehler, and S. Skiena, Proc. Combinatorial Pattern Matching (CPM 2006) discusses our method for identifying synonym sets of entities.
• Spatial Analysis of News Sources, by A. Mehler, Y. Bao, X. Li, Y. Wang, and S. Skiena, IEEE Trans. Visualization and Computer Graphics 12 (2006) 765-772 discusses our ``heatmap'' analysis.
• Lydia: A System for Large-Scale News Analysis by L. Lloyd, D. Kechagias, and S. Skiena, 12th Symp. of String Processing and Information Retrieval, (SPIRE '05), Lecture Notes in Computer Science, 3772 (2005) 161-166 provides an overview of the architecture of the Lydia system as of May 2005.
• Question Answering with Lydia by J. Kil, L. Lloyd, and S. Skiena, 14th Text REtrieval Conference (TREC 2005), NIST Gaithersburg MD, November 15-18, 2005 describes an extension to Lydia for answering factoid, list, and open-ended English-language questions.

Publications employing Lydia/GS analysis in other disciplines include:

• J. Sides and L. Vavreck, The Gamble: Choice and Chance in the 2012 Presidential Election Princeton University Press, 2013.
• S. Goldman and D. Mutz, The Obama Effect: How the 2008 Campaign Changed White Racial Attitudes K. Kenski, B. Hardy, and K. Jamieson, The Obama Victory: How Media, Money, and Message Shaped the 2008 Election, Oxford University Press, New York, 2010.
• E. Key, L. Huddy, M. Lebo, and S. Skiena. Large Scale Online Text Analysis Using Lydia. Paper presented at the annual meeting of the American Political Science Association, 2010, Washington, DC.
• J. Sides and L. Vavreck, The Gamble: Choice and Chance in the 2012 Presidential Election Princeton University Press, 2013.
• D. Fan, Tweets are not public opinion but can be used to predict public opinion. Midwest Association of Public Opinion Research annual meeting, Chicago, Nov. 16-17, 2012.
• S. Best and D. Fan, The Campaign Mattered... A Little Huffington Post. November 5, 2012.
• V. Pool, N. Stoffman, and S. Yonker, The People in Your Neighborhood: Social Interactions and Mutual Fund Portfolios The Journal of Finance, 2015. (Used our name ethnicity detector for analysis)

Related news/sentiment analysis systems include:

• The GDELT Project
• Opinion Crawl
• Media Cloud
• Student documentation