How Analytics can be Used to Fact Check News Items

Companies have been focusing on the capture of big data using tools such as Hadoop for a few years now. However, it appears that many of them still do not identify big data as a strategic resource worthy of enterprise wide analysis. As a result, while there is much data available, it is being used in scattered ways by various departments operating in silos.

Some of the major analytical tools - Cloudera, Hortonworks and Map R have responded with high level and sophisticated offerings using artificial intelligence and machine learning techniques. They feel this will make the tools easier to use and more powerful.

A good example come from Cloudera, in the form of their application Reuters Tracer. This remarkable tool is used by Reuters News Agency for analyzing twitter feeds, fact checking them and producing results in a matter of milliseconds.

They recognize that Twitter has become a major news source but that it is difficult for people to sort through all the "noise" and determine what is true and what isn't. Reuters Tracer "processes about 13 million tweets daily, capturing events as they happen and determining: is an event true, is it newsworthy, and what is the scope and impact of that event."

"If a tweet is an opinion, Reuters Tracer can determine whether it comes from a recognized expert, and is therefore of news value. In delivering its results, it provides journalists and businesses with a 'newsworthiness score' for each event that rates its assessed level of accuracy and credibility." These results can be produced in as little as 40 milliseconds - basically real time.

Fact checking in the old sense has become something of an industry Reuters Trace can automate, if not  all, much of the fact checking needed to provide a sound basis for decision making.

Many companies are considering applications like this for their own business purposes, perhaps with a big data source like customer or supplier activity with analytics embedded to focus on key strategic decisions. Auditors can definitely use tools like this to improve the analytical techniques used in audits - something that is badly needed.

For more, check out these references - A recent article on the subject and the Cloudera website.