Wednesday, December 21, 2016

ESMA Chooses iXBRL as Single Electronic Format



Earlier today, the European Securities and Markets Authority (ESMA) announced that starting in 2020, all publicly listed companies in the EU that report consolidated IFRS financial statements must provide them in Inline XBRL. The decision was made primarily because the use of Inline XBRL enables a single document to be both machine-readable and human-readable. 
Steven Maijoor, ESMA Chair, said “Financial reports are crucial for a full understanding of a company’s situation and moving to electronic reporting will facilitate analysis, comparability and accessibility of issuers’ financial statements."
"We believe that the vast pool of free, structured financial information that will emerge once electronic reporting starts will trigger technological innovation. The Inline XBRL format has the potential to bring financial reporting into the digital age.”

This is an important announcement for the community, and XBRL International strongly supports this move towards improved business reporting across Europe. We'll have much more on this in tomorrow's newsletter. 

Read the ESMA press release and the feedback statement detailing responses to the consultation paper. 

Monday, December 19, 2016

Data Analytics Disappears

It has sometimes been said that you know a technology has succeeded when it disappears. That means it has been embedded in regular day-to-day applications such that it is no longer visible to the user even though it is being used. A good example is HTML, the mark-up language used to create websites. At one time, people learned to code in HTML in order to create websites. Now that is not done any more because user friendly tools and templates exist that enable the users to create their website more effectively than ever without even seeing the HTML code, although it is still the backbone of most sites.

Analytics is being embedded in a large number of applications in just this way. People will be using data analytics every day, but will not need to see it or learn it as a separate application. Instead they will use it as part of their regular activities.

One good example is Salesforce, which now incorporates Tableau such that the power of Tableau fueled analytics can be used as part of sales force analysis. Numerous other examples are coming out as analytics becomes a part of SaaS services feeding into many applications.

All of this shows just how fast data analytics is evolving and how quickly it has become mainstream. For more on this development, click this link.

Tuesday, December 13, 2016

Information increases the complexity of life

Pew Research has released a major study exploring the effects of information overload on peoples' lives. The findings indicate that more people felt that information complicates their lives than felt that they feel overloaded by information. This finding held despite gender, age, ethnicity, income level, educational attainment, etc. The study also found that digital tools help in dealing with information overload. This was supported by 81% of the respondents. On the positive side, 79% felt that information gave them more control over their lives.

There is, however, an indication that there is widespread awareness that some of the information is not reliable. Nevertheless most people (80%) felt that they are able to quickly determine if it is unreliable. It was not clear, however, what criteria those people used to determine whether information is reliable. A study of how people determine this could be quite revealing. We know that false news, unchecked information and unfounded opinion proliferate on the internet. We are not sure what effect this is having on the perceptions and actions of the readers. For a summary of the study, check the Pew website here.

Tuesday, December 06, 2016

Securing the Internet of Things

The US Department of Homeland Security (DHS) recently issued a strategy for securing the Internet of Things. This was in response to s string of attacks that demonstrate the vulnerability of internet connected "things" to hackers and the breach of information security and privacy.

The strategy is an important document and sets out a number of principles to be followed.
The first one, that security be included at the design stage, puts manufacturers on notice that the products they make must be more secure before they are sold. For a copy of the announcements and the Strategy, check out the DHS website.

Friday, December 02, 2016

Free Software an Entry Point into Data Analytics

Would you or your organization like to venture into data analytics but can't afford the time and expense of formal training? A useful approach might be to start using free data analytics software. There are some available that have good functionality and can be used for basic data analytics. Of course, it still takes some time, but it may be that the process will yield results more quickly.

Some of the free products that may be useful include Microstrategy 10 by Microstrategy, H2O.ai, KNIME, and Google's Tensorflow. It may be hard to choose one, but you never know until you try.

Friday, November 25, 2016

The Position of the CDO in the Organization

Many companies over the past few years have developed the position of Chief Data Officer (CDO), often placing that position in the C-suite. However, exact organizations vary, as one would expect, since organizations vary in their need for data management.

A common aspect of data management in modern business, though, is that data, including big data, along with the infrastructure needed to handle and analyze and, most importantly, put it to good strategic use, is a growing and even critical phenomenon.

Nevertheless there has been debate about whether CDO's should even exist, whether they should report to the CIO, CFO, or the CEO. The roles of the CIO (chief information officer) and CDO have been evolving, particularly in terms of their relationship to the strategic aspects of the business.

It has been obvious that technology has pervaded all aspects of business over recent years. That being the case, the activities of the CIO directly affect all strategic and operational areas of the business. Accordingly some activities that at one time were traditionally CIO activities have been spun off into the functional areas, with the CIO acting as coordinator. At the same time, the CIO has assumed more of a strategic role, working with various functional areas, such as marketing and finance.

Along comes the CDO, with a knowledge of data management, data sources, data infrastructure needs and the importance of data to particular functional areas. Clearly the CDO needs to operate at a strategic level in cooperation with the affected functional areas. One challenge, therefore, is to avoid duplication of effort.

To some extent duplication is not an issue, since the CDO is focused on the data more than the infrastructure and the CIO is more focused on the infrastructure rather than the data. There is a role for both, therefore, at the C-suite table, but their roles need to be clearly defined. And they need to work together.

To have the CDO report to the CIO will likely take away some of the benefits to the organization of advanced data management, so both should report to the CEO.

According to a recent report by the Gartner Group, this is the dominant trend.


Thursday, November 17, 2016

An Era of Machine Intelligence

Gartner Group recently released their annual list of the top ten strategic trends in business. These trends focus on new technologies and how they are being used. They tell us a lot about how technology is shaping our world.

One of the major underlying themes in the list they produced this year is that of intelligence. It is clear that your technology is going to be much more intelligent in future and be able to interact with users on a much more intelligent basis.

By intelligent, they mean an ability to receive input and draw conclusions from it, make decisions and recommend courses of action.

Artificial intelligence plays a big role in this trend and is already being embraced by many companies. So we will see smarter devices in the world of the Internet of Things, as well as smart apps that can make decisions on the basis of data and other input. With the growing usage of oral input to computers, where it is predicted that in the relatively near term, most interaction with computers will be oral, combined with the greater intelligence of computers, we will see the prospect of being able to discuss problems with our computers and come to better decisions. Add in virtual reality and it is possible to foresee an ability to interact in such a way that all sensory perception is used to help in analyzing data, something that could be a big help in coping with big data.

As always, there are pros and cons of these developments. Technology has already replaced a lot of jobs but smarter computers will be able to replace a lot more. So far, most of the jobs replaced have been of the menial variety - repetitious and simple. However, as the intelligence of computers grows, more complex jobs ill be replaced, including some of the traditional professions, like accounting, architecture, many aspects of law and even medicine. But new professions are being created. We already see a strong demand for data analysts. And we are beginning to see a demand for machine trainers - devoted to providing direction to that intelligence.

The question one has to ask is - when will computers be given the right to vote (sic)??

Tuesday, November 08, 2016

Next Generation Technologies for Mobile Commerce

Mobile Commerce has been one of the fastest growing aspects of electronic commerce. It involves the use of smart phones, tablets and other mobile devices to conduct commercial transactions.

While mobile commerce has been growing, often smaller companies are unaware of the potential for enhanced marketing techniques or if aware, are fearful that such techniques will require too much of an investment in technical IT resources.

Enhanced marketing techniques depend on adopting mobile apps that enable some interaction with the customers, such as trying out products in some way, perhaps on a virtual basis, or enabling location based services to gain certain benefits.

The idea is that the mobile devices generate data that can be revealing about the actions and preferences of the customer.

Collecting such data is becoming easier with the development of new apps that involve some such interaction.  However, the issue that arises is how to make use of these data. That's where big data analytics, particularly predictive analytics.

The ability to implement predictive analytics on big data gathered from a wide variety of mobile devices is something that takes some preparation but is becoming increasingly important to the success of mobile e-commerce and therefore to the success of most companies selling products to the public.

Customers will expect marketing efforts directed their way to be customized to their needs and habits, and this is only possible by having the company adopt mobile apps that can collect big data and develop the ability to use analytics to generate those customized marketing techniques.

Frost and Sullivan recently released a useful research report titled "Next Generation Technologies for Mobile E-commerce" which explores these issues and provides some useful guidance. Some interesting summaries of this report can be found at Information Week and on the Frost and Sullivan website.

Mobile Commerce, big data analytics and customized marketing are coming together to create important changes in customer relations.

Thursday, November 03, 2016

Scotland to Start Stock Exchange Using Blockchain


Scotex, The Scottish Stock Exchange, is seeking to raise as much as 15 million pounds ($18.4 million) to start a new regulated equity market next year, according to a statement on Thursday. Executed trades will be processed by Blockchain, the distributed-ledger technology that drives bitcoin. For more on this, click here. And here.


Friday, October 14, 2016

Needing an Infrastructure for Big Data

Anytime there is need to handle data, there is a need for an underlying infrastructure. The data needs to be gathered, organized, stored, processed, and used. In the case of big data, a particular infrastructure is needed, and it often involves tools like Hadoop, Hive, Spark and visualization software. The basic elements are the same as any data, but the tools and capacity are different. Without a proper infrastructure, big data will lose some or all of it value.  Some more detailed thoughts can be found in this article.

Friday, September 30, 2016

DDD - The Move Away From Intuition

Data Driven Decision making is taking hold in large companies. Not all, but many. This according to a survey carried out by Harvard, Stanford and the US Census Bureau.

Companies have access to more data than at any time in history. They are learning to use it and finding that the quality of the decisions to be better than the largely intuitive decisions of the past. This applies in manufacturing, marketing and other areas. There is a growing preference in many organizations to make fact-based decisions rather than those based just on opinon. For a summary of this research, check this link.

Tuesday, September 27, 2016

The New/Old Role of Libraries

New research into the nature and role of libraries confirms that libraries are here to stay as a vital part of our communities and that they are changing forever. The days of libraries being built around rooms full of books are drawing to a close. The new libraries feature digital reading, classes in technology use, places to do research using computers and safe havens for study and contemplation.

A majority of survey respondents believe that libraries need to emphasize the use of technologies, like computers, tablets and smartphones. They include the use of 3-D printers among the skills that libraries should teach. They also believe that libraries can play a role of helping people to learn what information they can rely on, something increasingly difficult in this age of social media. Follow this link for the report on libraries 2016.


Monday, September 26, 2016

Free Predictive Analytics Software

Predictive Analytics is an increasingly useful tool for business to assess risks and opportunities in some organized fashion. It doesn't necessarily provide answers but can often offer useful insights. Free software can provide a inexpensive but potentially useful way to venture into the world of predictive analytics. Here is a summary of some of the more popular free software available, such as R, Orange, Rapidminder, Anaconda and others.

Wednesday, September 21, 2016

The Basics of Predictive Analytics

For those who are getting into predictive analytics, a useful article by Eric Seigel, founder of Predictive Analytics World, provides a number of definitions that are useful to understanding the basics of predictive analytics. Many applications of predictive analytics have emerged in recent years, including in auditing, finance, marketing and others. Some effort is required, such as reading , taking courses, etc, to gain a level of knowledge that enables useful applications. However, knowing the definitions of such terms as Predictive analytics itself, predictive models, uplift models, vast search and others is a starting point. The article can be found through this link.

Wednesday, September 14, 2016

How to Develop Your Data Analytics Skills

Data analytics is becoming important to established professionals in many fields. Others are viewing it as a new career opportunity. Indeed the demand for data analytics professionals is strong and growing. There are many opportunities to  develop or improve skills, including back to school, online courses, technical training, networking and the like. This recent article sets out a number of options and provides examples.

Friday, September 09, 2016

The Sharing, Collaborative and Gig Economy

Pew Research has released an interesting study on the new App-based Economy which reveals a number of splits in population usage of the new services. They focused primarily on ride-hailing, home sharing and crowdfunding apps. Issues explored involve jobs, regulation and the potential of a new digital divide. Uber, AirBNB and Kickstarter.

All are used by a low percentage of adults, and most have never heard of these apps. As one would expect, most users are relatively young - under 45 - and all of the services are evolving with new regulations.

The new digital economy is a permanent part of the landscape, and such research is fundamentally important for business and policy-makers.

See the report here.

Monday, August 15, 2016

Lack of Skilled Staff Inhibits Adoption of IoT

A recent report by Cisco and Capita states that IT leaders are concerned that they do not have enough skilled resources to carry out potential IoT adoptions. This is an issue in several important and new areas of IT Management. It points to a need to change the approach to IT Education and certification, perhaps to one of emphasizing cooperative training with industry rather than gaining university degrees.

University degrees take a long time and the people acquiring them have little or no experience. Cooperative programs with trade schools and community colleges are faster and produce people with important and useful on-the-job experience. Check out this short summary of the report.

Friday, August 12, 2016

IoT Connections to Exceed Mobile Phones

The Internet of Things has been growing exponentially. A recent report by Ericsson finds that by 2018, the number of IoT connections will exceed that of mobile phones.

This growth is generated by such phenomena as smart houses, transport logistics, medical applications, autonomous cars and remote manufacturing.

Many of these applications are highly sensitive and require a very high degree of reliability and stability. Therefore they are a focus of IT management and a drain on  resources. With a projected annual growth rate of 23%, such IoT devices are becoming ubiquitous.

For a rundown on the Ericsson report, check this link.

Friday, August 05, 2016

Growing Volumes of Data Create Management Issues

The world supply of data is growing rapidly. That's no secret. At present the global volume of data is 4.4 zettabytes. One Zettabyte equals one sextillion bytes or 10 to the 16th power or one trillion gigabytes. It's hard to imagine, but clearly that's a lot of data. Predictions by IDC in a recent study indicate that global data will increase from 4.4 to 44 zettabytes over the next four years. Business will bear the brunt of this growth in data and will need to manage it.

The cloud is crucial to managing the data. But there also needs to be an organized and efficient means of integrating the data with corporate systems, including ERP and other enterprise systems. An additional complication is that much of the ultimate output needs to be channeled to mobile devices, including smartphones. There is a growing realization that the mobile world in which we live is an ongoing reality and the primary means of consuming information for the foreseeable future.

While systems integration has been with us for many years, the integration process involved with the new data is a relatively new field. It must take place in the cloud first and that primarily involves organizing the data so it can be processed. Companies are using "data lakes" for the first stage and then transitioning to cloud based data warehouses and various applications that feed into mobile apps.

For an interesting overview of the processes, here's a good article on the subject.



Friday, July 29, 2016

The Sharing Economy Grows

In May, Pew Research released a report on its surveys about various key aspects of the sharing economy. They covered ride sharing, home sharing and crowdfunding.  The results are interesting. they show that a large proportion of adults have used ride sharing (like Uber) and find it a satisfactory (or better) experience. Most are young urban adults. Fewer use home sharing and there is a common concern expressed among users or potential users about safety. Even fewer have used crowdfunding, but its use is growing.  For the reports, follow this link.

Wednesday, July 20, 2016

How to Make your Company Data-driven

Optimum use of data in management is rapidly shifting from competitive advantage to competitive necessity. But implementing an effective data use policy can result in spending massive amounts of money with poor results if it is not done well.

Serious planning is a necessity. As with any planning activity, definition of objectives is crucial, including specific definition of strategic and operational business objectives, and identification of data that will meet these objectives.

There are massive amounts of data on the internet, but not all are readily available for consumption. This is a significant consideration. Also, if you are planing to use internet based data, don't even think about developing your own systems to handle it. It should be capable of being analyzed without downloading. There are numerous tools available in the cloud where online cloud-based analysis can be carried out.

Ensure the results of the analysis are placed in the right hands. This would have been planned in the objectives identification phase. But you need to make sure that the results are configured and visualized in ways that make good use of graphs and other visuals or pictorials. Often the recipients are busy and won't take the time to spend a additional time on the analysis unless there is a rapid gratification.

This leads to another point about presenting the results. It must be done with a minimum of disruption to the recipients normal routines.  Again, this approach will lead to better use of the analysis.

For additional thoughts on this, and a source for this posting, click this link.


Tuesday, July 19, 2016

Job Automation - The big Trend of the 21st Century

People who do physical jobs have been faced with job losses because of automation for several decades now. The trend has been accelerating with the automation of entire factories and the need of retrain millions of workers.

Now the automation trend is spreading into the professions, even though their jobs are often not physical. At risk are accountants, lawyers, architects, to name a few. The automation of these fields does not mean that all will lose their jobs, but it does mean that large parts of their jobs can be automated, leading to a lower demand for them. For example a recent major report for the Canadian Bar Association called "Futures: Transforming the Delivery of Legal Services in Canada that outlined a strategy to deal with transformative change in the delivery of legal services. One of the driving forces for the change was technological innovation.

Last year, CPA Canada released a report setting out Drivers of Change in the accounting profession, with a strong emphasis on technological change. Other writers have written about how mobile devices, social media and data analytics will change the profession.

Behind the rapid spread of automation into the professions is the growing power of artificial intelligence. This enables technology to replace or significantly modify jobs with more and more intellectual content.

The Pew Research Center recently did a major survey into the views of a wide spectrum of professions on this issue. The results are quite revealing.

Tuesday, July 05, 2016

Mobile Brings Data to Real Time Decision Making

The increasing us of mobile smartphones and tablets is making it possible to generate data in the field and transmit it directly for analysis. In one example recently cited where such an approach was used for helping refugees,  mobile data, including GPS data and the results of interviews were collected and sent directly to a center using ESRI analytics tools.

There, the data could be immediately visualized, analyzed and used for ongoing decisions.

"With ESRI software, we were able to visualize what was going on,'' said Andrew Schroeder, director of research and analysis at Direct Relief, a Santa Barbara, Calif., nonprofit organization that coordinates humanitarian aid and care for people in poverty or emergencies, noting that ArcGIS's mapping capabilities "allowed us to understand some of the dynamics around issues such as who seeks care, where they are from and what their neighborhood conditions are."

For more on this experience with real time data analytics, check out this article.

Tuesday, June 28, 2016

How Big Data can Enable Targeted Marketing

Traditional marketing has used the broad broom approach, where advertisements are placed on media for general consumption, such as TV, Newspapers, etc. However, these media are attracting a smaller cohort than they did and therefore the ads miss their targets. Also, advertising on broad based media has become very expensive. The best example of this is the massive millions that are spend on superbowl ads.

Many of the people who are missed through the broad based marketing are now using hand held devices, such as smart phones and tablets. This applies to many younger people but is not limited to them. So advertising campaigns need to be segmented so that multiple campaigns can be run simultaneously that are directed to those segments.

This is where big data comes in. Data can be obtained from social media networks and internet usage data that show where and when the target audiences are using their media. Such advertising can then be much more effective in reaching audiences and promoting sales.

This article shows how targeted marketing techniques can be used in election campaigns and what business can learn from politics. Check it out.

Friday, June 10, 2016

Big Data for Cybersecurity

The growing use of big data has definite links to the problem of cybersecurity. Contrary to what many might think, the linkage is not all negative. While the risks in use of big data need to be controlled as do any other IT application, nevertheless big data can be useful in helping to control that risk. For example, predictive analytics can be applied to a combination of historical data and statistical metadata to give enterprises the ability to predict the probability of an intrusive event happening in the future. Also, big data analytics enables the data to be retained in its original form, whether structured or unstructured, thus providing much more flexibility in the analytical process. For a good rundown on this area, click this link.

Wednesday, May 25, 2016

Your Job is Being Redefined

A recent report by McKinsey & Co examines the role of automation in work life. It shows that 45% of work activities overall could be automated. It goes on to say, however, that only 5% of occupations can be replaced with current technology. So few occupations are likely to be replaced with technology in the near future. However, 60% of all occupations could have 30% or more of their activities automated.

This redefinition of jobs extends across the spectrum, from professionals like accountants and lawyers to travel agents (already largely redefined), bank tellers (heavily redefined) and airline pilots (who often steer as little as 4% of a flight.

Job redefinition is a major challenge for people as well as their employers. It calls for constant retraining and continuing change. Clearly adaptability and learning are more and more crucial to success in this evolving world.

Friday, May 20, 2016

Use Big Data Analytics for Marketing Strategy


A report by McKinsey on more than 250 engagements over five years shows that companies that put data at the centre of their marketing and sales decisions improve their marketing return on investment (MROI) by 15 – 20 percent. If this is applied to the estimated $1 trillion in global annual marketing spending, that adds up to $150 – $200 billion of additional value.

This is the primary reason why big data analytics has revolutionized marketing and sales. Analysis of big data can reveal new opportunities for a company. And the companies can tailor their product to customer wishes and beliefs, thus influencing their decision behaviour. In order to make analytics work, it is essential that a company invest on the latest techniques to enable fast analysis on the rapidly expanding pool of big data available to them., such as automated “algorithmic marketing,” an which provides for the processing of vast amounts of data through a “self-learning” process to create better and more relevant interactions with consumers - a kind of combination of data analytics and artificial intelligence. For an interesting take on this point of view, check this link.

Tuesday, May 10, 2016

How Big Data Analytics can Help with People/Talent Management

Analytics is entering into management at all levels. No longer a geek thing, it has become the fodder of top management. Check out this article on the subject.

Wednesday, May 04, 2016

How Big Data Analytics Helps Toyota Manage Accounts

For those who wonder how big data analytics is being used to help management, the case of Toyota Financial Services (TFS) serves as an illuminating one. TFS finances the ale of cars to customers and carries a portfolio of about $80 billion worldwide.

During the financial recession, delinquencies rose dramatically. The conventional collection techniques seemed to be consuming a lot of resources with mixed results.

TFS implemented a big data approach under which they collected data pertinent to the customers and applied algorithms to optimize the collection processes.

"their teams collaborated to create a new approach that included multiple technologies to assess individual consumers for their risk. FICO had developed an algorithm that allowed TFS to estimate which customers needed attention and the best way to approach each of them. Other technologies included SAS for statistics and predictive analytics, Oracle software and database software, IBM Pure Data (formerly known as Netezza), Tableau Software integrated into the user interface, Informatica for data integration, VMware for virtualization, and more. The solution relies on multiple technologies from multiple vendors and resides in Toyota Financial Services' many data centers."

The combination of optimization techniques, predictive analytics and prescriptive analytics all combined to yield fairer treatment of their customers, better use of resources and a better collections outcome.

For a more complete rundown, and the source of the above quote, click this link.

Tuesday, May 03, 2016

Google Analytics vs Spark

In a recent exhaustive, study, Mammoth Data found that Google Cloud Dataflow outperformed Apache Spark in several categories. This is important because both Google and Spark are widely used for big data analytics. Most companies are jumping into the big data world for reasons of competitive necessity.

In its benchmark, Mammoth Data identified five key areas where Google Cloud Dataflow equalled or exceeded Apache Spark:

  • Greater performance
  • Developer friendly
  • Operational simplicity
  • Easy integration
  • Open-source
For more detail, check out this link.

Monday, May 02, 2016

Data Analytics on iPhones and iPads

iPhones and iPads are not often thought of as useful for data analytics. However, they can be so because some apps are available that simply act as a client under which the data remains on the cloud and the analysis takes place there too. Analyzing data on the cloud is the way of the future, since the idea for downloading is not feasible for much big data and users would have to resort to samples if downloading is used.

Data analytics apps range from simple analysis tools like Google Analytics to WolframAlpha, Statistics Visualizers and Roambi and even some analytics programming languages like Scala and Python. Those who are interested can spend hours experimenting with these tools while learning at the same time. Most are free from the Apple App Store. Some work in conjunction with Siri, thus enabling some StarTrek-like analysis.

For a brief rundown on 10 of these apps, check out this page.

Friday, April 22, 2016

Is Technology Going to Take your Job?

With the explosion of big and not-so-big data there has been an explosion in research on ways in which to use that data. One of the avenues of exploration has been in the area of artificial intelligence, where the capabilities of machines to emulate human behaviour is growing. For example, in the areas of professional services, like accounting and law, new software tools combined with the availability of massive amounts of data are making it possible to increase the numbers of evidence-based decisions as opposed to judgmental decisions. This in turn eliminates or reduces the more routine roles.

There is a long history of machines replacing human endeavour, and it is clear that this trend will if anything accelerate with the growth in the capabilities of technology.


"A 2013 study by researchers at Oxford University posited that as many as 47% of all jobs in the United States are at risk of “computerization.” And many respondents in a recent Pew Research Center canvassing of technology experts predicted that advances in robotics and computing applications will result in a net displacement of jobs over the coming decades – with potentially profound implications for both workers and society as a whole." (Pew Research Website)

Interestingly a majority of those surveyed said there own job would still exist in future.

For more on this trend, you can click on this link.

Saturday, April 16, 2016

The Changing Face of Data

The availability of data from corporate systems, legacy systems, social media, public databases, Internet of Things and numerous other sources has been much discussed. Most and perhaps all of these sources of data have often been grouped under the label of Big Data.

While companies are coming to recognize the growing importance of big data, there are issues around the ability to actually use it. Some of the data is structured (organized in standard formats and understandable on its own). Other data is unstructured, meaning any useful analysis can only come after some restructuring is carried out to make the data understandable by the analytics tools being used.  Some of these tools, often based on the Hadoop framework, can handle unstructured data. Others have difficulty.

The difficulty is compounded by the fact that the data of interest is becoming available in different forms beyond that of simple numeric data. It includes text (for which analytical tools have been available for years), video, audio, graphics and other forms. The latter are very difficult to structure, particularly with tools that can be used across platforms.

The answer comes in different forms. One approach is to structure as much data as possible, using recognized standards such as XML and XBRL. But that generally applies in an effective way only to numeric data or structured non-numeric data.

Besides structuring data, an approach is to build larger data storage areas, where tools can be used across a variety of formats in some consistent way. While this is not a magic wand to fix all the analysis issues, it does allow for a more coordinated approach to data analytics and management. Many companies are going this route.

Check out this link.

Thursday, April 14, 2016

Big Data Analytics Merging with Enterprise Systems

In a recent address in San Jose California, Doug Cutting, creator of Hadoop, the open source framework for big data analytics, reviewed the course of Hadoop and Big Data over the past ten years (yes, its been happening for ten years!). He pointed out how Big Data analytics has now moved into the space previously held by ERP and other Enterprise systems. Companies are using big data analytics increasingly to enable evidence-based decisions.

Over those 10 years, new and more powerful technologies have been introduced to improve Hadoop and enable better analysis. While much analysis started with MapReduce, many organizations are now using Apache Spark - also open source and powerful.

In a new study issued by Oxford Economics, it was pointed out that the use of big data analytics will explode with the availability of new data from the Internet of Things (IoT), a rapidly growing feature of the internet under which all kinds of items are connected to the internet and generating data. This would include appliances, houses, cars, and so on - your imagination is the limit. The study supports the predictions of Mr Cutting that big data analytics using Hadoop will become a central part of enterprise systems over the next ten years.

For more on these topics, check out this article and this report.  

Monday, March 28, 2016

Big Data Defined

With all the discussion about big data, there is a persistent problem. There is not general agreement on a definition of big data. For some, it means data available on the internet generally; for others, it's data coming from social media, or the internet of things. It sometimes refers to unstructured data and for others includes structured data such as that available from relational databases.

Sometimes big data is defined according to the tools used to analyze it, such as Hadoop or Spark. For others it relates to data from enterprise systems, like ERP and CRM.

Thee are lots of definitions around. Wikipedia, for example, says "big data is a term for data sets that are so large or complex that traditional data processing applications are inadequate." Most people would say this definition is too narrow.

Webopedia defines it as follows: "Big data is a buzzword, or catch-phrase, meaning a massive volume of both structured and unstructured data that is so large it is difficult to process using traditional database and software techniques."

This definition is better as it focusses on structured and unstructured data, which encompasses both data from traditional business systems as well as internet data such as that from social media. It also refers to massive in quantity, which is one of the defining characteristics.

A more analytical approach to defining big data is through the use of the words Volume, Variety and Velocity, perhaps with the word Variability. But sole use of these words does not clearly define big data. Nevertheless, they do provide a framework for a definition. Volume means very large amounts of data. Variety means data coming from very different sources, from business systems to the Internet of Things. Velocity is important because big data is always moving in fast, and there is a trend now to streaming analytics to recognize this. Variability means the data changes in volume, format and source.

Wrap these together and we can approach a definition. Big data is structured and unstructured data coming from a variety of sources, such as business systems, social media and the internet of things, moving at a high velocity and with frequently changing sources, formats and subject matter.

This definition may not be perfect and elegant, but at least it is broad but specific and encompasses the generally understood characteristics of big data.

For some further reading, check out these references: from Techtarget, Wikipedia and Webopedia.



Thursday, March 24, 2016

Bank of Russia Implements XBRL for SMEs


Online media portal Russia Today is reporting that the Bank of Russia is planning on simplifying procedures for the issuance of securities by SMEs, including the introduction of XBRL. The intention is to improve the bond market by cutting costs and improving the flow of information to investors. They also plans legislation changes to improve overall handling of debt with an eye towards reducing reliance on banks as a source of funding for SMEs.

More regulators are seizing on the opportunity presented by using structured data and the resulting transparency to meet strategic aims like promoting economic growth, transforming capital markets or improving government performance. This is a trend we expect to see more of and to make its way down into the enterprise as well. 

(source: XBRL International Newsletter)

Monday, March 21, 2016

Business Goes for Streaming Analytics

In a move that is likely to prove a watershed, business has been moving into Streaming Analytics. A recent report by Forrester Research pinpointed this trend in a study of the adoption of data analytics. They pointed out that three years ago, business was struggling with ways to apply analytics to their existing data stores and some big data.

Forrester defines streaming analytics software as technology that "can filter, aggregate, enrich, and analyze a high throughput of data from multiple disparate live data sources and in any data format to identify simple and complex patterns to visualize business in real-time, detect urgent situations, and automate immediate actions."(source)

A number of prominent software providers and technologies are available, including "Apache Spark Streaming, Apache Storm, Data Torrent, IBM, Informatica, SAP, Software AG, SQLstream, Strim (WebAction), TIBCO, and Vitria," says Forrester.

The advent of streaming analytics could herald a new era in business decision making. In the past, decisions have largely been based on historical information with attempts to extrapolate into the future using whatever current information and intuition is available. Streaming analytics will reduce the uncertainty of this approach and add some real science to the decision making process.




Thursday, March 17, 2016

Where Big Data Analytics is Headed

We hear much about big data and big data analytics. And we are told that it is the big next thing. But how much of this is hype. And where is it really going?

Forbes has published a summary of predictions that sheds some light on the whole matter. It highlights how the amount of data will grow exponentially, that analytics tools will be changing to move past SQL into Spark and others. More importantly they talk of how new user friendly tools are being released, such as those from Microsoft and Salesforce, that do not require programming expertise. Even more importantly they predict that machine learning will play a big role in the future of data analytics, with perhaps even tools that operate free of people.  They also stress the importance of prescriptive analytics, which takes descriptive and predictive analytics to the next stage by indicating not only what will happen in the future but why it will happen, paving the way for serious support of decision making.

Overall the development of data analytics is heading for uncharted waters and may take directions we don't see yet.

For this thought-stimulating article in Forbes Magazine, follow this link.

Tuesday, March 15, 2016

Predictive Analytics can Improve Business Decisions

Predictive Analytics is a means of studying large amounts of data and drawing from it inferences about future behaviour of customers, employees, stakeholders, and others. While other kinds of analysis can indicate problem areas in, say sales, and tell management what isn't working, predictive analytics can indicate what policies are likely to work before they are implemented.

The availability of big data is particularly useful for predictive analytics because of the sheer volume of data and the coverage of behaviour it encompasses.

Companies are therefore using predictive analytics with increasing success in a variety of circumstances, including analyzing individual customer traits to determine how best to serve them and to determine the most effective procurement strategies in advance. Specific information on the elasticity of demand can also be used to determine the best price/production strategies. There is a myriad of possible scenarios where predictive analytics can be used, which accounts for its popularity.

For some specific examples, check out this site.

Monday, March 14, 2016

Data Analytics for the Internet of Things

As the numbers of buildings, cars, appliances and other things get connected to the internet, and the data generated by this connectivity grows in volume, And yet, the variety and sheer scope of the data available almost defies interpretation and analysis.  The task is not only one of analyzing data from different platforms, but the more difficult task of analyzing data coming from widely disparate devices.

Data showing driving conditions in a particular area, for example, can be distorted by a myriad of non relevant events. The same goes for liveability conditions in a particular type of building. The volume of IoT data is so great that is can't even be analyzed on the cloud.

Data analytics is attempting to address this data, but it has been recognized that there needs to be some sort or order brought into the data and the analytical approach taken.

A team of researchers supported by the National Science Foundation in the US is looking into this issue and is charged with developing a framework for conducting data analytics across a variety of IoT devices. The framework will consist of an organization of software that will facilitate communications and research into the data. The research is led by Stacy Patterson, the Clare Boothe Luce Assistant Professor of Computer Science at Rensselaer Polytechnic Institute (RPI). Read more at: http://phys.org/news/2016-03-internet-thingsa-framework-analytics-digital.html#jCp


Friday, March 11, 2016

Using Data Analytics for Developing Emerging Markets

Data Analytics, including that involving big data, is increasingly being used for decision making in a variety of organizations. Some of the data is embedded in new systems and some is gathered from the internet on, for example, social media and through the internet of things.

The essential task of big data analytics is to transform raw and structured data from a variety of disparate sources into actionable knowledge.

This process involves using new tools, often based on Hadoop, such as Google DataQuery and BigQuery. It makes use of contemporary computer capabilities such as high speed communications, massive storage capability and super powerful processors.

A good example of big data analytics in action is that of emerging energy markets, such as that in the Gulf of Mexico's Mexican region.

"Data analytics is being used through most of the lifecycle of offshore activities. During seismic and reservoir characterization studies, data sources with 3D seismic data, well logs and faults, are integrated and analyzed to support decisions related to achieving key targets in flow assurance, field optimization, drilling performance, well categorization and so forth. Benefits range from attaining optimal reservoir exploitation rate to forecasting the decline of new wells.

"For fixed, floating and subsea assets, data analytics starts with collecting data at the asset level, including operating parameters, equipment status, structural stresses and environmental data. For moving assets such as offshore support vessels and dynamic positioning floaters/vessels, data collection can also include location, direction and speed."

In this way big data analytics facilitates decision making in a difficult market. For more on this particular application, check out this link.

Monday, March 07, 2016

LLoyds and Google Team Up

Lloyds Group has teamed up with Google in analyzing big data relating to its insurance customer's non-personal behaviour. In the initial project they analyzed a year's worth of data in under one minute using Google BigQuery. One of the outcomes is that they were able to reduce certain response times from 96 hours to 30 minutes - a remarkable achievement.

The work will continue with a wider array of data and more Google tools for big data analytics, including Data Flow and Big Table.

Big Data Analytics is starting to show real results and while there is still a novelty factor to it, within the year it will be a competitive necessity in many industries.


Friday, March 04, 2016

Big Data Analytics Begins to Mature

The ability to make effective use of big data has been hampered by the lack of big data skills along with the lack of useful tools for analysis. Both of these areas are being addressed by interested organizations.

The lack of big data scientists has been bemoaned since the advent of big data and the realization of its potential. Rutgers is making an important step forward by offering a new program that focuses on big data skills. On March 29, the Center for Innovation Education at Rutgers University will begin its 44-week skills-based technology career certificate program for professionals who want to gain skills in big data disciplines. Initially the course is only open to recent grads in the US. But is marks the beginning of much needed data oriented education that hopefully will spread.

And also there are important innovations going on in the availability of analytical tools. Google features strongly in this field with its announcement that Google Dataproc, its managed Apache Hadoop and Apache Spark service, is now available to the public. Who other than the world's most prominent exploiter of data would step up to the major challenges of offering powerful user oriented tools for big data analysis using the Hadoop system which has been the core of much big data analytical activity.

We can expect a rash of new product announcements as big data gains in importance for business policy. Check out this article for more.

Friday, February 05, 2016

Blockchain Gains Traction in Banking and Finance



An opinion piece in TechCrunch Online has some interesting things to say about the uptake of blockchain technology in the finance sector. The article relates how both the NASDAQ in the US and the Australian Stock Exchange, along with many other financial institutions, are looking to employ the technology as a way to provide shared infrastructure which allows transactions, that can now can take days to finalise, to instead be settled instantly and transparently. Blockchain is the technology underlying Bitcoin. (Source)

Tuesday, January 26, 2016

Hadoop Changing How Companies use Data

The era of big data has been changing corporate systems. First is the influx of data from systems like CRM and others. The advent of data availability from social media is greatly increasing the flow of big data. In this there is much opportunity.

The issue is that the data is very large in volume and usually too large to place in a data base or to download. Also, much of it is unstructured data that is difficult to analyze.

Hadoop is a technology that enables distributed storage of data and distributed analysis on an efficient basis. It enables the data to be analyzed without downloading it, which is a major advantage. It also is increasingly being tied into various data analytics tools that can deal with vast quantities of unstructured data and perform predictive analysis.

With the opportunities beginning to be realized, a new vigour is being added to decision making. For more, check out this link.