Analytics industry has witnessed significant growth over the years but is still prone to a lot of challenges in terms of talent, reaching the right consumers, cumulating data points, among others. This month we reached out to the analytics leaders to understand the challenges that they most often face in the industry. We asked them questions around challenges of analytics adoption, challenges of moving from pilot to production, challenges specific to the industry they are serving, budget allocation, among others.
3 Key Challenges That Analytics Industry Still Faces Today
Translating data to business impact: The journey of data from being just observation to information to knowledge and ultimately wisdom is complex. “While the challenges of the journey from observation to information have been solved almost completely, challenges remain in gaining actionable insights out of information. The separation of insight from the noise of information is the next big leap that analytics has to take,” says Abhinav Kimothi, Senior Manager, Analytics, Lymbyc Solutions.
Aayush Rai, CEO and Co-founder of Inito also believes that the biggest challenge faced by the analytics industry today is not having sufficient high-quality datasets. “Though companies love big data, most of them don’t know what to do with it in order to use it effectively for their business goals,” he said.
Faisal Husain, Co-founder and CEO, Synechron also shared similar thoughts that most companies today recognise the importance of big data and analytics for their business, but they fail to devise a robust big data strategy to create business value for the data residing with them.
Multiple sources of data: The second big challenge that industry thinks they face is that data is scattered and there are multiple sources of data making its accessibility quite difficult. “Enterprises are dealing with an overwhelming amount of data coming in from multiple disjointed channels. Manually compiling this data becomes time-consuming and prone to human error, resulting in an inaccurate analysis. Lack of a centralised platform or system leads to difficulty in synchronising the data sets and deriving meaningful insights,” says Husain.
“With our tremendous growth in advanced analytics space and newer applications being explored every day, there is a huge demand for unconventional data. This has resulted in the birth of multiple sources to procure data thereby creating the paradox of choice. One of the major challenges this industry is facing, is to choose, store, manage, utilise and analyse this data in a reliable manner and derive value for investment,” shares Prajwal R, Senior Partner at TheMathCompany.
A spokesperson from FSS says that data comes in different sources and formats.
Paramjeet Virdi, Director, Marketing, Strategy and Analysis, Publicis Sapient echoes similar views. She says that data duplication, data anomalies and integrity issues have increased making it essential to be addressed upfront. “An incorrect input of data leads to a misleading analytics output, hence diminishing the trust in data and it’s derived insights,” she added.
Data quality: Data quality is one of the issues given the large volumes of streaming and unstructured data that organisations must deal with. “Gaps in data quality tend to have a cascading effect on the efficacy of analytics, and most organisations are only able to deal with data quality issues retrospectively. Being able to identify gaps and enhance data quality at the source is an essential step to succeed with analytics,” says Nilesh Teli, Sr. Vice President – Data Science and Consulting, CitiusTech.
Talent Shortage Remains One Of The Biggest Concerns
Talent shortage and a gap in demand and supply remain two of the biggest challenges. While there is a huge demand for data scientists and analytics professionals and that there are ample job openings in the field, the end goal of hiring analytics talent fails to meet. “Also, due to huge demand and supply gap, the talent does not shy away from looking out for more lucrative opportunities, leading to high compensation-related attrition,” says Manashi Kumar, Chief of Strategy and People Operations, BARC India.
“Finding the right talent that understands the domain, useful insights, the approach and the mix of tech skills remains a challenge,” says Srikanth Gunturu, Senior Principal Software Architecture, Sabre.
Kaushik Srinivasan, SVP Strategy – Products and Research, eMudhra shares similar views as he says that the biggest challenge for companies that engage in development is keeping pace with technological changes, getting the required skill set in the market and training the skillset to solve client-specific use cases.
Prajwal R also shared that there is a limited pool of this talent with specific skills, and hence, getting the right talent is a big challenge. “One of the many solutions that organisations could adopt is to up-skill their existing employee force to ensure they are adapted to the need of time,” he shares.
However, Adrit Raha, CEO, Vivant believes that while it is still a challenge, the scenario has increased since the last 5 to 6 years. “The situation is improving for sure, but we still have some distance to cover and as the industry matures, I am sure we would have a big enough talent pool,” he says.
While hiring is a challenge, many companies are resorting to upskilling to get the ends meet. “Today the challenge is not about the talent availability but understanding the depth of knowledge and applying the skills they hold in the context of business problems. However, hiring at senior levels is still a challenge. A few organisations such as ours spend considerable time creating training programs to groom internal talent,” says Virdi.
Teli also shared that they are addressing talent requirements through a mix of focused hiring and internal competency development across multiple areas such as data management, performance management, AI/ML, healthcare domain, etc.
Vineet Chaturvedi, Co-Founder, Edureka is hopeful that the situation might improve in the coming years as there’s already an increasing interest and urgency for upskilling among tech and data professionals. “Meanwhile, in-house referrals and upskilling also help meet the demand,” he said.
Key Challenges While Setting Up Analytics Function
Many industry leaders believe that analytics function should not be separated from other functions in an organisation. And that one of the key requirements of setting up analytics function is to have a skilled workforce. Teli Shares that identifying, grooming and retaining talent is likely to be the biggest challenge in establishing an analytics function. “Organisations also need to understand and build aligned capabilities like data integration, data mining, enterprise data warehouse management, data lakes, data security, user experience management, etc,” he says.
“To be able to set up analytics function in an organisation, it is important to understand analytics and define what you need,” says Virdi. She also shares other key steps such as investing early in analytics to hope for the expected results and defining a roadmap for the organisation and keep evolving.
Defining goals and roadmap, getting buy-in and sponsorship from different stakeholders, selecting the right tools, getting the right data for analytics use cases, are some of the key ways to set up analytics functions in an organisation.
Biggest Challenge In Carving AI Roadmap – Budget, Talent Or Senior Management Buy-In?
All three factors i.e. budget, talent and senior management buy-in comes as a major challenge in carving out AI roadmap in an organisation. “A successful AI Roadmap is dependent on all three – budget, talent and senior management buy-in, as adopting an AI approach would mean revisiting their budget allocations and understanding how AI-enabled tools will affect the current workforce,” says Husain.
Prashantha Shet, Senior Director, Software Engineering at Sabre believes that the biggest challenge in AI roadmap is to first determine the business value of applying AI. another big challenge is talent especially if a company decides to make something in-house.
Raha shares that Management buys remain the single biggest challenge. “Rest all could be managed and arranged for, but getting all relevant stakeholders aligned to the initiative, effort involved and outcomes expected, remains the toughest proposition,” he says.
Kimothi also says that if he had to choose one, it will be the alignment of the decision-makers (or senior management) with the AI strategy. The ultimate challenge is on the management to have a well-defined plan of using AI at different levels,” he says.
Prajwal R says that the biggest hurdle today is the mindset of organisations. “The first step is to believe in the potential value of analytics and push themselves to take the right risk and make the right investment to make it happen. This has to be followed by excellence in execution make the investment-worthy,” he said.
On the other hand, Kumar and Chaturvedi believe that talent is one of the biggest challenges.
Challenges Of Moving From Pilot To The Production Stage
Kimothi of Lymbyc Solution shares that the challenge occurs even before the pilot step. “The scale-up strategy has to be defined before the pilot otherwise the pilot will end up making insignificant gains,” he says.
Srinivasan believes that the biggest challenge in moving from pilot to production is the ability of the system to handle scale, processing and accuracy. Whereas Teli says that data access, people & process and tools & technologies remain three significant challenges. The spokesperson from FSS has a take that data quality and missing data fields come as a challenging aspect.
Paramjeet Virdi says that in building production-ready analytics solutions an initial pilot may work on smaller sets of data and basic infrastructure. “However, in the case of production, it requires a clear choice of data platform and infrastructure. Hence it is advisable to decide on tech-stack required, volumes and type of data to be managed, type of governance that will be required among others,” she says.
“What works well with a hundred people in a pilot may not work for thousands. In analytics, you have to consider a lot of external and environmental factors that may contribute to change in the variables that can affect the data so when you scale up some things might not work. Companies need to iterate fast in improving there dataset and algorithms based on the most up-to-date learnings that can be generated in trials,” says Rai.
Industry-Specific Challenges
Some of the most common challenges that these leaders face across various industries such as healthcare, finance, FMCG and others remain the same. Based on the inputs of the leaders we compiled a list of common industry-specific challenges as below:
- Lack of analytics talent
- Historically the data hasn’t been captured and organised well
- Quality and specificity of data insights
- Clinical data is extremely complex and diverse
- Availability of Data for analytics due to privacy and security concerns of data
- Data is in different sources and in different formats
- Ease of consumption of data
- Difficulty in integrating with legacy/proprietary systems
- Huge data sets which are both structured and unstructured
Ways To Overcome Commonly Faced Challenges
While the leaders listed a lot of challenges that the analytics industry is currently facing, they also had ways to overcome the commonly faced challenges. While Gunturu believes that automating decision process and introducing smart systems and solve the problems, Raha says that the most crucial aspect is to clearly define business objectives from the very beginning.
Kimothi lists a few things such as aligning business strategy with analytics strategy, the democratisation of data, augmentation of analytics with decision making and thinking scale rather than speed.
Husain echoes on the importance of having a data strategy in place to leverage data and analytics for competitive advantage. “The Data and Analytics team should use an ontological approach while applying the principles of data science and follow a set of guiding principles for their data-analytics framework,” he says.
Performance Metrics Used By C-suite To Measure Business Outcome
Analytics adoption is crucial to businesses and analytics leaders depend on a lot of performance metrics to measure business outcomes based on analytics play. Srikant Gunturu says that growth in revenue, business margins and enhanced user experience are some of the metrics.
Kimothi shares that it depends on the strategic objectives of the organisation. “The ultimate performance metrics for AI and analytics are the same as any other process or function. For a more developed traditional business, the measure would be margins while modern social media businesses focus on creating a loyal user base,” he said.
“The goal, eventually, for all stakeholders, is to figure if the initiative has resulted in better, faster, and smarter decisions, with measurable business results. So, the first metrics are definitely w.r.t the quality of the resulting business decisions such as profit, customer service, or cost. The second factor is speed. The third key metrics is around robustness,” said Raha.
Rai shares that revenue, margins and growth are all lagging indicators. “Customer feedback and engagement are key metrics that help measure our business outcome,” he says on a concluding note.
