Big Data and Analytics: Eight Obstacles You Need to Overcome

Companies that leverage big data and analytics are disrupting the logistics industry.

Companies that make data-based decisions outperform companies that rely on opinion-based decisions – however informed they may be.

Big data and analytics are disrupting the logistics industry. It’s doing so mainly because it’s leveraging new technologies. These technologies offer new ways to store, manage, analyze and exploit, data. They’re moving beyond the pervasive Excel spreadsheet.

Shippers and 3PLs don’t always agree on what’s important. Yet when it comes to big data and analytics, they’re in overwhelming agreement. For example, 93% of shippers and 98%  of 3PLs believe data driven decision-making is essential to supply chain management.

To make big data and analytics work for you, you must be aware of the obstacles. But, if you’re unaware of the obstacles, they can disrupt your supply chain.

What you don’t know can hurt you.

In the case of big data and analytics, ignorance is not bliss.

They’re not insurmountable, but you must address them. It will make the difference between a successful or unsuccessful implementation.

In developing a big data and analytics program it’s important to know what not to do as well as what to do. Then you can avoid the obstacles that can disrupt your supply chain.

Awareness of these eight obstacles will help you in designing a big data and analytics program.


#1: Collecting Huge Amounts of Data

To make a big data and analytics program work you need to have huge amounts of data. The more data the better. (But the data must be useful – more on that later.) Big data can help you achieve better outcomes when data is plentiful. Plentiful data smooths out aberrations, outliers, and one-off events. It gives you a better version of the truth to inform your decisions.

Thus, it’s important to identify all the sources of data you’ll need for analysis. Moreover, having huge amounts of data can help in identifying trends not seen before. It enables data mining, a potentially powerful element of big data and analytics. That leads to the next obstacle.


#2: Collecting Data from Multiple Sources

As you map out your data needs, you’ll find that all the data you need is not centralized. It won’t be in one database, data warehouse or data lake. That means you will likely have varying types of data. You will have structured and unstructured data.

Putting them in a system to make use of different data types presents a challenge. Some data will be real-time, and some will be near-real-time. Both will likely be transaction data. You’ll also find aggregated, historical data. That brings us to ensuring the data you collect is, well, useful.


#3: Collecting Useful Data

You don’t want to collect data for data’s sake. The data you collect must have a purpose. It must relate directly to issues that impact your supply chain. One example is performance. You can use big data and analytics to streamline your supply chain. You can leverage it to increase visibility. Or you can apply big data to improving cycle time.

Another example is financial: improving ROI and reducing distribution and warehousing costs. So, it’s imperative you identify why you need the data. That will inform what data you collect. It also helps identify where you’ll obtain the data. Finally, it will ensure you collect useful data. To ensure usefulness your data must align with your business goals.


#4:  Ensuring Data Quality

With those obstacles out of the way, you now need to focus on achieving data quality. If you don’t have good data, your decisions will reflect that. It’s the familiar GIGO or Garbage In, -Garbage Out.

There is no substitute for clean data. Back in 2014, target Canada experienced a significant data disconnect. The barcodes on items shipped to warehouses came in faster than they were going out for delivery. This happened with the opening up of new distribution centers and 124 stores.

The data disconnect was detrimental. Its scale was damaging.

Bad data not only affects operations, it can also affect your brand.

Without clean data you won’t have the confidence of the decisions you make. The underlying data must support the decisions you’re looking for. So, you must ensure data is in the right format, is consistent, is not duplicated, or incomplete.

With data coming from multiples sources, you’ll find a single data standard is missing. Data formats will vary. Omissions of data will show up. And currency/latency issues will appear.


#5:  Presenting Data Visually

The obstacles discussed this far are familiar. This next one is too, but it’s often overlooked. Yes, you need huge amounts of data. You need a variety of data. Your data must have purpose, and it must be clean. That said, you also need to ensure your data lends itself to visual presentation.

There are many presentation tools on the market to choose from. Some are better than others. The key is to find a presentation tool that meets your needs, your purpose. Senior executives will want to see the big picture. They don’t want to get lost in the details. Additionally, analysts will need dashboards to visually present critical information in real-time.

Seeing data, especially large amounts of data will help expedite decision-making.  It may also facilitate early warning notifications. That allows analysts to head off issues before becoming problems. That’s the power of data visualization. Finally, data visualization helps make data more accessible.


#6: Ensuring Data Accessibility

Data accessibility is a crucial component of big data and analytics. As mentioned above data visualization promotes data accessibility. That means you increase accessibility as more users access data to find meaning. The more common aspect we all think of, however, is being able to physically access data.

Accessing huge amounts of data isn’t easy. Should you centralize or decentralize data? That depends on your particular needs. Regardless, you need to ensure those who need the data have physical access, so they can put the relevant data to use.

In short, you must ensure your data is accessible when  and where it’s needed. Whether users need access to real-time data to make dynamic decisions or to access to historical data to identify trends, ease of accessibility is crucial.


# 7: Ensuring Data Security

Data must be accessible, but not to unauthorized outsiders. To make use of big data and analytics you must ensure data security. Once you’ve identified what type of data you need and how you’re going to use it, you must secure it. That’s no easy task. You have to contend with hackers as well as competitors. You also have to combat other ne’er-do-wells that want to access or corrupt your data.

As with presentation software, there are many cyber security solutions on the market. These solutions protect your networks and your data. You need to determine which solution work best for you by assessing your needs. Again it comes down to balancing protection and accessibility.

That said your data security must be robust. It must be comprehensive and powerful enough to withstand all types of data breaches. Data breaches can severely compromise a single operation, or they can bring a business to its knees. It can skew timing, confuse coordination, and hamper supply availability. These are just a few examples.


#8: Ensuring Data Talent

Last but not least is data talent. This is a looming obstacle that affects every business. Business

are eager to hire data scientists that can analyze data and provide new insights. Big data and analytics, including AI is fueling the growth of this career field.

Studies reveal a shortage of data scientists exists today and will only worsen. Quanthub cites several reports estimating the shortage at 85 million positions in 2020. That’s huge, and the demand for data scientists is increasing. Data show that more businesses are adopting big data and analytics programs. Gaining competitive advantage is the most cited reason for this growth.

In another LinkedIn report (2015), Austin, TX, had a shortage of 26 data scientists. In 2018, just three years later that shortage ballooned to 5,000.

Complicating the issue of talent is that data scientists do not have a uniform skill set. There is a baseline, but the skills required vary by industry. So finding data scientists specializing in logistics poses a challenge. And that has no immediate solution. That’s why many logistics companies now grow that talent in-house supplementing their new hires.

Finally, turnover tends to be high in this career field. It’s so new companies have ill-defined positions and don’t know how to leverage this talent. As a result, data scientists leave after short stints with their companies.


How to Shore Up Your Big Data and Analytics Program

Mastering big data and analytics can give you a competitive advantage. Mastering big data and analytics, however, isn’t simple. But you can ensure a smooth implementation with awareness of these eight obstacles.

There are other issues, to be sure. But addressing this helps avoid the disruption caused by being unaware of what can hurt you.

With the rise of big data and analytics, you need to develop a data program to make sense your data. From developing long-term strategies to monitoring performance to heading off risk, big data looms large.

At the core of any big data and analytics program is using data smartly. To do that you must align your data needs with your business strategy. That may sound simple, but it’s often overlooked.

At American Global Logistics, we’ll work with you to focus on what’s important to you. We take a customer-centric approach, so your data needs will focus on what you want to achieve.

Some of the best ideas come from our clients. That’s why we take a collaborative approach – to avoid overlooking good ideas. In working with American Global Logistics, you’ll get a kick-start as we avoid the eight obstacles discussed in this post.