In today’s digital-first world of marketing and sales, companies are turning to automation to connect with more customers. While live chat widgets proliferate across websites all over the world, chatbots and digital assistants have become more and more popular for their ability to engage with customers 24/7. Unfortunately, the majority of “do it yourself” chatbots are providing a poor user experience, and one of the big reasons is web scraping.
Web scraping is how most chatbots get their information; crawling your website for data and then using that information to serve to users when they ask a question. Rarely do these web scraping chatbots return a correct, or relevant, answer.
While the pros of using a web scraping chatbot include lower upfront costs and ease of implementation, the cons are plentiful, and can greatly impact your business in the long run.
Data inaccuracy
Web scraping chatbots rely on the static data that resides on your website. This information can quickly become obsolete, resulting in inaccurate responses being provided to the user. In addition, multiple members of a team could be making changes to website content and mistakes are more likely. In addition, because web scraping chatbots do not pull the information in real-time for every conversation, any change in the information on a website may not be reflected in the answers a chatbot provides.
Inability to understand intent
As we know from using the search function of any website, keywords are often what trigger the results we see. All content, from products to events to blog posts and pages, can be returned in any given search that includes a specific keyword. Obviously, a user did not want to see all these results, which could number in the thousands. Unfortunately, web scraping chatbots are set up the same way, and cannot understand the intent of the user’s question. This leads to unrelated search results and incorrect information being returned to the user.
Data analysis and code translation
Much of the information pulled by these lower-end chatbots contains HTML and other web code. Most web scraping chatbots are not able to parse the language from the code, resulting in users being served a combination of HTML, text and more. Manual work must be done to remove this code from search results. Often, these chatbots require the website to be encoded in very specific ways, otherwise the performance of the chatbot will break. For older websites, this could create a huge amount of work for the team to update a site that was not built exactly to specifications.
Access to secure information
While web scrapers can access all of the public-facing content on your website, they don’t have any access to the content that lies beyond a login screen. From account details to e-commerce order information to personal financial data, web scraping chatbots cannot help customers seeking related information. Customers want to engage with a chatbot to access tracking information for their order, or potentially use a digital assistant to check their account balance. Only virtual assistants that are connected directly into these systems can provide a level of customer experience that today’s consumer would take for granted.
Drain on resources
Chatbots that automatically scrape a company’s website for information will use an incredible amount of web resources, negatively impacting the user experience. These chatbots will slow down the performance of a site as it crawls for content changes. And because these chatbots cannot see only the data that has changed since its last crawl, it must read the entire website each time, looking for new or changed information.
Hidden costs to reputation
As the problems of data inaccuracy, intent understanding and data analysis begin to compound, the seemingly low-cost solution begins to cost the company in numerous ways. One of the biggest negative impacts for a brand will be reputation. Consumers have long memories when it comes to bad experiences, and if a customer is doing research online between a few competitors and has a poor experience interacting with your chatbot, they’ll quickly move on to the next brand. Just like your website, any chat widget, whether it’s live chat, a chatbot or a virtual assistant, is critical to making a positive first impression. While chatbots are inherently better than a live chat feature because they are available 24 hours a day, a chatbot that just returns search results scraped from your website still provide a subpar experience compared to a smart digital assistant. Companies can’t afford to assume that just because they have a chatbot installed on their website, they are providing a great customer experience.
