Insights from data analytics help ShopBack offer the best deals

ShopBack partners with more than 1,300 merchants in six different countries to bring happiness to online shoppers in the form of rewards and new store discoveries. With the help of Elastic, it’s enhancing users’ online experiences and making it faster and easier for them to find the best deals.

ShopBack initially used Amazon Web Services (AWS) CloudSearch to power user search but performance was lagging; product searches took an average of two seconds while bringing up a store listing could take up to 30. The business migrated to Amazon’s Elasticsearch service for faster search before finally setting up its own cluster.

“Through AWS, we were using an older version of Elasticsearch and missing out on the full features and benefits of the Elastic Stack. Also, the service we had was not set up to handle multi-lingual search. So to scale the business we knew we had to make a switch,” said Alberto Resco Perez, Engineering Manager at ShopBack.

ShopBack made the decision to implement its own cluster, subscribing to both the Elastic Stack and X-Pack. The premium support  that came with the subscription helped ShopBack to design its Elasticsearch schemas while the features offered by X-Pack helped to ensure the security of its cluster.

With the help of the local Elastic team, ShopBack spun up its first cluster in just a few hours. This cluster supports the indexing and search of ShopBack’s product catalogue and has evolved over time to support the indexing of 13 million products and 15 thousand categories.

Performance has improved significantly with product searches taking as little as one millisecond and store listings coming up in just four seconds. Also, auto-suggestion is now live on the site, helping ShopBack’s two million users find what they want even faster.

“We’ve migrated from Elasticsearch 2.4 to 5.0 and not only has performance improved but we now have a schema for multi-country, multi-lingual search. This has aided our expansion into Indonesia and Taiwan where we support local languages,” said Alberto.

New insights improve site performance and availability

More recently ShopBack has implemented a new cluster of the Elastic Stack to simplify logging and metric collection. In the past, ShopBack had used Papertrail, AWS Cloud Watch and New Relic to collect and analyse various metrics. However, the combination of all three solutions made it expensive and complex to scale. ShopBack also wanted to collect more metrics and get more insight into the performance of its services.

It now uses Logstash and Metricbeat to ingest logs and collect metrics from a variety of sources. It uses Kibana to visualise and analyse the data. The Elastic-based solution offers the ShopBack team greater insights at a reduced cost. It’s also more secure with Shield (security capability of X-Pack) used to protect both of ShopBack’s clusters.

“In the past, we had limited visibility into our environment. The Elastic Stack and additional monitoring and alerting capabilities of the X-Pack has resolved that and helps us to detect and resolve issues so much faster,” said Alberto. “It also helps us to analyse user behaviour so we can scale our systems and meet demand.”

As an example, ShopBack now receives updates on CPU memory and disk demand every 15 seconds as opposed to every five minutes. This gives the business more time to manoeuvre to scale capacity and reduce the risk of downtime.

ShopBack has plans to extract even more benefit from the Elastic Stack in the future. This includes using Filebeat to collect all its service metrics and leveraging machine learning for fraud detection.

“Fraud is a big issue for e-commerce  and one that we hope to tackle with the help of machine learning from Elastic,” said Alberto.

Data analytics challenges

Matias Cascallares, Solution architect at Elastic, says the Elastic Stack solves many of the challenges of big data, helping organisations to search, analyse and visualise large and complex data sets. With the subscription-based X-Pack it also offers capabilities for security, alerting, monitoring and reporting.

“The Elastic Stack is open and extensive and integrates with more than a hundred tools and data sources; one of the many beauties of open source is that it is backed by a community of developers that extend the technology with new features and benefits.”

Cascallares noted that one of the challenges when it comes to data analytics today is extracting real-time insights from these huge datasets that are constantly growing and evolving.

“In the past, organisations simply didn’t have the massive amount of data that they have today. Also, technology was not real-time so there were not the same expectations there are today for immediate insights,” said Cascallares in response to questions from Networks Asia.

“The emergence of the Hadoop ecosystem was a catalyst for change in the way organisations looked at data. It opened the door to analysing data on a much larger scale and has led to demand for real-time, actionable insights. The technology available today enables this and makes data and security analytics much more effective.”

Cascallares also said that unstructured and geo-aware data are the next big forms of data that organisations should be looking at.

“We see a number of opportunities that can be unlocked using this type of data. For example, we are currently working with one of the top banks in Singapore to support analytics of the geo-data on customers’ credit card transactions. Using this data, they’ll be able to make real-time decisions to improve the customer experience or offer targeted promotions.”

Cascallares further said that with the right technology in place, data analytics will become more actionable and effective. “However, companies should embrace analytics early on in their transformation and leverage the power and support of the open source community to accelerate their progress,” he added.