In March 2017, Elastic detailed how Singapore-based cashback and rewards platform ShopBack used the Elastic Stack to speed up search and gain visibility into its systems. At the time, ShopBack worked with more than 1,300 merchants in six countries, and data analytics underpinned its effort to help shoppers find the best deals faster.
The company had moved from Amazon Web Services’ CloudSearch, through Amazon’s Elasticsearch service, to a cluster it ran itself under an Elastic subscription that included X-Pack.
Why ShopBack left AWS search
According to Elastic, ShopBack first used AWS CloudSearch to power user search, but performance lagged. Product searches averaged two seconds, and bringing up a store listing could take as long as 30 seconds. The business then moved to Amazon’s Elasticsearch service before deciding to run its own cluster.
Alberto Resco Perez, ShopBack’s engineering manager, said in 2017 that the AWS-hosted service ran an older Elasticsearch version and so lacked the full Elastic Stack feature set. It also was not set up for multi-lingual search, which he said made a switch necessary if the business was to scale.
The Elastic cluster and what it delivered
ShopBack subscribed to both the Elastic Stack and X-Pack. Elastic said the premium support in the subscription helped ShopBack design its Elasticsearch schemas, while X-Pack features helped secure the cluster. With help from Elastic’s local team, ShopBack spun up its first cluster in a few hours.
That cluster handled indexing and search for the product catalogue. By 2017 it covered 13 million products and 15 thousand categories. Elastic reported that product searches could take as little as one millisecond and that store listings appeared in four seconds. Auto-suggestion also went live for ShopBack’s two million users.
ShopBack had migrated from Elasticsearch 2.4 to 5.0. Resco Perez said the move brought a schema for multi-country, multi-lingual search, which supported expansion into Indonesia and Taiwan, where the company handled local languages. His own December 2016 presentation at an Elastic meetup in Singapore described the same move to a 2.4 cluster, with a “Product” micro-service and indices built per country and language.
A second cluster for logs and metrics
ShopBack later added a second Elastic Stack cluster to simplify logging and metric collection. It had previously used Papertrail, AWS CloudWatch and New Relic together, which Elastic said made scaling expensive and complicated. ShopBack also wanted more metrics and more insight into how its services performed.
The new setup used Logstash and Metricbeat to ingest logs and collect metrics from different sources, and Kibana to visualise and analyse them. Shield, the security component of X-Pack, protected both clusters. ShopBack said this gave it more insight at lower cost.
Resco Perez said visibility had been limited before, and that the monitoring and alerting features of X-Pack helped the team detect and fix problems faster. They also helped it analyse user behaviour in order to scale systems to demand. One concrete change: updates on CPU, memory and disk demand arrived every 15 seconds instead of every five minutes, giving the team more time to add capacity and lowering the risk of downtime.
ShopBack’s plans in 2017 included using Filebeat to collect all of its service metrics and applying Elastic’s machine learning to fraud detection. Resco Perez called fraud a major problem for e-commerce.
Elastic’s view of the analytics challenge
Matias Cascallares, a solution architect at Elastic, said in 2017 that the Elastic Stack addressed big-data problems by letting organisations search, analyse and visualise large, complex data sets, with X-Pack adding security, alerting, monitoring and reporting on a subscription basis. He pointed to integrations with more than a hundred tools and data sources, and to an open-source community that kept extending the technology.
He identified real-time insight from fast-growing data as the main difficulty. In his account, organisations once held far less data and expected nothing immediate, and the arrival of the Hadoop ecosystem changed that by making large-scale analysis possible and raising demand for real-time, actionable insight.
Cascallares named unstructured and geo-aware data as the next big categories. He said Elastic was working with one of Singapore’s leading banks on analysing the location data in customers’ credit card transactions, so it could make real-time decisions on service or targeted promotions. He advised companies to adopt analytics early in their transformation and to lean on the open-source community.
FAQ
What did ShopBack’s catalogue architecture look like?
Resco Perez’s 2016 slides showed a crawler feeding product updates through a synchroniser, mapper, image uploader and indexer into the catalogue. An early version used a single index with separate types per country for products and stores, and later versions refined the language analysers.
Which monitoring tools did the Elastic Stack replace?
In 2017 ShopBack said it was replacing the combination of Papertrail, AWS CloudWatch and New Relic, which it had found costly and hard to scale.