How we verify payments with ELK and Kibana

The site is an international e-commerce operating with several currencies, countries and with a big portfolio of products. Transactions can get very intense, especially on days such as Black Friday, and Sales, so it is really important to verify these in the most effective way possible.

Why it is important to verify payments in real time 

There are a few reasons why it is so important to establish an automated system to verify payments on an ecommerce platform, but most importantly is to understand that a lack of verification or a failure system can cost a pretty good amount of money. 

These are the main problems we can detect with a payment verification system: 

• Problem detection: First a foremost, our payment verification system will detect any problem in the workflow: slow response times, disruptions of service, communication errors and any other issue that can hinder the purchase process.
• Fraud: Many fraud situations and attempts can be detected in real time and either prevented or investigated: purchases with multiple recurring cards, random access to payment methods, suspicious transactions and much more can be detected and then stored for future reference should similar patterns repeat.
• Call center support: it’s an invaluable tool for any customer care service, allowing the quick tracing and reconstruction of the payment flow for any order inquiry. Providing a  satisfactory solution to most customer inquiries.
• Application errors: E-Commerce platforms are complex entities and this service allows the instant detection of some disrupting issues like: wrong pricing, errors in scheduled jobs or privacy violation like the involuntary printing of sensible information.

What technologies we use for establishing a payment verification system

For this specific project, we have used the following technologies:

• Custom ELK implementation that collects and parses payment events in real time
• Kibana tailored dashboards for instant monitoring and drill down analysis

In addition we also set the option to deploy the log monitoring system in the different paying methods to offer a unified solution (depends on Payment technology and platform)

How many data we track in a single day 

Every successful transaction is composed of multiple Request/Reply events up to dozens for each order. When, incorrect or failed d transactions generate much more traffic. That is why it is so important to verify this data in real time.

For this specific project we have collected and displayed up to 150.000 events a day during peak season. Without a verification system it would be indeed impossible to track every single payment action happening in such a high traffic ecommerce.