Real-time C2C matching of social media messages

Abstract

Social media enables personalization of the Consumer to Consumer (C2C) business model where people could directly do business with each other without an intermediary by sharing their products, services, and consumer requirements. However, messages shared by both the sellers and potential buyers do not reach each other as they are embedded among other social media messages. Moreover, C2C buy/sell interest matching in real time is nontrivial due to the complexities of interpreting social media messages, number of messages, and diversity of products and services. We present a platform for real-time matching of microblogging messages related to product selling or buying in C2C. We adopt a combination of techniques from natural language processing, complex event processing, and distributed systems. First, we extract the semantics of messages such as product attributes and commercial intention of the message either buying or selling using information extraction. Then the extracted buy/sell messages are matched using a complex event processor. Moreover, NoSQL and in-memory computing are used to enhance scalability and performance. The proposed solution shows a high accuracy where commercial intent classification and Conditional random fields based named entity recognition recorded an accuracy of 98.5% and 82.07%, respectively when applied to a real-world dataset. Information extraction, in-memory data manipulation, and complex event processing steps introduced low latency were latencies were 0.5 ms, 5 ms, and 0.2 ms, respectively. For the given setup with modest hardware, we were able to process 3,400 messages per second and overall latency was 0.76 ms.