Real Time Credit Card Fraud Detection with Apache Spark and Event Streaming
https://mapr.com/blog/real-time-credit-card-fraud-detection-apache-spark-and-event-streaming/
Editor's Note: Have questions about the topics discussed in this post? Search for answers and post questions in the Converge Community.
In this post we are going to discuss building a real time solution for credit card fraud detection.
There are 2 phases to Real Time Fraud detection:
- The first phase involves analysis and forensics on historical data to build the machine learning model.
- The second phase uses the model in production to make predictions on live events.
Building the Model
Classification
Classification is a family of supervised machine learning algorithms that identify which category an item belongs to (for example whether a transaction is fraud or not fraud), based on labeled examples of known items (for example transactions known to be fraud or not). Classification takes a set of data with known labels and pre-determined features and learns how to label new records based on that information. Features are the “if questions” that you ask. The label is the answer to those questions. In the example below, if it walks, swims, and quacks like a duck, then the label is "duck".
Let’s go through an example of car insurance fraud:
- What are we trying to predict?
- This is the Label: The Amount of Fraud
- What are the “if questions” or properties that you can use to predict ?
- These are the Features, to build a classifier model, you extract the features of interest that most contribute to the classification.
- In this simple example we will use the the claimed amount.
Linear regression models the relationship between the Y “Label” and the X “Feature”, in this case the relationship between the amount of fraud and the claimed amount. The coefficient measures the impact of the feature, the claimed amount, on the label, the fraud amount.
Multiple linear regression models the relationship between two or more “Features” and a response “Label”. For example if we wanted to model the relationship between the amount of fraud and the the age of the claimant, the claimed amount, and the severity of the accident, the multiple linear regression function would look like this:
AmntFraud = intercept+ coeff1 age + coeff2 claimedAmnt + coeff3 * severity + error.
The coefficients measure the impact on the fraud amount of each of the features.
Let’s take credit card fraud as another example:
- Example Features: transaction amount, type of merchant, distance from and time since last transaction .
- Example Label: Probability of Fraud
Logistic regression measures the relationship between the Y “Label” and the X “Features” by estimating probabilities using a logistic function. The model predicts a probability which is used to predict the label class.
- Classification: identifies which category (eg fraud or not fraud)
- Linear Regression: predicts a value (eg amount of fraud)
- Logistic Regression: predicts a probability (eg probability of fraud)
Linear and Logistic Regression are just a couple of algorithms used in machine learning, there are many more as shown in this cheat sheet.
Feature Engineering
Feature engineering is the process of transforming raw data into inputs for a machine learning algorithm. Feature engineering is extremely dependent on the type of use case and potential data sources.
(reference Learning Spark)
Looking more in depth at the credit card fraud example for feature engineering, our goal is to distinguish normal card usage from fraudulent card usage.
- Goal: we are looking for someone using the card other than the cardholder
- Strategy: we want to design features to measure the differences between recent and historical activities.
For a credit card transaction we have features associated with the transaction, features associated with the card holder, and features derived from transaction history. Some examples of each are shown below:
Model Building Workflow
A typical supervised machine learning workflow has the following steps:
- Feature engineering to transform historical data into feature and label inputs for a machine learning algorithm.
- Split the data into two parts, one for building the model and one for testing the model.
- Build the model with the training features and labels
- Test the model with the test features to get predictions. Compare the test predictions to the test labels.
- Loop until satisfied with the model accuracy:
- Adjust the model fitting parameters, and repeat tests.
- Adjust the features and/or machine learning algorithm and repeat tests.
Read Time Fraud Detection Solution in Production
The figure below shows the high level architecture of a real time fraud detection solution, which is capable of high performance at scale. Credit card transaction events are delivered through the MapR Streams messaging system, which supports the Kafka .09 API. The events are processed and checked for Fraud by Spark Streaming using Spark Machine Learning with the deployed model. MapR-FS, which supports the posix NFS API and HDFS API, is used for storing event data. MapR-DB a NoSql database which supports the HBase API, is used for storing and providing fast access to credit card holder profile data.
Streaming Data Ingestion
MapR Streams is a new distributed messaging system which enables producers and consumers to exchange events in real time via the Apache Kafka 0.9 API. MapR Streams topics are logical collections of messages which organize events into categories. In this solution there are 3 categories:
- Raw Trans: raw credit card transaction events.
- Enriched: credit card transaction events enriched with card holder features, which were predicted to be not fraud.
- Fraud Alert: credit card transaction events enriched with card holder features which were predicted to be fraud.
Topics are partitioned, spreading the load for parallel messaging across multiple servers, which provides for faster throughput and scalability.
Real-time Fraud Prediction Using Spark Streaming
Spark Streaming lets you use the same Spark APIs for streaming and batch processing, meaning that well modularized Spark functions written for the offline machine learning can be re-used for the real time machine learning.
The data flow for the real time fraud detection using Spark Streaming is as follows:
1) Raw events come into Spark Streaming as DStreams, which internally is a sequence of RDDs. RDDs are like a Java Collection, except that the data elements contained in RDDs are partitioned across a cluster. RDD operations are performed in parallel on the data cached in memory, making the iterative algorithms often used in machine learning much faster for processing lots of data.
2) The credit card transaction data is parsed to get the features associated with the transaction.
3) Card holder features and profile history are read from MapR-DB using the account number as the row key.
4) Some derived features are re-calculated with the latest transaction data.
5) Features are run with the model algorithm to produce fraud prediction scores.
6) Non fraud events enriched with derived features are published to the enriched topic. Fraud events with derived features are published to the fraud topic.
Storage of Credit Card Events
Messages are not deleted from Topics when read, and topics can have multiple different consumers, this allows processing of the same messages by different consumers for different purposes.
In this solution, MapR Streams consumers read and store all raw events, enriched events, and alarms to MapR-FS for future analysis, model training and updating. MapR Streams consumers read enriched events and Alerts to update the Card holder features in MapR-DB. Alerts events are also used to update Dashboards in real time.
Rapid Reads and Writes with MapR-DB
With MapR-DB (HBase API), a table is automatically partitioned across a cluster by key range, and each server is the source for a subset of a table. Grouping the data by key range provides for really fast read and writes by row key.
All of the components of the use case architecture we just discussed can run on the same cluster with the MapR Converged Data Platform. There are several advantages of having MapR Streams on the same cluster as all the other components. For example, maintaining only one cluster means less infrastructure to provision, manage, and monitor. Likewise, having producers and consumers on the same cluster means fewer delays related to copying and moving data between clusters, and between applications.
Summary
In this blog post, you learned how the MapR Converged Data Platform integrates Hadoop and Spark with real-time database capabilities, global event streaming, and scalable enterprise storage.
References and More Information:
- Free Online training on MapR Streams, Spark, and HBase at learn.mapr.com
- Getting Started with MapR Streams Blog
- Ebook: New Designs Using Apache Kafka and MapR Streams
- Ebook: Getting Started with Apache Spark: From Inception to Production
- https://www.mapr.com/blog/parallel-and-iterative-processing-machine-learning-recommendations-spark
- https://www.mapr.com/blog/fast-scalable-streaming-applications-mapr-streams-spark-streaming-and-mapr-db
- https://www.mapr.com/blog/apache-spark-machine-learning-tutorial
- https://www.mapr.com/blog/life-message-mapr-streams
- https://www.mapr.com/blog/spark-streaming-hbase
- Apache Spark Streaming Programming Guide
- Fraud Analytics Using Descriptive, Predictive, and Social Network Techniques: A Guide to Data Science for Fraud Detection Book, by Wouter Verbeke; Veronique Van Vlasselaer; Bart Baesens
- Learning Spark Book, By Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia
Real Time Credit Card Fraud Detection with Apache Spark and Event Streaming的更多相关文章
- WARN deploy.SparkSubmit$$anon$2: Failed to load org.apache.spark.examples.sql.streaming.StructuredNetworkWordCount.
前言 今天运行Spark Structured Streaming官网的如下 ./bin/run-example org.apache.spark.examples.sql.streaming.Str ...
- Apache Spark 2.2.0 中文文档
Apache Spark 2.2.0 中文文档 - 快速入门 | ApacheCN Geekhoo 关注 2017.09.20 13:55* 字数 2062 阅读 13评论 0喜欢 1 快速入门 使用 ...
- Apache Spark 2.2.0 中文文档 - Structured Streaming 编程指南 | ApacheCN
Structured Streaming 编程指南 概述 快速示例 Programming Model (编程模型) 基本概念 处理 Event-time 和延迟数据 容错语义 API 使用 Data ...
- codeforces 893D Credit Card 贪心 思维
codeforces 893D Credit Card 题目大意: 有一张信用卡可以使用,每天白天都可以去给卡充钱.到了晚上,进入银行对卡的操作时间,操作有三种: 1.\(a_i>0\) 银行会 ...
- 论文泛读:Click Fraud Detection: Adversarial Pattern Recognition over 5 Years at Microsoft
这篇论文非常适合工业界的人(比如我)去读,有很多的借鉴意义. 强烈建议自己去读. title:五年微软经验的点击欺诈检测 摘要:1.微软很厉害.2.本文描述了大规模数据挖掘所面临的独特挑战.解决这一问 ...
- (原创)北美信用卡(Credit Card)个人使用心得与总结(个人理财版) [精华]
http://forum.chasedream.com/thread-766972-1-1.html 本人2010年 8月F1 二度来美,现在credit score 在724-728之间浮动,最高的 ...
- Educational Codeforces Round 33 (Rated for Div. 2) D. Credit Card
D. Credit Card time limit per test 2 seconds memory limit per test 256 megabytes input standard inpu ...
- magento 开启 3D secure credit card validation
因为国外盗刷严重,于是得开启验证. 首先可以去 https://developer.cardinalcommerce.com/try-it-now.shtml.这上面有测试账号,截图如下:
- [Angular] Using directive to create a simple Credit card validator
We will use 'HostListener' and 'HostBinding' to accomplish the task. The HTML: <label> Credit ...
随机推荐
- 转:java内部类作用
原文地址:https://www.cnblogs.com/uu5666/p/8185061.html 一. 定义 放在一个类的内部的类我们就叫内部类. 二. 作用 1.内部类可以很好的实现隐藏, 一般 ...
- SQL row_number() over(partition by函数
1)row_number() over(partition by 列名1 order by 列名2 desc)的使用 表示根据 列名1 分组,然后在分组内部根据 列名2 排序,而此函数计算的值就表示每 ...
- Code once, debug everywhere.
1.通常语言调用一个函数会出exception的情况,在javascript里面返回的是undefined.等到程序运行不正常的时候,你看到数据结构的有些地方为什么是undefined,只能哭了. 2 ...
- 【雅思】【写作】【大作文】Advantage VS. Disadvantage
Advantage VS. Disadvantage Advantage vs. Disadvantage 社会现象或者做法 “People can work or study on the Inte ...
- 深入SQL Server优化【推荐】
深入sql server优化,MSSQL优化,T-SQL优化,查询优化 十步优化SQL Server 中的数据访问故事开篇:你和你的团队经过不懈努力,终于使网站成功上线,刚开始时,注册用户较少,网站性 ...
- head 命令 读取文件的前n行,默认查看文件的前十行
head 命令 读取文件的前n行 默认查看文件的前十行 head /etc/passwd # 默认查看文件的前十行 /etc/passwd # 查看文件的前两行
- 教你一步步composer安装Magento2.3
以前外贸建站一直用zencart,这段时间ytkah比较有时间,就决定用magento来创建一下站点.magento不像普通的程序一样下载就可以直接安装,需要借助composer安装,还没没compo ...
- 升级my.cnf注意
升级my.cnf注意 mkdir -p /ngbs/data/{logs,tmp} vi /etc/init.d/mysqlbasedir=/usr/local/mysql datadir=/ngb ...
- 20180310 KindEditor 富文本编辑器
问题: 如何判断富文本编辑器文本内容非空 错误的办法,采用js 对控件本身的txt ID 号抓取获取值,由于加载富文本编辑器时,界面的ID 已经经过了修改或者可以用转换来说,所以抓取是无效果的. 需要 ...
- Python3学习之路~8.3 socket 服务端与客户端
通过8.2的实例1-6,我们可以总结出来,socket的服务端和客户端的一般建立步骤: 服务端 步骤:1创建实例,2绑定,3监听,4阻塞,5发送&接收数据,6关闭. #Author:Zheng ...