Tải bản đầy đủ
Chapter 10. Use Case: HBase as a Master Data Management Tool
require more memory and leverage expensive solid-state drives (SSDs). On the flip
side, HBase utilizes commodity hardware and SATA drives. This is what led Collec‐
tive to start looking at HBase. Luckily, Collective already had a Hadoop cluster in
play, and had the advantage of seamlessly integrating HBase into the existing infra‐
structure with minor development and cost overhead.
Collective currently has 60 RegionServers deployed serving 21 TBs of data out of
HBase alone. Collective’s HBase deployment is pretty straightforward. For this use
case, there is a single table that handles the user profile data. The table is broken into
three column families consisting of visitor, export, and edge. The “visitor” CF con‐
tains the metadata about the user. This will consist of information such as date of
birth, behavior information, and any third-party lookup IDs. The “export” CF con‐
tains the segment information (an example of a segment would be male, 25 years old,
likes cars), and any relevant downstream syndication information needed for pro‐
cessing. The “edge” CF contains the activity information of the user, along with any of
the additional data that may come in from the batch imported data:
edge:batchimport ts=1391769526303, value=\x00\x00\x01D\x0B\xA3\xE4.
export:association:cp ts=1390163166328, value=6394889946637904578
export:segment:13051 ts=1390285574680, value=\x00\x00\x00\x00\x00\x00\x00\x00
export:segment:13052 ts=1390285574680, value=\x00\x00\x00\x00\x00\x00\x00\x00
export:segment:13059 ts=1390285574680, value=\x00\x00\x00\x00\x00\x00\x00\x00
visitor:ad_serving_count ts=1390371783593, value=\x00\x00\x00\x00\x00\x00\x1A
visitor:behavior:cm.9256:201401 ts=1390163166328, value=\x00\x00\x00\x0219
visitor:behavior:cm.9416:201401 ts=1390159723536, value=\x00\x00\x00\x0119
visitor:behavior:iblocal.9559:2 ts=1390295246778, value=\x00\x00\x00\x020140120
visitor:behavior:iblocal.9560:2 ts=1390296907500, value=\x00\x00\x00\x020140120
visitor:birthdate ts=1390159723536, value=\x00\x00\x01C\xAB\xD7\xC4(
visitor:retarget_count ts=1390296907500, value=\x00\x00\x00\x00\x00\x00\x00\x07
As already mentioned, Collective is a digital advertising company that enhances
offers and profiles through consumer interactions. Each user is tracked through a
custom cookie ID that is generated upstream in the process. The row key is a reverse
of that cookie ID. This begs the question of why to reverse a generated UUID. There
are two primary offenders that require reverse keys: websites and time series data. In
this case, the beginning of the cookie ID has the timestamp on it. This would lead to
monotonically increasing row keys; by simply reversing the row key, the randomly
generated portion now occurs first in the row key.
In Chapters 8 and 9, we looked at near real-time ingest pipelines and batch loading
processes. Next, we are going to look at combining the two while using HBase as the
system of record, which is sometimes referred to as master data management
| Chapter 10: Use Case: HBase as a Master Data Management Tool
(MDM); a system of record is used as the “golden copy” of the data. The records con‐
tained in HBase will be used to rebuild any Hive or other external data sources in the
event of bad or wrong data. The first piece we will examine is batch processing. For
this system, Collective built a tool that pulls in third-party data sources from numer‐
ous data sources covering S3, SFTP sites, and a few proprietary APIs (Figure 10-1).
The tool pulls from these data sources on an hourly basis and loads the data into a
new HDFS directory using the Parquet file format. A new Hive partition is then cre‐
ated on top of the newly loaded data, and then linked to the existing archive table.
Figure 10-1. Ingest dataflow
The other side of the house is the near real-time processing, which is currently being
handled by Flume. Flume brings in data from different messaging services, which is
pushed into a system known as Parquetify. The Parquetify system, which runs hourly,
prepares the data into a unified file format, wraps a temporary table around the data,
and then inserts the data into the main archive Hive tables. In this case, the file for‐
mat used for this data is Parquet files. Once the data is loaded into the system, the
aptly named preprocessor Harmony is run every hour in a custom workflow from
Celos. Harmony collects and preprocesses the necessary data from the previously lis‐
ted sources. This is used to normalize the output together for the formal transforma‐
tion stages. Currently this runs in a series of Flume dataflows, MapReduce jobs, and
Hive jobs. Collective is in the process of porting all of this to Kafka and Spark, which
will make processing both easier and faster.
Once Harmony has joined the inbound data together, it is then sent for final process‐
ing into another internal system that coordinates a series of MapReduce jobs together
known as Pythia (Figure 10-2). The MapReduce jobs create a write-ahead log (WAL)
for maintaining data consistency in case of failures. There are three total steps in this
• Update MDM system
Figure 10-2. Processing dataflow
Both the Aggregator and ProfileMod steps in the pipeline also represent backed and
partitioned Hive/Impala tables. The first thing done is to read the output of the Har‐
mony job into an Avro file for our Aggregator job. Once new edits (Harmony) and
previous hour of data (HDFS) are read and grouped in the mapper, they are passed to
the reducer. During the Reduce phase, numerous HBase calls are made to join the full
profile together for the next hourly partition. The reducer will pull the full HBase
record for each record that has an existing profile. The Aggregator job then outputs a
set of differences (typically known as diffs) that is applied during the ProfileMod
stage. These diffs are used as a sort of WAL that can be used to rectify the changes if
any of the upstream jobs fail.
Next, the ProfileMod job is executed. ProfileMod will be a Map job because we
already extracted the data we needed from HBase in the reducer from the Aggregator
flow. The mapper will read all of the data from the previous output and rectify the
diffs. Once the diffs are all combined together, ProfileMod will use these as the actual
diffs that need to be written back to HBase. The final output of this job is a new
hourly partition in the ProfileMod Hive table.
Chapter 10: Use Case: HBase as a Master Data Management Tool
Finally, the MDM system of record (HBase) needs to get updated. The final step is
another MapReduce job. This job (the Updater) reads the output of the profile mode
data and then builds the correct HBase row keys based off the data. The reducer then
updates the existing rows with the new data to be used in the next job. Figure 10-3
shows the complete dataflow from the different inputs (stream and bach) up to the
destination Hive table.
Figure 10-3. Complete dataflow
Implementation of HBase as a Master Data
In Chapter 10, we reviewed the implementation of a customer 360 solution. In addi‐
tion to HBase, it uses several different applications, including MapReduce and Hive.
On the HBase side, we described how MapReduce is used to do lookups or to gener‐
ate HBase files. In addition, as discussed in the previous chapter, Collective plans to
improve its architecture by using Kafka. None of this should be new for you, as we
covered it in detail in previous chapters. However, Collective is also planning to use
Spark, and this is where things start to be interesting. Indeed, over the last several
years, when applications needed to process HBase data, they have usually used Map‐
Reduce or the Java API. However, with Spark becoming more and more popular, we
are seeing people starting to implement solutions using Spark on top of HBase.
Because we already covered Kafka, MapReduce, and the Java API in previous chap‐
ters, instead of going over all those technologies again to provide you with very simi‐
lar examples, we will here focus on Spark over HBase. The example we are going to
implement will still put the customer 360 description in action, but as for the other
implementation examples, this can be reused for any other use case.
MapReduce Versus Spark
Before we continue, we should establish the pros and cons of using Spark versus
using MapReduce. Although we will not provide a lengthy discussion on this topic,
we will briefly highlight some points for you to consider as you narrow down your
Spark is a recent technology, while MapReduce has been used for years. Although
Spark has been proven to be stable, you might be more comfortable with a technol‐
ogy deployed in hundreds of thousands of production applications—if so, you should
build your project around MapReduce. On the other hand, if you prefer to rely on
recent technologies, then Spark will be a good fit.
For companies with good MapReduce knowledge that already have many MapReduce
projects deployed, it might be easier, faster, and cheaper to stay with something they
know well. However, if they start a brand-new project and are planning many
projects, they might want to consider Spark.
Some use cases require fast and efficient data processing. MapReduce comes with a
big overhead. It gives you reliability and many other benefits, but at the price of a per‐
formance hit. If your use case has tight SLAs, you might want to consider Spark.
Otherwise, MapReduce is still a good option.
One last consideration is the development language. Spark is compatible with both
Java and Scala code, while MapReduce is mostly for Java developers.
So which one should you choose? If you are still deciding between the two, we recom‐
mend trying Spark, as it has nice benefits over MapReduce.
Get Spark Interacting with HBase
As with MapReduce, there are two main ways for Spark to interact with HBase. The
first is to run a Spark application on top of an HBase table. The second is to interact
with HBase while running Spark on other data. We will assume that you have at least
basic spark knowledge. Refer to the Spark website for more information.
Run Spark over an HBase Table
When we are running a MapReduce job on top of an HBase table, each mapper pro‐
cess a single region. You can also run a simple Java application that will scan the
entire HBase table. This will be very similar with Spark. The HBase Spark API can
return you an RDD representing your entire HBase table. But you partition this RDD
to get it processed over multiple executors where each of them will process a single
Calling HBase from Spark
The HBase Java API is accessible from the Spark code. Therefore, you can perform
lookups like what we did when implementing our Flume interceptor. You can also use
Spark to enrich and store data into HBase leveraging Puts and Increments. Last, the
same way you can use MapReduce to generate HFiles for BulkLoads, you can use
Spark to generate the same kind of files that you will later BulkLoad into HBase.
Chapter 11: Implementation of HBase as a Master Data Management Tool