Tải bản đầy đủ
Chapter 5. Big Data Storage Concepts

Chapter 5. Big Data Storage Concepts

Tải bản đầy đủ

•dataismanipulatedtobemadeamenablefordataanalysis
•dataisprocessedviaanETLactivity,oroutputisgeneratedasaresultofan
analyticaloperation
DuetotheneedtostoreBigDatadatasets,ofteninmultiplecopies,innovativestorage
strategiesandtechnologieshavebeencreatedtoachievecost-effectiveandhighlyscalable
storagesolutions.InordertounderstandtheunderlyingmechanismsbehindBigData
storagetechnology,thefollowingtopicsareintroducedinthischapter:
•clusters
•filesystemsanddistributedfilessystems
•NoSQL
•sharding
•replication
•CAPtheorem
•ACID
•BASE

Clusters
Incomputing,aclusterisatightlycoupledcollectionofservers,ornodes.Theseservers
usuallyhavethesamehardwarespecificationsandareconnectedtogetherviaanetworkto
workasasingleunit,asshowninFigure5.1.Eachnodeintheclusterhasitsown
dedicatedresources,suchasmemory,aprocessor,andaharddrive.Aclustercanexecute
ataskbysplittingitintosmallpiecesanddistributingtheirexecutionontodifferent
computersthatbelongtothecluster.

Figure5.1Thesymbolusedtorepresentacluster.

FileSystemsandDistributedFileSystems
Afilesystemisthemethodofstoringandorganizingdataonastoragedevice,suchas
flashdrives,DVDsandharddrives.Afileisanatomicunitofstorageusedbythefile
systemtostoredata.Afilesystemprovidesalogicalviewofthedatastoredonthestorage
deviceandpresentsitasatreestructureofdirectoriesandfilesaspicturedinFigure5.2.
Operatingsystemsemployfilesystemstostoreandretrievedataonbehalfofapplications.
Eachoperatingsystemprovidessupportforoneormorefilesystems,forexampleNTFS
onMicrosoftWindowsandextonLinux.

Figure5.2Thesymbolusedtorepresentafilesystem.
Adistributedfilesystemisafilesystemthatcanstorelargefilesspreadacrossthenodes
ofacluster,asillustratedinFigure5.3.Totheclient,filesappeartobelocal;however,this
isonlyalogicalviewasphysicallythefilesaredistributedthroughoutthecluster.This
localviewispresentedviathedistributedfilesystemanditenablesthefilestobe
accessedfrommultiplelocations.ExamplesincludetheGoogleFileSystem(GFS)and
HadoopDistributedFileSystem(HDFS).

Figure5.3Thesymbolusedtorepresentdistributedfilesystems.

NoSQL
ANot-onlySQL(NoSQL)databaseisanon-relationaldatabasethatishighlyscalable,
fault-tolerantandspecificallydesignedtohousesemi-structuredandunstructureddata.A
NoSQLdatabaseoftenprovidesanAPI-basedqueryinterfacethatcanbecalledfrom
withinanapplication.NoSQLdatabasesalsosupportquerylanguagesotherthan
StructuredQueryLanguage(SQL)becauseSQLwasdesignedtoquerystructureddata
storedwithinarelationaldatabase.Asanexample,aNoSQLdatabasethatisoptimizedto
storeXMLfileswilloftenuseXQueryasthequerylanguage.Likewise,aNoSQL
databasedesignedtostoreRDFdatawilluseSPARQLtoquerytherelationshipsit
contains.Thatbeingsaid,therearesomeNoSQLdatabasesthatalsoprovideanSQL-like
queryinterface,asshowninFigure5.4.

Figure5.4ANoSQLdatabasecanprovideanAPI-orSQL-likequeryinterface.

Sharding
Shardingistheprocessofhorizontallypartitioningalargedatasetintoacollectionof
smaller,moremanageabledatasetscalledshards.Theshardsaredistributedacross
multiplenodes,whereanodeisaserveroramachine(Figure5.5).Eachshardisstoredon
aseparatenodeandeachnodeisresponsibleforonlythedatastoredonit.Eachshard
sharesthesameschema,andallshardscollectivelyrepresentthecompletedataset.

Figure5.5AnexampleofshardingwhereadatasetisspreadacrossNodeAandNode
B,resultinginShardAandShardB,respectively.
Shardingisoftentransparenttotheclient,butthisisnotarequirement.Shardingallows
thedistributionofprocessingloadsacrossmultiplenodestoachievehorizontalscalability.
Horizontalscalingisamethodforincreasingasystem’scapacitybyaddingsimilaror
highercapacityresourcesalongsideexistingresources.Sinceeachnodeisresponsiblefor
onlyapartofthewholedataset,read/writetimesaregreatlyimproved.
Figure5.6presentsanillustrationofhowshardingworksinpractice:
1.Eachshardcanindependentlyservicereadsandwritesforthespecificsubsetofdata
thatitisresponsiblefor.
2.Dependingonthequery,datamayneedtobefetchedfrombothshards.

Figure5.6AshardingexamplewheredataisfetchedfrombothNodeAandNodeB.
Abenefitofshardingisthatitprovidespartialtolerancetowardfailures.Incaseofanode
failure,onlydatastoredonthatnodeisaffected.
Withregardstodatapartitioning,querypatternsneedtobetakenintoaccountsothat
shardsthemselvesdonotbecomeperformancebottlenecks.Forexample,queriesrequiring
datafrommultipleshardswillimposeperformancepenalties.Datalocalitykeeps
commonlyaccesseddataco-locatedonasingleshardandhelpscountersuchperformance
issues.

Replication
Replicationstoresmultiplecopiesofadataset,knownasreplicas,onmultiplenodes
(Figure5.7).Replicationprovidesscalabilityandavailabilityduetothefactthatthesame
dataisreplicatedonvariousnodes.Faulttoleranceisalsoachievedsincedataredundancy
ensuresthatdataisnotlostwhenanindividualnodefails.Therearetwodifferentmethods
thatareusedtoimplementreplication:
•master-slave
•peer-to-peer

Figure5.7AnexampleofreplicationwhereadatasetisreplicatedtoNodeAandNode
B,resultinginReplicaAandReplicaB.

Master-Slave
Duringmaster-slavereplication,nodesarearrangedinamaster-slaveconfiguration,and
alldataiswrittentoamasternode.Oncesaved,thedataisreplicatedovertomultiple
slavenodes.Allexternalwriterequests,includinginsert,updateanddelete,occuronthe
masternode,whereasreadrequestscanbefulfilledbyanyslavenode.InFigure5.8,
writesaremanagedbythemasternodeanddatacanbereadfromeitherSlaveAorSlave
B.

Figure5.8Anexampleofmaster-slavereplicationwhereMasterAisthesinglepoint
ofcontactforallwrites,anddatacanbereadfromSlaveAandSlaveB.
Master-slavereplicationisidealforreadintensiveloadsratherthanwriteintensiveloads
sincegrowingreaddemandscanbemanagedbyhorizontalscalingtoaddmoreslave
nodes.Writesareconsistent,asallwritesarecoordinatedbythemasternode.The
implicationisthatwriteperformancewillsufferastheamountofwritesincreases.Ifthe
masternodefails,readsarestillpossibleviaanyoftheslavenodes.
Aslavenodecanbeconfiguredasabackupnodeforthemasternode.Intheeventthatthe

masternodefails,writesarenotsupporteduntilamasternodeisreestablished.Themaster
nodeiseitherresurrectedfromabackupofthemasternode,oranewmasternodeis
chosenfromtheslavenodes.
Oneconcernwithmaster-slavereplicationisreadinconsistency,whichcanbeanissueifa
slavenodeisreadpriortoanupdatetothemasterbeingcopiedtoit.Toensureread
consistency,avotingsystemcanbeimplementedwhereareadisdeclaredconsistentifthe
majorityoftheslavescontainthesameversionoftherecord.Implementationofsucha
votingsystemrequiresareliableandfastcommunicationmechanismbetweentheslaves.
Figure5.9illustratesascenariowherereadinconsistencyoccurs.
1.UserAupdatesdata.
2.ThedataiscopiedovertoSlaveAbytheMaster.
3.BeforethedataiscopiedovertoSlaveB,UserBtriestoreadthedatafromSlave
B,whichresultsinaninconsistentread.
4.ThedatawilleventuallybecomeconsistentwhenSlaveBisupdatedbytheMaster.

Figure5.9Anexampleofmaster-slavereplicationwherereadinconsistencyoccurs.

Peer-to-Peer
Withpeer-to-peerreplication,allnodesoperateatthesamelevel.Inotherwords,thereis
notamaster-slaverelationshipbetweenthenodes.Eachnode,knownasapeer,isequally
capableofhandlingreadsandwrites.Eachwriteiscopiedtoallpeers,asillustratedin
Figure5.10.

Figure5.10WritesarecopiedtoPeersA,BandCsimultaneously.Dataisreadfrom
PeerA,butitcanalsobereadfromPeersBorC.
Peer-to-peerreplicationispronetowriteinconsistenciesthatoccurasaresultofa
simultaneousupdateofthesamedataacrossmultiplepeers.Thiscanbeaddressedby
implementingeitherapessimisticoroptimisticconcurrencystrategy.
•Pessimisticconcurrencyisaproactivestrategythatpreventsinconsistency.Ituses
lockingtoensurethatonlyoneupdatetoarecordcanoccuratatime.However,this
isdetrimentaltoavailabilitysincethedatabaserecordbeingupdatedremains
unavailableuntilalllocksarereleased.
•Optimisticconcurrencyisareactivestrategythatdoesnotuselocking.Instead,it
allowsinconsistencytooccurwithknowledgethateventuallyconsistencywillbe
achievedafterallupdateshavepropagated.
Withoptimisticconcurrency,peersmayremaininconsistentforsomeperiodoftime
beforeattainingconsistency.However,thedatabaseremainsavailableasnolockingis
involved.Likemaster-slavereplication,readscanbeinconsistentduringthetimeperiod
whensomeofthepeershavecompletedtheirupdateswhileothersperformtheirupdates.
However,readseventuallybecomeconsistentwhentheupdateshavebeenexecutedonall
peers.
Toensurereadconsistency,avotingsystemcanbeimplementedwhereareadisdeclared
consistentifthemajorityofthepeerscontainthesameversionoftherecord.As
previouslyindicated,implementationofsuchavotingsystemrequiresareliableandfast
communicationmechanismbetweenthepeers.
Figure5.11demonstratesascenariowhereaninconsistentreadoccurs.
1.UserAupdatesdata.
2.a.ThedataiscopiedovertoPeerA.

b.ThedataiscopiedovertoPeerB.
3.BeforethedataiscopiedovertoPeerC,UserBtriestoreadthedatafromPeerC,
resultinginaninconsistentread.
4.ThedatawilleventuallybeupdatedonPeerC,andthedatabasewillonceagain
becomeconsistent.

Figure5.11Anexampleofpeer-to-peerreplicationwhereaninconsistentreadoccurs.

ShardingandReplication
Toimproveonthelimitedfaulttoleranceofferedbysharding,whileadditionally
benefitingfromtheincreasedavailabilityandscalabilityofreplication,bothshardingand
replicationcanbecombined,asshowninFigure5.12.