Tải bản đầy đủ
6 Oh No, the Data Doesn’t Fit into Memory!

6 Oh No, the Data Doesn’t Fit into Memory!

Tải bản đầy đủ

the counts or totals for that chunk and record them. After reading all the
chunks, we add up all the counts or totals in order to calculate our grand
means or proportions.
As another example, suppose we are performing a statistical operation,
say calculating principle components, in which we have a huge number of
rows—that is, a huge number of observations—but the number of variables
is manageable. Again, chunking could be the solution. We apply the statistical operation to each chunk and then average the results over all the
chunks. My mathematical research shows that the resulting estimators are
statistically efficient in a wide class of statistical methods.


Using R Packages for Memory Management

Again looking at a bit more sophistication, there are alternatives for accommodating large memory requirements in the form of some specialized R
One such package is RMySQL, an R interface to SQL databases. Using it
requires some database expertise, but this package provides a much more
efficient and convenient way to handle large data sets. The idea is to have
SQL do its variable/case selection operations for you back at the database
end and then read the resulting selected data as it is produced by SQL.
Since the latter will typically be much smaller than the overall data set,
you will likely be able to circumvent R’s memory restriction.
Another useful package is biglm, which does regression and generalized
linear-model analysis on very large data sets. It also uses chunking but in a
different manner: Each chunk is used to update the running totals of sums
needed for the regression analysis and then discarded.
Finally, some packages do their own storage management independently of R and thus can deal with very large data sets. The two most commonly used today are ff and bigmemory. The former sidesteps memory constraints by storing data on disk instead of memory, essentially transparently
to the programmer. The highly versatile bigmemory package does the same,
but it can store data not only on disk but also in the machine’s main memory, which is ideal for multicore machines.

Performance Enhancement: Speed and Memory



R is a great language, but it can’t do everything well. Thus, it is sometimes desirable
to call code written in other languages from
R. Conversely, when working in other great languages, you may encounter tasks that could be better
done in R.
R interfaces have been developed for a number of other languages,
from ubiquitous languages like C to esoteric ones like the Yacas computer
algebra system. This chapter will cover two interfaces: one for calling
C/C++ from R and the other for calling R from Python.


Writing C/C++ Functions to Be Called from R
You may wish to write your own C/C++ functions to be called from R. Typically, the goal is performance enhancement, since C/C++ code may run
much faster than R, even if you use vectorization and other R optimization
techniques to speed things up.
Another possible goal in dropping down to the C/C++ level is specialized I/O. For example, R uses the TCP protocol in layer 3 of the standard
Internet communication system, but UDP can be faster in some settings.

To work in UDP, you need C/C++, which requires an interface to R for those
R actually offers two C/C++ interfaces via the functions .C() and
.Call(). The latter is more versatile but requires some knowledge of R’s
internal structures, so we’ll stick with .C() here.


Some R-to-C/C++ Preliminaries

In C, two-dimensional arrays are stored in row-major order, in contrast to R’s
column-major order. For instance, if you have a 3-by-4 array, the element in
the second row and second column is element number 5 of the array when
viewed linearly, since there are three elements in the first column and this
is the second element in the second column. Also keep in mind that C subscripts begin at 0, rather than at 1, as with R.
All the arguments passed from R to C are received by C as pointers.
Note that the C function itself must return void. Values that you would
ordinarily return must be communicated through the function’s arguments, such as result in the following example.


Example: Extracting Subdiagonals from a Square Matrix

Here, we will write C code to extract subdiagonals from a square matrix.
(Thanks to my former graduate assistant, Min-Yu Huang, who wrote an earlier version of this function.) Here’s the code for the file sd.c:

// required

// arguments:
m: a square matrix
n: number of rows/columns of m
k: the subdiagonal index--0 for main diagonal, 1 for first
subdiagonal, 2 for the second, etc.
result: space for the requested subdiagonal, returned here
void subdiag(double *m, int *n, int *k, double *result)
int nval = *n, kval = *k;
int stride = nval + 1;
for (int i = 0, j = kval; i < nval-kval; ++i, j+= stride)
result[i] = m[j];

The variable stride alludes to a concept from the parallel-processing
community. Say we have a matrix in 1,000 columns and our C code is looping through all the elements in a given column, from top to bottom. Again,
since C uses row-major order, consecutive elements in the column are 1,000
elements apart from each other if the matrix is viewed as one long vector.

Chapter 15