MapReduce
mapreduce est une technique de programmation qui convient à l’analyse de jeux de données trop volumineux pour être stockés dans la mémoire de votre ordinateur. Cette technique utilise un datastore pour traiter les données par petits blocs. Elle se compose d’une phase « Map » qui formate les données ou effectue un calcul préliminaire, et d’une phase « Reduce » qui agrège tous les résultats de la phase « Map ». Pour plus d’informations, consultez Getting Started with MapReduce.
Pour plus d’informations sur l’utilisation d’autres produits avec mapreduce, consultez Speed Up and Deploy MapReduce Using Other Products.
Fonctions
Objets
KeyValueStore | Store key-value pairs for use with mapreduce |
ValueIterator | An iterator over intermediate values for use with mapreduce |
Rubriques
- Getting Started with MapReduce
Learn about the MapReduce programming technique and run an example calculation.
- Write a Map Function
Create a map function for use in a
mapreducealgorithm. - Write a Reduce Function
Create a reduce function for use in a
mapreducealgorithm. - Speed Up and Deploy MapReduce Using Other Products
Capabilities of other products to speed up and share
mapreducealgorithms.
Résolution des problèmes
This example shows how to debug mapreduce algorithms in
MATLAB®. Debugging enables you to follow the movement of data between the different
phases of mapreduce execution and inspect the state of all
intermediate variables.
Sélection d՚exemples
Analyze Big Data in MATLAB Using MapReduce
Use the mapreduce function to process a large amount of file-based data. The MapReduce algorithm is a mainstay of many modern "big data" applications. This example operates on a single computer, but the code can scale up to use Hadoop®.
Find Maximum Value with MapReduce
Find the maximum value of a single variable in a data set using mapreduce. It demonstrates the simplest use of mapreduce since there is only one key and minimal computation.
Compute Mean Value with MapReduce
Compute the mean of a single variable in a data set using mapreduce. It demonstrates a simple use of mapreduce with one key, minimal computation, and an intermediate state (accumulating intermediate sum and count).
Create Histograms Using MapReduce
Visualize patterns in a large data set without having to load all of the observations into memory simultaneously. It demonstrates how to compute lower volume summaries of the data that are sufficient to generate a graphic.
Compute Mean by Group Using MapReduce
Compute the mean by group in a data set using mapreduce. It demonstrates how to do computations on subgroups of data.
Simple Data Subsetting Using MapReduce
Extract a subset of a large data set.
Using MapReduce to Compute Covariance and Related Quantities
Compute the mean and covariance for several variables in a large data set using mapreduce. It then uses the covariance to perform several follow-up calculations that do not require another iteration over the entire data set.
Compute Summary Statistics by Group Using MapReduce
Compute summary statistics organized by group using mapreduce. It demonstrates the use of an anonymous function to pass an extra grouping parameter to a parameterized map function. This parameterization allows you to quickly recalculate the statistics using a different grouping variable.
Using MapReduce to Fit a Logistic Regression Model
Use mapreduce to carry out simple logistic regression using a single predictor. It demonstrates chaining multiple mapreduce calls to carry out an iterative algorithm. Since each iteration requires a separate pass through the data, an anonymous function passes information from one iteration to the next to supply information directly to the mapper.
Tall Skinny QR (TSQR) Matrix Factorization Using MapReduce
Compute a tall skinny QR (TSQR) factorization using mapreduce. It demonstrates how to chain mapreduce calls to perform multiple iterations of factorizations, and uses the info argument of the map function to compute numeric keys.
Compute Maximum Average HSV of Images with MapReduce
Use ImageDatastore and mapreduce to find images with maximum hue, saturation and brightness values in an image collection.
