{"id":1283,"date":"2017-01-04T17:55:48","date_gmt":"2017-01-04T22:55:48","guid":{"rendered":"http:\/\/www.xavignu.com\/?p=1283"},"modified":"2017-01-04T17:55:48","modified_gmt":"2017-01-04T22:55:48","slug":"sum-function-in-apache-pig","status":"publish","type":"post","link":"https:\/\/www.xavignu.com\/?p=1283","title":{"rendered":"SUM function in Apache Pig"},"content":{"rendered":"

I have been learning Big Data<\/a> and among other things Apache Pig<\/a>. So initially I thought that once you have loaded the data a SUM<\/a> applied after a FOREACH would bring the total amount, but that’s not the case. A grouping needs to be performed first else SUM would give an error.<\/p>\n

<\/p>\n

We will perform the below script:<\/p>\n

[code language=”text”]
\n— This PIG script sums the total amount of sales
\n— First load the data from sales.txt file
\ndata = LOAD ‘sales.txt’ USING PigStorage(‘,’) AS (name:chararray, price:int, country:chararray);<\/p>\n

— Group the data
\ngrouped = GROUP data ALL;<\/p>\n

— Once grouped generate total sum of all sales
\ntotal = FOREACH grouped GENERATE SUM(data.price);<\/p>\n

— Print to screen
\nDUMP total;
\n[\/code]<\/p>\n

Save the above code with .pig extension. Test data will be loaded from below file.
\n[code language=”text”]
\nAlice,3000,us
\nAlice,2000,us
\nBob,500,ca
\nJuan,500,mx
\nHans,2000,de
\nJoan,1000,fr
\nPiero,6000,it
\n[\/code]<\/p>\n

and execute locally:<\/p>\n

pig -x local  totalsales.pig \r\n17\/01\/04 14:28:57 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead\r\n17\/01\/04 14:28:57 INFO util.ProcessTree: setsid exited with exit code 0\r\n17\/01\/04 14:28:58 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead\r\n(15000) \r\n<\/pre>\n
Reference:
\nThomas Henson<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
I have been learning Big Data and among other things Apache Pig. So initially I thought that once you have loaded the data a SUM applied after a FOREACH would bring the total amount, but that’s not the case. A grouping needs to be performed first else SUM would give an error.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[74],"tags":[20,75,6,23,70],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/pTQgt-kH","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/posts\/1283"}],"collection":[{"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1283"}],"version-history":[{"count":7,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/posts\/1283\/revisions"}],"predecessor-version":[{"id":1290,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/posts\/1283\/revisions\/1290"}],"wp:attachment":[{"href":"https:\/\/www.xavignu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1283"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1283"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1283"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}