Apache Pig allows you to use the GROUP statement to combine data by a key, and unlike SQL you do not need to apply an aggregation function like SUM, MAX, MIX to return just a single row for each group. Pig just groups values for each key into separate bags that you can iterate and transform as needed.

Let’s consider the following meteo data set containing state, city, annual high and annual low temperature as follows:

CA,Irvine,73,54
CA,San Diego,70,58
CA,Berkeley,68,48
CA,San Jose,73,51
MA,Boston,59,44
MA,Bedford,60,38
MA,Charlton,58,36
WA,Seattle,60,45
WA,Tacoma,62,45

Now we will group data by state and see the results:

-- Load input data from a file
d = load 's3://epic/dmtolpeko/meteo.txt' 
  using PigStorage(',') as (state:chararray, city:chararray, 
                            high:chararray, low:chararray);

-- Group data by state
g = group d by state; 

-- Show the results
dump g;

(CA,{(CA,San Jose,73,51),(CA,Berkeley,68,48),(CA,San Diego,70,58),(CA,Irvine,73,54)})
(MA,{(MA,Charlton,58,36),(MA,Bedford,60,38),(MA,Boston,59,44)})
(WA,{(WA,Tacoma,62,45),(WA,Seattle,60,45)})

You can see that data are grouped by each key, and we do not need to apply an aggregate function as this is required by GROUP BY in SQL.

Now let’s write a Python UDF to iterate each item in the group and return just 2 first rows with values city and low temperature only.

Note that you can get this functionality in pure Pig syntax, and this example is just intended to show how you can handle bag items inside Python UDF that can be useful to implement some more complex transformations and aggregations.

# Pig UDF returns a bag of 2-element tuples
@outputSchema('result:{t:(city:chararray,low:chararray)}')
def getCitiesLow(data):
    result = []
    # Select first 2 items i group only
    for i in range(2):
        city = data[i][1]
        low = data[i][3]
        result.append((city, low))
    return result

Put this Python code to a file getCitiesLow.py and run the following Pig script:

-- Register UDF
register './getCitiesLow.py' USING jython as udf;

-- Transforming data using UDF
s = foreach g generate group, udf.getCitiesLow(d);

-- Show the results
dump s;

(CA,{(San Jose,51),(Berkeley,48)})
(MA,{(Charlton,36),(Bedford,38)})
(WA,{(Tacoma,45),(Seattle,45)})

From this example you can learn how nicely you can handle Pig a bag of tuples in Python, it just becomes a list of tuples that you can iterate and extract individual items. You can also see how the input group can be transformed: in our example we selected only 2 rows from each group and returned a different number of columns. This can useful in some advanced transformations.

Typically an user-defined function (UDF) is a scalar function i.e. it returns a single value, and quite often you need to call many UDFs to get different calculations on the same input data.

In some cases parsing the same data over and over again can be expensive, so it makes sense to write a UDF that can process input data once but returns multiple values i.e. it returns a tuple.

Let’s complicate our example a little bit and write an UDF that returns a tuple of arrays (bags in terms of Pig Latin, lists in terms of Python).

For example, assume we have a string of numbers (many rows, but single column) as follows:

1,2,10,5,15,8
4,31,2,76,3,16
...

And we want to split these lists of numbers to 2 lists: one for odd and another for even numbers. Here is UDF code in Python:

# Pig UDF returns a tuple of 2 bags
@outputSchema('result:tuple(odd:{t:(val:int)},even:{t:(val:int)})')
def getOddEven(data):
    odd_bag = []
    even_bag = []
    for i in str(data).split(','):
        if int(i) % 2 != 0:
            odd_bag.append(int(i))
        else:
            even_bag.append(int(i))
    return (odd_bag, even_bag)

Put this Python code to a file getOddEven.py and run the following Pig script:

-- Register UDF
register './getOddEven.py' USING jython as udf;

-- Load input data from a file
d = load 's3://epic/dmtolpeko/data.txt' 
         using PigStorage() as (c1:chararray);

-- Transforming data using UDF
s = foreach d generate udf.getOddEven(c1);

-- Show the results
dump s;

Here is the result:

{(1),(5),(15)}, {(2),(10),(8)}
{(31),(3)},     {(4),(2),(76),(16)}

You can see that getOddEven.py function transformed input data into 2 bags: one for odd numbers and another for even numbers, and unlike the traditional scalar UDF approach, we parsed the input data only once.