{"id":1385,"date":"2017-03-27T17:40:14","date_gmt":"2017-03-27T21:40:14","guid":{"rendered":"http:\/\/www.xavignu.com\/?p=1385"},"modified":"2017-03-27T17:40:14","modified_gmt":"2017-03-27T21:40:14","slug":"apache-flume-to-write-web-server-logs-to-hadoop","status":"publish","type":"post","link":"https:\/\/www.xavignu.com\/?p=1385","title":{"rendered":"Apache Flume to write web server logs to Hadoop"},"content":{"rendered":"

In this post we will use flume<\/a> to dump Apache<\/a> webserver logs into HDFS<\/a>. We already have a web server running and flume installed, but we need to configure a target and a source.<\/p>\n

We use the following file as target.<\/p>\n

[xml]
\n## TARGET AGENT ##
\n## configuration file location: \/etc\/flume-ng\/conf
\n## START Agent: flume-ng agent -c conf -f \/etc\/flume-ng\/conf\/flume-trg-agent.conf -n collector<\/p>\n

#http:\/\/flume.apache.org\/FlumeUserGuide.html#avro-source
\ncollector.sources = AvroIn
\ncollector.sources.AvroIn.type = avro
\ncollector.sources.AvroIn.bind = 0.0.0.0
\ncollector.sources.AvroIn.port = 4545
\ncollector.sources.AvroIn.channels = mc1 mc2<\/p>\n

## Channels ##
\n## Source writes to 2 channels, one for each sink
\ncollector.channels = mc1 mc2<\/p>\n

#http:\/\/flume.apache.org\/FlumeUserGuide.html#memory-channel<\/p>\n

collector.channels.mc1.type = memory
\ncollector.channels.mc1.capacity = 100<\/p>\n

collector.channels.mc2.type = memory
\ncollector.channels.mc2.capacity = 100<\/p>\n

## Sinks ##
\ncollector.sinks = LocalOut HadoopOut<\/p>\n

## Write copy to Local Filesystem
\n#http:\/\/flume.apache.org\/FlumeUserGuide.html#file-roll-sink
\ncollector.sinks.LocalOut.type = file_roll
\ncollector.sinks.LocalOut.sink.directory = \/var\/log\/flume-ng
\ncollector.sinks.LocalOut.sink.rollInterval = 0
\ncollector.sinks.LocalOut.channel = mc1<\/p>\n

## Write to HDFS
\n#http:\/\/flume.apache.org\/FlumeUserGuide.html#hdfs-sink
\ncollector.sinks.HadoopOut.type = hdfs
\ncollector.sinks.HadoopOut.channel = mc2
\ncollector.sinks.HadoopOut.hdfs.path = \/user\/training\/flume\/events\/%{log_type}\/%y%m%d
\ncollector.sinks.HadoopOut.hdfs.fileType = DataStream
\ncollector.sinks.HadoopOut.hdfs.writeFormat = Text
\ncollector.sinks.HadoopOut.hdfs.rollSize = 0
\ncollector.sinks.HadoopOut.hdfs.rollCount = 10000
\ncollector.sinks.HadoopOut.hdfs.rollInterval = 600<\/p>\n

[\/xml]
\n<\/p>\n

and below as source
\n[xml]
\n## SOURCE AGENT ##
\n## Local instalation: \/home\/ec2-user\/apache-flume
\n## configuration file location: \/home\/ec2-user\/apache-flume\/conf
\n## bin file location: \/home\/ec2-user\/apache-flume\/bin
\n## START Agent: bin\/flume-ng agent -c conf -f conf\/flume-src-agent.conf -n source_agent<\/p>\n

# http:\/\/flume.apache.org\/FlumeUserGuide.html#exec-source
\nsource_agent.sources = apache_server
\nsource_agent.sources.apache_server.type = exec
\nsource_agent.sources.apache_server.command = tail -f \/var\/log\/httpd\/access_log
\nsource_agent.sources.apache_server.batchSize = 1
\nsource_agent.sources.apache_server.channels = memoryChannel
\nsource_agent.sources.apache_server.interceptors = itime ihost itype<\/p>\n

# http:\/\/flume.apache.org\/FlumeUserGuide.html#timestamp-interceptor
\nsource_agent.sources.apache_server.interceptors.itime.type = timestamp<\/p>\n

# http:\/\/flume.apache.org\/FlumeUserGuide.html#host-interceptor
\nsource_agent.sources.apache_server.interceptors.ihost.type = host
\nsource_agent.sources.apache_server.interceptors.ihost.useIP = false
\nsource_agent.sources.apache_server.interceptors.ihost.hostHeader = host<\/p>\n

# http:\/\/flume.apache.org\/FlumeUserGuide.html#static-interceptor
\nsource_agent.sources.apache_server.interceptors.itype.type = static
\nsource_agent.sources.apache_server.interceptors.itype.key = log_type
\nsource_agent.sources.apache_server.interceptors.itype.value = apache_access_combined<\/p>\n

# http:\/\/flume.apache.org\/FlumeUserGuide.html#memory-channel
\nsource_agent.channels = memoryChannel
\nsource_agent.channels.memoryChannel.type = memory
\nsource_agent.channels.memoryChannel.capacity = 100<\/p>\n

## Send to Flume Collector on Hadoop Node
\n# http:\/\/flume.apache.org\/FlumeUserGuide.html#avro-sink
\nsource_agent.sinks = avro_sink
\nsource_agent.sinks.avro_sink.type = avro
\nsource_agent.sinks.avro_sink.channel = memoryChannel
\nsource_agent.sinks.avro_sink.hostname = 192.168.46.169
\nsource_agent.sinks.avro_sink.port = 4545<\/p>\n

[\/xml]
\nWe start the target<\/p>\n

flume-ng agent -c conf -f flume-trg-agent.conf -n collector \r\n<\/pre>\n
And now we start the source<\/p>\n
flume-ng agent -c conf -f flume-src-agent.conf -n source_agent\r\n<\/pre>\n
And after some time we can see Apache logs being written to HDFS<\/p>\n
hdfs dfs -cat  \/user\/training\/flume\/events\/apache_access_combined\/170327\/FlumeData.1490635725898\r\n192.168.46.169 - - [27\/Mar\/2017:10:28:42 -0700] \"GET \/ HTTP\/1.1\" 200 401 \"-\" \"curl\/7.19.7 (x86_64-redhat-linux-gnu) libcurl\/7.19.7 NSS\/3.14.0.0 zlib\/1.2.3 libidn\/1.18 libssh2\/1.4.2\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:28:57 -0700] \"GET \/ HTTP\/1.1\" 200 401 \"-\" \"curl\/7.19.7 (x86_64-redhat-linux-gnu) libcurl\/7.19.7 NSS\/3.14.0.0 zlib\/1.2.3 libidn\/1.18 libssh2\/1.4.2\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:29:07 -0700] \"GET \/ HTTP\/1.1\" 200 401 \"-\" \"curl\/7.19.7 (x86_64-redhat-linux-gnu) libcurl\/7.19.7 NSS\/3.14.0.0 zlib\/1.2.3 libidn\/1.18 libssh2\/1.4.2\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:29:08 -0700] \"GET \/ HTTP\/1.1\" 200 401 \"-\" \"curl\/7.19.7 (x86_64-redhat-linux-gnu) libcurl\/7.19.7 NSS\/3.14.0.0 zlib\/1.2.3 libidn\/1.18 libssh2\/1.4.2\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:29:10 -0700] \"GET \/ HTTP\/1.1\" 200 401 \"-\" \"curl\/7.19.7 (x86_64-redhat-linux-gnu) libcurl\/7.19.7 NSS\/3.14.0.0 zlib\/1.2.3 libidn\/1.18 libssh2\/1.4.2\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:30:48 -0700] \"GET \/ HTTP\/1.1\" 200 401 \"-\" \"Mozilla\/5.0 (X11; Linux x86_64; rv:17.0) Gecko\/20131029 Firefox\/17.0\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:30:48 -0700] \"GET \/favicon.ico HTTP\/1.1\" 404 289 \"-\" \"Mozilla\/5.0 (X11; Linux x86_64; rv:17.0) Gecko\/20131029 Firefox\/17.0\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:30:48 -0700] \"GET \/favicon.ico HTTP\/1.1\" 404 289 \"-\" \"Mozilla\/5.0 (X11; Linux x86_64; rv:17.0) Gecko\/20131029 Firefox\/17.0\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:30:51 -0700] \"GET \/first.html HTTP\/1.1\" 200 291 \"http:\/\/192.168.46.169\/\" \"Mozilla\/5.0 (X11; Linux x86_64; rv:17.0) Gecko\/20131029 Firefox\/17.0\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:30:52 -0700] \"GET \/index.html HTTP\/1.1\" 200 401 \"http:\/\/192.168.46.169\/first.html\" \"Mozilla\/5.0 (X11; Linux x86_64; rv:17.0) Gecko\/20131029 Firefox\/17.0\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:30:53 -0700] \"GET \/second.html HTTP\/1.1\" 200 293 \"http:\/\/192.168.46.169\/index.html\" \"Mozilla\/5.0 (X11; Linux x86_64; rv:17.0) Gecko\/20131029 Firefox\/17.0\"\r\n192.168.46.169 - - [27\/Mar\/2017:10:32:01 -0700] \"GET \/ HTTP\/1.1\" 200 401 \"-\" \"curl\/7.19.7 (x86_64-redhat-linux-gnu) libcurl\/7.19.7 NSS\/3.14.0.0 zlib\/1.2.3 libidn\/1.18 libssh2\/1.4.2\"\r\n\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
In this post we will use flume to dump Apache webserver logs into HDFS. We already have a web server running and flume installed, but we need to configure a target and a source. We use the following file as target. [xml] ## TARGET AGENT ## ## configuration file location: \/etc\/flume-ng\/conf ## START Agent: flume-ng […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[74],"tags":[20,80,81,6,23,70],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/pTQgt-ml","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/posts\/1385"}],"collection":[{"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1385"}],"version-history":[{"count":6,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/posts\/1385\/revisions"}],"predecessor-version":[{"id":1392,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=\/wp\/v2\/posts\/1385\/revisions\/1392"}],"wp:attachment":[{"href":"https:\/\/www.xavignu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1385"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1385"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.xavignu.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1385"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}