Hadoop Hive Configuration

The sections to follow explain how to configure Hive for weblog analytics.

Requirements

• Java 1.6
• Hadoop 0.20.x.

Installing Hadoop Hive from a Stable Release

First, download the latest stable release of Hive from one of the Apache download mirrors.

Next, unpack the tarball, which will create a subdirectory named hive-x.y.z:

 $ tar -xzvf hive-x.y.z.tar.gz

Point the environment variable HIVE_HOME to the installation directory:

      $ cd hive-x.y.z$ export HIVE_HOME={{pwd}}

Finally, add $HIVE_HOME/bin to your PATH:

       $ export PATH=$HIVE_HOME/bin:$PATH

Running Hadoop Hive

Because Hive uses Hadoop either:

• you must have Hadoop in your path, OR
• export HADOOP_HOME=

In addition, you must create /tmp and /user/hive/warehouse (aka hive.metastore.warehouse.dir) and set them chmod g+w in HDFS before a table can be created in Hive.

Commands to perform this setup are as follows:

  $ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp  $ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

I also find it useful, but not necessary, to set HIVE_HOME as follows:

$ export HIVE_HOME=

To use the Hive command line interface (CLI) from the shell:

$ $HIVE_HOME/bin/hive

Hive Runtime Configuration

• Hive queries are executed using MapReduce queries. Therefore, the behavior of such queries can be controlled by the Hadoop configuration variables.
• The CLI command SETcan be used to set any Hadoop (or Hive) configuration variable. For example:

  hive> SET mapred.job.tracker=myhost.mycompany.com:50030;hive> SET -v;

Hive, MapReduce and Local-Mode

The Hive compiler generates MapReduce jobs for most queries. These jobs are then submitted to the Map-Reduce cluster indicated by this variable:

mapred.job.tracker

This usually points to a MapReduce cluster with multiple nodes, but Hadoop also provides an option to run MapReduce jobs locally on the user's PC. This can be very useful for running queries over small data sets because in such cases local mode execution is usually much faster than submitting jobs to a large cluster. Data is accessed transparently from HDFS. Conversely, local mode runs with only one reducer and can be very slow when processing larger data sets.

Starting with version 0.7, Hive fully supports local mode execution, which you can activate by enabling the following option:

Hive> SET mapred.job.tracker=local;

In addition, mapred.local.dir should point to a path that's valid on the local machine (for example /tmp/<:username>/mapred/local). (Otherwise, the user will get an exception allocating local disk space).

Starting with version 0.7, Hive also supports a mode to run MapReduce jobs in local-mode automatically. The relevant options are:

 Hive> SET hive.exec.mode.local.auto=false;

Hadoop Hive Data Load

Hive provides tools to enable easy data ETL, a mechanism to put structures on the data, and defines a simple SQL-like query language, called QL, that enables users familiar with SQL to query the data. At the same time, Hive QL also allows programmers familiar with MapReduce to plug in their custom mappers and reducers to perform more sophisticated analysis that may not be supported by the built-in capabilities of the language.

Below is the create table command for Hive.

create table weblogs (client_ip    string,full_request_date string,day    string, month    string, 
month_num int, year    string, hour    string, minute    string, second    string,timezone string,http_verb    string, 
uri    string, http_status_code    string,bytes_returned  string,referrer string,user_agent string) 
row format delimited fields terminated by '	' stored as textfile 

The below command is used for loading data from an HDFS location to a Hive table.

LOAD DATA INPATH '' INTO TABLE 
After loading the data into the table, the normal user can query Hive using Hive QL. Below is an example query for getting user counts of each location.
SELECT client_ip  , COUNT(client_ip) FROM weblogs GROUP BY client_ip 
Hadoop Hive JDBC Support
Hive also supports JDBC connections. To connect Hive with JDBC, you need to start the Hive Thrift Server as follows.
Export HIVE_PORT=9999      Hive –service hiveserver 
Here are the steps to establish a Hive JDBC connection:


Add hive-jdbc0.7.jar in the classpath; this is a type-4 driver.
Use the org.apache.hadoop.hive.jdbc.HiveDriver driver for the connection.
Connection String : jdbc:hive://:/
Use Hive QL to query the table in Hive, and it will return the result set.
Using the result set, you can project the output in graphs or charts easily.
Here is a sample Hive JDBC program:
    Class.forName("org.apache.hadoop.hive.jdbc.HiveDriver");    Connection con = DriverManager.getConnection("jdbc:hive://10.0.0.1:9999/default", "", "");    Statement stmt = con.createStatement();    Resultset res = stmt.executeQuery("SELECT client_ip  , COUNT(client_ip) FROM weblogs GROUP BY  client_ip");    while (res.next()) {      System.out.println(res.getInt(1) + "	" + res.getString(2));    }

				

					

			

				Previous			
						

				Next			
		
				
				

				

			Share the Post:		
				

				

					

								

						

															
															
																				
					
									

						

															
															
																				
					
									

						

															
															
																				
					
						
				
				

				

					

				

				
			

		
				

				

			
				The Digital Panopticon: Is Big Brother Always Watching Us Online?			
		
				

					
			May 26, 2023		
				
				

			
In the age of digital transformation, the internet has become a ubiquitous part of our lives. From socializing, shopping, and learning to more sensitive activities such as banking and healthcare,
		
				
				
				

				
			

		
				

				

			
				Embracing Change: How AI Is Revolutionizing the Developer’s Role			
		
				

					
			May 25, 2023		
				
				

			
The world of software development is changing drastically with the introduction of Artificial Intelligence and Machine Learning technologies. In the past, software developers were in charge of the entire development
		
				
				
				

				
			

		
				

				

			
				The Benefits of Using XDR Solutions			
		
				

					
			May 24, 2023		
				
				

			
Cybercriminals constantly adapt their strategies, developing newer, more powerful, and intelligent ways to attack your network. Since security professionals must innovate as well, more conventional endpoint detection solutions have evolved
		
				
				
				

				
			

		
				

				

			
				How AI is Revolutionizing Fraud Detection			
		
				

					
			May 23, 2023		
				
				

			
Artificial intelligence – commonly known as AI – means a form of technology with multiple uses. As a result, it has become extremely valuable to a number of businesses across
		
				
				
				

				
			

		
				

				

			
				Companies Leading AI Innovation in 2023			
		
				

					
			May 22, 2023		
				
				

			
Artificial intelligence (AI) has been transforming industries and revolutionizing business operations. AI’s potential to enhance efficiency and productivity has become crucial to many businesses. As we move into 2023, several
		
				
				
				

				
			

		
				

				

			
				Step-by-Step Guide to Properly Copyright Your Website			
		
				

					
			May 18, 2023		
				
				

			
Creating a website is not easy, but protecting your website is equally important. Implementing copyright laws ensures that the substance of your website remains secure and sheltered. Copyrighting your website
		
				
				
				

				
			

		
				

				

			
				Fivetran Pricing Explained			
		
				

					
			May 17, 2023		
				
				

			
One of the biggest trends of the 21st century is the massive surge in analytics. Analytics is the process of utilizing data to drive future decision-making. With so much of
		
				
				
				

				
			

		
				

				

			
				Kubernetes Logging: What You Need to Know			
		
				

					
			May 16, 2023		
				
				

			
Kubernetes from Google is one of the most popular open-source and free container management solutions made to make managing and deploying applications easier. It has a solid architecture that makes
		
				
				
				

				
			

		
				

				

			
				Why Is Ransomware Such a Major Threat?			
		
				

					
			May 15, 2023		
				
				

			
One of the most significant cyber threats faced by modern organizations is a ransomware attack. Ransomware attacks have grown in both sophistication and frequency over the past few years, forcing
		
				
				
				

				
			

		
				

				

			
				Tools You Need to Make a Data Dictionary			
		
				

					
			May 12, 2023		
				
				

			
Data dictionaries are crucial for organizations of all sizes that deal with large amounts of data. they are centralized repositories of all the data in organizations, including metadata such as
		
				
				
				

				
			

		
				

				

			
				10 Software Development Tips to Get Early Funding for your Startup			
		
				

					
			May 11, 2023		
				
				

			
If you’re thinking about a startup, it’s likely you need to raise an initial round of funding for your venture. This article covers some of the very early development techniques
		
				
				
				

				
			

		
				

				

			
				How to Earn Money in the Gig Economy? JumpTask App Review			
		
				

					
			May 10, 2023		
				
				

			
Are you aware of the possibility of a recession in 2023? This year has been challenging for the economy, with reports of high prices and significant corporations laying off workers
		
				
				
				
					
							
		
				

				

			
						
						Show More