How to write hive scripts in python

This article introduces how to write hive script in python, the text of the sample code through the introduction of the very detailed, for everyone's learning or work has a certain reference learning value, you can refer to the following friends

1. Direct execution of .sql scripts

import numpy as np
import pandas as pd
import lightgbm as lgb
from pandas import DataFrame
from sklearn.model_selection import train_test_split
from io import StringIO
import gc
import sys
import os
hive_cmd = "hive -f ./sql/"
output = (hive_cmd)
data_cart_prop = pd.read_csv(StringIO(unicode((),'utf-8')), sep="\t",header=0)

2. Hive statement execution

Suppose there is the following hive sql:

hive_cmd = 'hive -e "select count(*) from hbase.routermac_sort_10;"'

This hive sql is generally executed in python as follows:

(hive_cmd)

---------------------

hive_cmd1 = "hive -f ./"
output1 = (hive_cmd1)
test_user = pd.read_csv(StringIO(unicode((),'utf-8')), sep="\t",header=0)
 
hive_cmd2 = "hive -f ./"
output2 = (hive_cmd2)
test_action = pd.read_csv(StringIO(unicode((),'utf-8')), sep="\t",header=0)
 
hive_cmd3 = "hive -f ./"
output3 = (hive_cmd3)
test_click = pd.read_csv(StringIO(unicode((),'utf-8')), sep="\t",header=0)

To display the table header, add a line to the script: set =true;

Alternatively, use the following statement:

hive_cmd = 'hive -e "set =true;SELECT * FROM dev.temp_dev_jypt_decor_user_label_phase_one_view_feature WHERE(dt = "2018-09-17");"'
output = (hive_cmd)
data_cart_prop = pd.read_csv(StringIO(unicode((),'utf-8')), sep="\t",header=0)

3、tf Display Memory Occupancy

import tensorflow as tf
tf.enable_eager_execution()
x = tf.get_variable('x', shape=[1], initializer=tf.constant_initializer(3.))
with () as tape:   
  y = (x)
  y_grad = (y, x)    
print([(), y_grad.numpy()])

This is the whole content of this article.