For an HVAC system with ID 20 and system age of 25 years, the building is hot ( prediction=1.0). Row(SystemInfo=u'7 22', prediction=0.0, probability=DenseVector()) Row(SystemInfo=u'17 10', prediction=1.0, probability=DenseVector()) Row(SystemInfo=u'9 22', prediction=1.0, probability=DenseVector()) Row(SystemInfo=u'16 9', prediction=1.0, probability=DenseVector()) Row(SystemInfo=u'4 15', prediction=0.0, probability=DenseVector()) The output is similar to: Row(SystemInfo=u'20 25', prediction=1.0, probability=DenseVector()) Selected = lect("SystemInfo", "prediction", "probability") # Make predictions on test documents and print columns of interest # SystemInfo here is a combination of system ID followed by system ageįinally, make predictions on the test data. The model predicts whether the building with that system ID and system age will be hotter (denoted by 1.0) or cooler (denoted by 0.0). To do so, you pass on a system ID and system age (denoted as SystemInfo in the training output). Prepare a data set to run the trained model against. The value for label in the first row is 0.0, which means the building isn't hot. Notice how the actual temperature is less than the target temperature suggesting the building is cold. For example, the first row the CSV file has this data: The output is similar to: +-+-+-+Ĭomparing the output against the raw CSV file. Verify the training document to checkpoint your progress with the application. Pipeline = Pipeline(stages=)įor more information about pipeline and how it works, see Apache Spark machine learning pipeline.įit the pipeline to the training document. Lr = LogisticRegression(maxIter=10, regParam=0.01) HashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features") tokenizer = Tokenizer(inputCol="SystemInfo", outputCol="words") Otherwise the building is cold, denoted by the value 0.0.Ĭonfigure the Spark machine learning pipeline that consists of three stages: tokenizer, hashingTF, and lr. If the actual temperature is greater, the building is hot, denoted by the value 1.0.
SPARK JUPYTER NOTEBOOK TUTORIAL CODE
In the code snippet, you define a function that compares the actual temperature with the target temperature. # Load the raw HVAC.csv file, parse it using the functionĭata = sc.textFile("/HdiSamples/HdiSamples/SensorSampleData/hvac/HVAC.csv")ĭocuments = data.filter(lambda s: "Date" not in s).map(parseDocument) Return LabeledDocument((values), textValue, hot) TextValue = str(values) + " " + str(values) # Define a function that parses the raw CSV file and returns an object of type LabeledDocument LabeledDocument = Row("BuildingID", "SystemInfo", "label") Load the data (hvac.csv), parse it, and use it to train the model. from pyspark.ml import Pipelineįrom pyspark.ml.classification import LogisticRegressionįrom pyspark.ml.feature import HashingTF, Tokenizerįrom import LogisticRegressionWithSGDįrom import LabeledPoint Paste the following snippet in an empty cell, and then press SHIFT + ENTER. Import the types required for this scenario. For the instructions, see Create a Jupyter Notebook file. Do the following steps to create the application.Ĭreate a Jupyter Notebook using the PySpark kernel. In the pipeline, you split the document into words, convert the words into a numerical feature vector, and finally build a prediction model using the feature vectors and labels. The DataFrames help users create and tune practical machine learning pipelines. ML Pipelines provide a uniform set of high-level APIs built on top of DataFrames. This application uses a Spark ML pipeline to do a document classification. You can predict whether a building will be hotter or colder based on the target temperature, given system ID, and system age.ĭevelop a Spark machine learning application using Spark MLlib The System column represents the system ID and the SystemAge column represents the number of years the HVAC system has been in place at the building. The data shows the target temperature and the actual temperature of some buildings that have HVAC systems installed. The file is located at \HdiSamples\HdiSamples\SensorSampleData\hvac. The application uses the sample HVAC.csv data that is available on all clusters by default. For more information, see Load data and run queries with Apache Spark on HDInsight. See Create an Apache Spark cluster.įamiliarity with using Jupyter Notebooks with Spark on HDInsight.
0 kommentar(er)
